How to Build an AI Agent with Function Calling

Key takeaways

• 01

  Function calling is a schema and a gateway, not a prompt that lists APIs in prose.

• 02

  The executor must enforce allowlists, argument validation, and query-time user scopes; the model cannot be trusted to police itself.

• 03

  Mark every tool as read or write; retries, idempotency, and confirmation rules differ completely between the two.

• 04

  Cap the loop (steps, time, tokens) and require a terminal action — answer, refuse, or escalate — so agents cannot spin.

• 05

  Traces of message → tool name → args → result are the eval dataset; without them you cannot measure task success.

Function calling is an API contract: you send the model a list of tools with JSON Schema parameters; it may return a structured call instead of (or before) a user-visible answer; your code runs the call and returns a result; the model continues. That loop is how you build an AI agent with tools rather than a chatbot that only talks.

The model never gets a raw shell. If your design requires arbitrary code or URLs, you have left function calling and entered remote execution. Stay on named tools with validated arguments. MCP servers and native provider tools are both valid transports; the security model — registry, auth, audit — must be yours either way.

Vague descriptions produce vague arguments. Name tools after user jobs (`get_invoice`, `create_jira_ticket`), not after internal microservices. Constrain strings with enums and formats. Keep arity small; a 30-field blob is how you get missing required keys. Include examples in the description only if they match production — stale examples become copied mistakes.

Version schemas (`get_invoice@v2`) and pin what the serving prompt includes. A silent field rename is an outage. When two tools overlap, the model will pick at random; merge them or write a router tool with a single enum. Test schemas with a golden set of user utterances → expected tool + args before you test prose quality.

Treat every tool call as an untrusted request that happens to be JSON. The gateway allowlists names, validates arguments against schema, injects the user identity from the session (never from model-supplied user IDs), and checks scopes the way your API gateway already does. Query-time ACLs apply to tools that read documents or rows: the executor filters, the model does not.

Log denied calls. A burst of denied `delete_user` is either an injection test or a mis-prompted agent; both deserve a ticket. Do not return stack traces to the model. Return a short, structured error (`permission_denied`, `not_found`, `validation_error`) so the model can recover without leaking internals.

Reads and writes are different products. Reads may retry. Writes need an idempotency key stored before the HTTP call, a confirmation payload for irreversible actions, and a status API so a timeout can be resolved. If the user already confirmed, a retry must not pop a second confirmation that creates a second object.

Confirmation copy should show the tool, the arguments that matter (amount, destination, resource id), and a timeout. Policy can auto-approve low-risk writes (draft a ticket in a sandbox project) and require a human for money, access, or production mutations. Shadow mode should simulate writes even when confirmation is on, so evals can score intent without mutating prod.

A trace is the unit of eval. Store the user message, the tools offered (by version), each call’s name and a hash of arguments, latency, error, and the final answer. Redact secrets. From those traces you can score: right tool selected, arguments valid, task completed, extra calls, unauthorized attempts.

Build a golden set of 50+ multi-step jobs with expected tool sequences — not only expected prose. CI should fail when tool-selection accuracy or task success drops. Provider-specific function-calling formats change; keep a thin adapter so Anthropic, OpenAI, Google, or open-source tool use can be swapped without rewriting executors. No token markup: the client pays the model provider.

• Registry: name, schema version, owner, read/write, required scopes
• Gateway: allowlist, validate, bind user identity, deny unknown tools
• Writes: idempotency key, confirmation, no blind retries
• Evals: golden sequences, traces, shadow mode before cutover

The usual failures are schema drift, timeout-plus-replay duplicates, prompt injection via tool results (treat returned text as data), and loops that re-call a failing tool. Cap retries per tool per trace. If a tool fails twice with the same error, the model must stop and tell the user.

Do not give the agent a generic `run_sql` or `run_shell` to “move faster.” If you need Text-to-SQL or computer use, those are separate designs with read-only users, sandboxes, and their own evals. Function calling done well is boring: few tools, strict schemas, loud traces.

Step-by-step build

1. 1

   Inventory tools and side effects

   List the actions the agent must take, who owns each API, and whether the call is read, write, or irreversible. Start with a small allowlist that covers real tasks; a 40-tool catalog on day one is how you get an agent that never picks the right one.

2. 2

   Write JSON Schema function definitions

   For each tool, publish a name, description, and JSON Schema for arguments with types, enums, and required fields. Include error shapes. Version the schema. The description is for the model; the schema is for validation — both must be accurate or the model will hallucinate arguments.

3. 3

   Build the allowlist gateway and executors

   The model returns a function name and JSON arguments. The gateway checks the name against the registry, validates args, resolves the user, and checks scopes. Only then does the executor run. Unknown names and extra fields fail closed.

4. 4

   Add retries, idempotency, and write confirmation

   Retry idempotent reads with backoff and jitter. For writes, require an idempotency key and a confirmation step (UI button or policy) before execution. On timeout, look up the key; do not blindly replay.

5. 5

   Constrain the agent loop

   Cap tool rounds (typically 4–8), wall-clock time, and tokens. After each result, the model either calls another allowed tool, answers, or refuses. Kill runaway loops. Do not let the model chain into a tool that was not in the original allowlist for that task.

6. 6

   Trace, evaluate, and shadow

   Emit a span per model call and per tool. Score task success on a golden set of multi-step jobs. Run the agent in shadow mode where writes are simulated. Cut over when success rate, confirmation hit rate, and unauthorized-attempt rate meet the bar.

Common pitfalls

The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.

Frequently asked questions