Function-Calling and Tool-Use Systems
Function calling architecture for LLM tool-use systems: schema registry, execution sandbox, idempotency, write confirmations, traces, and key failure modes.
- Read time
- 16 min
- Level
- Advanced
- Updated
- 2026-08-25
- Components
- Schema registry · Execution sandbox · Idempotency layer
A function-calling system is the production layer that lets an LLM select tools from a versioned schema registry, execute them in a sandbox with idempotency keys, and apply confirmation gates on writes — while every call is traced, authorized, and bounded by timeout and retry policy.
Key takeaways
- 01
Tool schemas are an API you own: version them, validate arguments server-side, and never trust the model to honor a description-only constraint.
- 02
Execute tools in a sandbox with least privilege; the model proposes calls, the runtime decides whether they run.
- 03
Idempotency keys on every side effect turn retries and double-emits into no-ops instead of duplicate payments or tickets.
- 04
Reads can be autonomous; writes, sends, and payments pause for confirmation unless a narrow, tested policy says otherwise.
- 05
Most tool-use failures are schema drift, missing authz, and unbounded retries — not the model picking the wrong function name.
Function calling architecture: registry to runtime
Function calling architecture is how an LLM tool use system actually runs in production: a registry of tools, a model that emits structured calls, a runtime that authorizes and executes them, and a trace of every attempt. "Agent tools" are not prompt text. They are typed functions with owners, scopes, side-effect class, and SLAs, exposed to one or more models through a single gateway.
Keep the model on the proposing side of a hard boundary. It may choose a tool and fill arguments; it may not hold credentials, talk to the network, or skip validation. That split is what makes the system auditable when a call goes wrong.
- Registry: name, JSON schema, side-effect class, owner, version
- Authorizer: identity, allow-list, argument constraints
- Executor: sandbox, timeout, retries, idempotency
- Trace: call, args (redacted), result, latency, cost
Schema registry and argument validation
The schema registry is the source of truth for what the model is allowed to request. Each tool publishes a JSON Schema (or equivalent) for arguments, a side-effect flag (read, write, send, payment), and a version. The prompt the model sees is compiled from the registry, so a retired tool disappears from both docs and runtime in one change.
Validate arguments on the server against the schema before execution. Enum constraints, ID formats, and max lengths in the description are hints; they are not enforcement. Reject unknown fields. Pin tool versions per agent so a schema change cannot silently alter a live workflow mid-flight.
Execution sandbox, authz, and retries
Run tools with least privilege in a sandbox: scoped credentials, network allow-lists, timeouts, and output size caps. Authz is evaluated at call time against the end user's identity, not a shared service account that can see every record. A research tool and a refund tool should not share a credential.
Retries belong to the runtime. Use bounded exponential backoff on idempotent reads; on writes, retry only with the same idempotency key. Surface structured errors to the model (not stack traces) so it can correct arguments without guessing. Kill the call at the timeout — hanging tools stall agents more often than wrong tools do.
Idempotency and confirmation gates
Every side-effecting tool takes an idempotency key derived from the conversation or task ID plus tool name plus a canonical argument hash. The executor stores the first result and returns it on duplicates. Without that, a model that "tries again" creates two tickets, two emails, or two charges.
Confirmation gates sit in front of irreversible tools. The runtime pauses, presents a human-readable diff of the proposed call, and proceeds only on approval (or on a documented auto-approve policy with a tight allow-list). Reads stay autonomous. Mixing those two classes in one un-gated toolkit is how agents send mail while "looking something up".
Failure modes and when not to use tool calling
Typical failures: the model invents a tool name, omits a required field, passes a plausible but unauthorized ID, or loops on a tool that always errors. Schema drift — prompt catalog and runtime registry out of sync — produces calls that look valid to the model and 404 at execution. Unbounded agent loops turn a flaky API into a cost incident.
Do not put a tool behind the model when a deterministic workflow already exists. If the path is "always create a ticket from this form," a rules engine is cheaper and safer. Function calling earns its complexity when the next action is genuinely conditional on unstructured input. Also skip it when you cannot sandbox the side effect at all.
- Hallucinated tools or arguments that bypass client-side hints
- Shared over-privileged credentials
- Duplicate side effects without idempotency keys
- Retry storms and missing confirmation on writes
Evaluating a tool-use system
Golden cases should cover tool selection, argument validity, refusal to call unauthorized tools, and end-state correctness after execution. Deterministic graders can check schema validity, allow-list hits, and idempotent replay. Trace every call with redacted arguments so you can replay incidents. Gate registry changes in CI: a new required field is a breaking change for every agent that uses that tool.