Structured Outputs and JSON Mode FAQ
JSON mode asks the model to speak JSON. Structured outputs and constrained decoding make invalid JSON a failed decode, not a production incident.
Structured outputs are when the model is constrained to a schema — JSON Schema, a grammar, or a tool-argument type — so the result parses or the call fails. JSON mode only asks for JSON and still breaks. Production systems validate, retry, and keep a schema registry; they do not regex the model’s feelings into a dict.
What structured outputs are
If you only need one line: the model must fill a type, not a vibe. Everything else is how you enforce that.
What are structured outputs and JSON mode?
JSON mode is a vendor setting that biases the model to emit a JSON object. Structured outputs go further: you pass a schema and the API guarantees the parse, or it errors. Constrained decoding does it at the token level with a grammar. All three exist because free-form chat models like to add markdown fences and apology sentences. For production extraction and tool calls, treat anything weaker than schema enforcement as a prototype.
Why does the model still break JSON?
Because next-token prediction is not a parser. It will truncate, quote badly, emit NaN, wrap the object in a preamble, or drop a required key when the prompt is long. JSON mode reduces fences; it does not implement your schema. The failure is normal. The defect is shipping without validation and a retry budget.
Is structured outputs better than prompting “return JSON”?
Yes. A prompt is a suggestion. Structured-output APIs and grammars change the allowed token set so a missing brace cannot appear. You still need the right schema and a semantic check — valid JSON can still invent an amount. Use the prompt for field meanings; use the constraint for shape. We will not ship a regex on markdown fences as the parser.
What is constrained decoding?
A decoder that masks tokens that would make the output leave a grammar or JSON Schema, so the sequence is valid by construction. Libraries and serving stacks (Outlines, xgrammar, vendor structured-output endpoints) implement it. It can cost latency and it can fight the model’s priors on odd schemas. It is still the right default for tool arguments and extraction. Measure p95; do not assume it is free.
Do we need a schema, or is a JSON object enough?
You need a schema: required keys, types, enums, ranges. “An object” is how you get extra keys, strings where you needed ints, and silent field renames after a model update. Share one JSON Schema (or Pydantic model) between the API, the validator, the docs, and the evals. If product cannot write the schema, the workflow is not ready for an LLM.
How does this relate to function calling?
Function calling is structured output with a name and side effects. The model emits a tool name plus arguments that must match the tool’s schema; your runtime executes it. Treat tool schemas with the same strictness as extraction schemas, plus permissions and idempotency. A chatty JSON blob that you then map to an API is a footgun. See the agents FAQ for the rest of the tool loop.
Is JSON mode enough for production?
No. Use it as a fallback on models that lack schema guarantees, then validate and retry. For OpenAI-style structured outputs, Anthropic tool use, or grammar-backed open-source serving, prefer the strict path. Production is parse rate on a golden set, not a demo that looked like JSON once. We fail CI when compliance drops, the same as any other quality gate.
How we implement the contract
The work is the registry, the validator, and the eval — the model is the untrusted speaker.
How do you repair invalid JSON?
Validate, send the parser error back once, and require a full object — not a chatty patch. If it still fails, route to a smaller repair model or to a human, and log the case into the golden set. Do not loop until the token budget dies. Some stacks skip repair entirely because the constraint cannot emit invalid JSON; then the remaining failures are refusals and truncated requests.
Should we use Pydantic or JSON Schema?
Either, as long as there is one source of truth that generates the other. Pydantic (or Zod, or a protobuf) is better for application code; JSON Schema is what many model APIs expect. Generate, do not hand-copy. We keep that artefact in your repo. Duplicate schemas in a prompt and in code are how fields diverge in week three.
Should agents write tool arguments as structured output?
Yes. Always. Free-text “please call billing with the last order” is how you get the wrong id. Strict tool schemas, constrained decoding where available, and a confirmation on writes. The agent’s plan can be text; the call cannot. This is the same rule as function calling, restated because teams still stringify args into a prompt.
What if the schema is huge?
Split it. Models lose required keys on 100-field objects. Use staged extraction (header, then lines), or retrieve the subset of fields the intent needs. Enums with hundreds of values belong in a tool lookup, not in the schema prompt. A huge schema is often a sign you wanted a form and a database, not a single completion.
Can we stream structured output?
You can stream tokens, but you cannot trust a partial object until the decoder says it is complete. UIs should wait for a valid parse or use a grammar that can emit closed prefixes. Voice and chat that speak fields as they arrive need a different, field-at-a-time schema. We will not stream a half-refund into the ledger.
How do you evaluate schema compliance?
A frozen set of inputs with expected objects. Report parse rate, required-field fill, type errors, and semantic accuracy against labels. A 100% parse rate with invented totals is a fail. Put the parse gate in CI so a model update cannot ship a fence-wrapped apology. Humans still sample the semantic misses.
Does constrained decoding cost more latency?
Often a little, sometimes a lot on huge grammars. It can also save retries, which are the real bill. Measure both. We will not invent a millisecond number. If the grammar blows your SLO, split the schema or constrain only the tool-args completion, not the user-facing prose.
Schemas in production
Production is versioned contracts, pinned models, and a failure that does not write garbage to the CRM.
What happens on schema mismatch?
The write does not happen. The run is marked failed, a retry may fire, then a human queue if it still fails. Partial writes need idempotency keys. Returning 200 with a best-effort object is how you corrupt the system of record. We fail closed on required fields. Optional fields can be null; invented required fields cannot.
How do you handle enums and strict fields?
Put closed lists in the schema. If the real world has an open tail, add an “other” with a string and a review queue — do not let the model mint a new status that the CRM rejects. Dates, currencies, and IDs get formats and checksums where you have them. Strict is a product decision; the model will happily be sloppy if you let it.
How do you log structured traces?
Store the schema version, model version, raw completion, parse result, and repair count. That is enough to reconstruct a bad object. Redact PII in the payload. A dashboard of “JSON success %” without the failed payloads is how you never fix the schema. Traces live in your store; you own them.
Who owns the schemas?
You do. They sit in your repo with the tools and the eval fixtures. We do not hide the contract in a prompt only we can edit. Handover includes the registry, the validators, and 30 days of on-call. Changing a field after week four is a versioned change your engineers can ship.
Do open-source models support this, or only OpenAI?
Open-source serving stacks support grammars and JSON Schema decoding today; quality still varies by model. OpenAI structured outputs and others’ tool-use APIs are the hosted equivalent. We are model-agnostic: we pick the enforcement the runtime actually has, then eval. A local Llama with no grammar is not “more private JSON”; it is weaker JSON.
What does handover include for structured generation?
Schema registry, validators, golden objects, CI gate, runbook, and the agent or extractor wired to them, in your environment. Thirty days on-call. You pay model tokens to the provider with no markup. If the only schema is in a chat log from the build, handover failed.
Where the contract earns its keep
Extraction, SQL, and tool loops all fail the same way when the type is optional in practice.
Is this how we extract data from PDFs?
It is the output side. You still need OCR, layout, and citations to a page or bounding box, plus a human queue for low-confidence fields. The schema is the invoice or claim object. Structured outputs without layout is how you get a fluent, ungrounded total. Use a review UI; do not auto-post every parse.
Is text-to-SQL just structured output?
SQL is a structured string with a much larger blast radius. You want a schema of allowed statements, a read-only database user, a dry-run, and golden queries — not “JSON mode but for SQL.” Constrained decoding to a dialect grammar helps. It does not replace permissions. If the user asked for SQL, say so; do not pretend a JSON object is the warehouse.
How do multi-step agents use schemas?
Each tool has its own. The plan can be loose; each hop is strict. Passing a previous object into the next tool should be typed, not dumped as prose. Multi-agent graphs fail when one worker emits a slightly different field name. The registry is how you stop that, not a longer system prompt.
When is unstructured text better?
When a person is the next reader: a draft email, a summary they will edit, a copilot paragraph. Forcing JSON on a customer-facing reply makes worse prose. Structure the object you store; keep the utterance human. Mixing both in one completion is how you get a JSON blob with a smiley in a string.
What are the common failure modes?
Prompt-only JSON, schemas copied by hand, repair loops that never page a human, valid objects with invented values, and streaming partial writes. Also: changing the schema in prod without versioning the eval. We design against those. If your current parser is a regex on the first brace, you are not in production yet, however good the demo looked.
JSON mode, structured outputs, and constrained decoding are three rungs: prompt, vendor-enforced schema, and token-level grammar. Agents that call tools depend on the third more than on a “return JSON” instruction. Retries, repair loops, and evals on schema-compliance belong in CI. ReinforcedX ships schema registries next to tool allowlists, inside your cloud. You own the schemas. Four weeks includes the contracts the model must obey, not a hope that GPT will stay in braces.
Keep reading
Ready to refine
your workflow?
Share your current process. We’ll help you identify what can be automated — and where efficiency can be reclaimed.