Structured Generation Systems
Structured generation for LLMs: constrained decoding, JSON schema, repair loops, validators, and binding model output to tool arguments for production agents.
- Read time
- 16 min
- Level
- Intermediate
- Updated
- 2026-08-25
- Components
- Schema compiler · Constrained decoder · Repair loop
A structured generation system forces an LLM to emit data that matches a schema — via constrained decoding, JSON schema validation, and a bounded repair loop — so tool arguments and downstream systems receive types, not best-effort prose.
Key takeaways
- 01
Constrained decoding (grammar or schema-guided sampling) makes invalid tokens impossible; prompting for JSON does not.
- 02
JSON mode is not a schema: it can still emit the wrong keys, types, and enums. Compile the real JSON schema into the decoder or a validator.
- 03
Keep a repair loop with a hard retry cap and a deterministic validator; infinite "fix your JSON" is a latency and cost bug.
- 04
Tool-arg binding should use the same schema path as user-visible structured output, or agents and APIs will drift.
- 05
Skip constrained decoding when the product is prose. Skip unconstrained generation when a computer will parse the result.
Structured generation and constrained decoding
Structured generation is the architecture that makes LLM output machine-readable on purpose. Constrained decoding compiles a JSON schema or grammar into token-level constraints so the model cannot emit keys, types, or punctuation the schema forbids. A JSON schema LLM path still needs a validator after decode — constraints can be incomplete, and providers differ in how strictly they enforce them.
The system is a pipeline: schema in, compiled constraint, model call, validate, repair or reject, then bind to a typed object (or to tool arguments). If any stage is "ask the model nicely," you do not have structured generation; you have a parser with hope.
- Schema compiler: JSON Schema, grammar, or typed model
- Constrained decoder or provider structured-output API
- Validator: types, enums, ranges, refs
- Repair loop and binder into tools or DB writes
JSON schema as the contract
The schema is the product contract. Keep it versioned, reviewed, and shared between the decoder, the validator, and the code that consumes the object. Prefer explicit enums, min/max, required fields, and additionalProperties: false. Loose object blobs recreate the original problem inside a JSON wrapper.
Generate the schema from the same types the application uses (OpenAPI, pydantic, protobuf, JSON Schema files in repo). Hand-maintained parallel schemas drift. When a field is added, the golden set gains a case; when a field is removed, old prompts that mention it are a regression.
Constrained decoding versus JSON mode versus repair
Constrained decoding (CFGs, regex, schema-guided sampling) is the strongest guarantee: invalid tokens are not sampled. Provider JSON mode typically only forces a JSON object, not your keys. Prompt-only JSON is the weakest and still common; it fails on long outputs, nested arrays, and the first truncated token.
A repair loop is the compatibility layer. On validator failure, send the schema errors back to the model (or a cheaper fixer) with a retry cap of one or two. After that, fail the request or fall back to a human queue. Do not stream invalid JSON to a tool. For tool args, prefer native structured tool-calling from the provider, then still validate server-side.
Binding to tools and downstream systems
Tool-arg binder applies the same schema to function calls: parse, validate, then execute. Reject unknown fields so a model cannot smuggle extra arguments. Coerce only where the schema says (string-to-int that is unambiguous); do not "fix" an enum by picking the closest word — that hides a prompt bug.
Downstream writes (CRM fields, tickets, SQL parameters) should consume the typed object, never the raw model string. That is the line between a structured generation system and a log of pretty JSON that still blows up a parser on Tuesday.
Failure modes and when not to constrain
Failures: schemas so wide they accept garbage; schemas so tight the model cannot express a valid case; truncation at max tokens mid-object; constrained decoding that dead-ends (no legal token left); repair loops that thrash; providers that claim schema support and still emit extra keys. Streaming UIs that display partial JSON as if it were final confuse operators.
Do not constrain when the output is an explanation, a draft email, or any prose a human will read as prose. Do not skip constraints when a compiler, a tool, or a database will parse the result. Also skip full grammar compilation on tiny, flat payloads if a validator-plus-one-retry already hits 100% on the golden set — complexity should follow measured invalid rates.
- JSON mode without a real schema
- Unbounded repair retries
- Truncation leaving incomplete objects
- Tool args not validated server-side
Evaluating structured generation
The primary grader is deterministic: schema validity, required fields, enum membership, and round-trip equality into your types. Add semantic cases ("refund_cents matches the cited order") with judges or exact checks. Measure first-pass valid rate, repair rate, tokens, and latency. Gate schema changes in CI — a new required field will fail old fixtures, which is the point. Trace the schema version on every span.