Structured Output Generation

A secure RAG prompt controls which documents enter the model. Structured output controls what leaves it: typed data that downstream code can parse, validate, and route without fragile string cleanup.

Structured output generation turns free-form model text into typed records that software can validate and route. Schemas, constrained decoding, and explicit recovery paths matter whenever LLM output feeds another system.

A CI triage pipeline that processes build logs automatically needs typed fields. A developer writes, "Run RUN-842 failed in unit-tests after auth-cache timed out." Your pipeline needs to extract the run ID and failure type so it can route the incident to the right workflow and look up the run in CI.

You ask an LLM to extract this information and return JSON. If the model replies with, "Extracted data: { 'run_id': 'RUN-842', 'failure_type': 'test_failure' }", your parser crashes because of the preamble text. That single failure blocks an entire automation pipeline.

When LLM output feeds software, it must be machine-parseable and validated. A missing comma, an unexpected field name, or a polite introduction can break downstream processing. Structured output generation is the set of techniques that make model outputs conform to a defined schema, replacing fragile string cleanup.

The practical range runs from best-effort JSON mode to token-level constraint enforcement via grammar-guided decoding ^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}, open-source runtimes like Outlines and SGLang ^{[2]Reference 2Outlines Documentationhttps://dottxt-ai.github.io/outlines/latest/[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104[4]Reference 4SGLang Structured Outputs Documentationhttps://docs.sglang.io/docs/advanced_features/structured_outputs}, and hosted patterns using OpenAI's Structured Outputs feature.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

The illustration above shows how a grammar engine filters the model's next-token probabilities. Invalid tokens (like a word when a number is required) are masked to negative infinity, so the sampler can only pick grammar-compliant tokens.

When does structured output matter most?

Answer

Use it whenever model output feeds software: parsers, queues, databases, tool calls, eval pipelines, or dashboards. If a human is reading the answer directly, free-form text may be better. If code consumes the answer, the model needs a contract.

Why "please return JSON" isn't enough

Before looking at solutions, watch how naive prompting fails. Suppose you send this prompt to a chat model:

Three real failures look like this:

None of these are model "hallucinations" in the usual sense. The model followed a weak natural-language request. Your request was underspecified. You asked for JSON, but you didn't enforce JSON.

Move from asking to enforcing. The enforcement tiers run from weakest to strongest.

Why does "return JSON" fail as an interface contract?

Answer

It's a request written in natural language, not a parser-enforced contract. The model may add preamble text, wrap JSON in Markdown, use single quotes, omit fields, or choose a shape your application doesn't accept.

Contract choices and enforcement mechanisms

Engineers need to choose both the contract exposed to application code and the mechanism that enforces it. These choices aren't a strict ladder: a hosted structured-output API may enforce a JSON Schema using constrained decoding internally, while a self-hosted runtime may expose grammar-guided decoding directly. Production machine-consumed output needs a checked contract, regardless of where enforcement runs.

To see how these differ in practice, imagine the same CI failure log. Each tier changes where enforcement happens:

Prompt-based: The model might return valid JSON. It might return a monologue. You need a regex to clean the output, then a JSON parser, then a schema validator, then a retry loop. This is best-effort formatting, not enforcement.

JSON mode: The API constrains successful completions to valid JSON syntax. You still need to explicitly tell the model to produce JSON, detect incomplete outputs, and validate the shape yourself. JSON mode could still return {"foo": "bar"} when you wanted {"run_id": "string", "failure_type": "string"}. JSON mode is syntax enforcement, not schema enforcement.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Structured-output APIs: The API takes a JSON Schema or Pydantic model and, when the request completes without refusal or truncation, returns output that matches supported schema features. The fields, types, and required properties are enforced within that supported subset.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Grammar-guided decoding: A runtime prevents the sampler from picking tokens that would break its compiled grammar. At every step, invalid tokens have their probabilities set to zero. This describes an enforcement mechanism inside generation, not a stronger semantic guarantee: a grammar-valid payload can still contain wrong values.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}

What is the difference between JSON mode and structured outputs?

Answer

JSON mode aims for valid JSON syntax. Structured outputs aim for a supported schema: required fields, object shapes, field types, and closed objects when configured. You still need to handle refusals, truncation, unsupported schema features, and semantic validation.

This executable comparison demonstrates the remaining gap after JSON syntax succeeds: an object can parse correctly and still fail its application schema.

The parser-gate analogy

Source code moving through a compiler parser with no grammar might be a valid statement, or it might be text that only looks close. This is prompt-based generation: you ask the model to "please produce JSON," but nothing stops it from outputting a monologue instead.

Grammar-guided decoding is like a parser gate during compilation. The grammar prevents the output from taking an invalid syntax path. In the context of an LLM, the gate is a finite state machine (FSM, a computational model that tracks transitions between allowed states) that monitors generation. Once the parser state expects a non-negative integer for retry_count, the FSM blocks letter tokens and any other token prefix that can't extend a legal integer. The decoder can't select a letter token that violates the grammar, the same way a compiler parser rejects a malformed statement.

What does grammar-guided decoding enforce?

Answer

It enforces structure during sampling by allowing only tokens that keep the output inside the grammar. It doesn't prove that extracted values are true, current, safe, or useful.

How grammar-guided decoding works

To understand how runtimes enforce structural compliance, start by examining the token generation process itself. By intercepting the token sampling phase, the runtime restricts selection to tokens that match the desired schema. Grammar constraints act as a filter during token generation:

Grammar-guided enforcement operates at the token sampling level. It transforms a schema or grammar into constraints that modify the model's output probabilities during generation.

From schema to logit masking

1. Schema Compilation: For regular languages such as Regex, the runtime can compile the allowed strings into a Deterministic Finite Automaton (DFA) (a state machine where the next state is uniquely determined by the current state and input symbol). For general JSON schemas, especially nested or recursive ones, many systems instead compile to a context-free grammar or another stack-aware parser representation.
   • Example Schema: {"type": "object", "properties": {"age": {"type": "integer", "minimum": 0}}}
   • Simplified regular-expression equivalent: \{"age":\s*(0|[1-9][0-9]*)\}
2. State Tracking: As the model generates tokens, the engine tracks the current DFA state or parser stack for the active grammar.
3. Logit Masking: At each step, the engine identifies which tokens are valid transitions.
   • Scenario: The model has generated {"age": .
   • Valid Next Characters / Bytes: 1, 2, 3, ... 9, 0, (space).
   • Invalid Next Characters / Bytes: ", a, b, {, [, etc.
   • Action: The runtime maps those allowed prefixes back to token IDs, then sets the logits (unnormalized probabilities) of all invalid token IDs to $-\infty$. When softmax is applied, their probability becomes 0.

The toy code below shows the core masking idea without requiring a model checkpoint. The grammar state says that only { is legal as the next token, so every other candidate is assigned negative infinity before sampling:

This enforces structural adherence to the allowed grammar as long as generation can continue normally. In production, you still need to handle refusals, truncation, and semantic mistakes in the returned values.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Why can a schema-valid result still be wrong?

Answer

The grammar can require { "run_id": "...", "status": "..." }, but it can't check your CI database, job logs, or deployment policy. It controls format; your application still validates meaning.

The tokenizer alignment problem

The clean DFA story above hides the hardest systems detail. Grammars are usually written over characters or bytes, but LLMs don't sample characters. They sample subword tokens. That tokenization boundary means the runtime must answer a much harder question than "is } valid here?" It must answer "which of the 100,000+ vocabulary items could extend some valid character prefix from this state?" ^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}

That token-to-grammar alignment step is where a lot of the real engineering work lives. Outlines precomputes an index used during guided generation, while SGLang's paper explicitly calls out compressed finite state machines for faster structured decoding.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104} If an implementation does this naively, masking can become a noticeable part of per-token latency.

Hosted APIs can hide some of that machinery. OpenAI's current docs include a general first-schema latency caveat for Structured Outputs: the first request with a schema can add processing latency, while later requests with the same schema avoid that extra work. The same page also notes a fine-tuned-model-specific version of that caveat for response_format, where other models don't have that particular limitation.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Why does tokenizer alignment make constrained decoding harder than parsing characters?

Answer

Schemas are usually written over characters or bytes, but models sample vocabulary tokens. The runtime has to map "which strings are valid next" into "which token IDs can still lead to a valid string" at every generation step.

Regex vs. context-free grammars

When working with flat data structures, regular expressions (Regex) compiled to DFAs work well. But what if your JSON object contains object lists or several nesting levels?

Regular-language constraints can't represent arbitrarily nested structures like general JSON. To enforce nested or recursive schemas, engines need a Context-Free Grammar (CFG) or another stack-aware parser representation. Such representations track open brackets [ and { until matching closing brackets ] and }. Modern libraries handle translation automatically; benchmark nested schemas because parser state and output size can affect latency.

When is a regular expression enough, and when do you need a CFG?

Answer

Regex is enough for flat patterns like a run ID, a bounded enum, or one shallow object shape. Nested JSON, arrays of objects, recursive trees, and balanced brackets need a stack-aware grammar or parser representation.

Open-source engines: Outlines, llama.cpp, SGLang, and XGrammar

While hosted Structured Outputs APIs are convenient, open-source runtimes give you direct control over constrained decoding for self-hosted models. For teams deploying open-weight models, managed API features may not fit. Instead, they use inference engines or libraries that support constrained generation natively and can cache compiled grammars or shared prefixes inside the serving stack.

Outlines

Outlines ^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702[2]Reference 2Outlines Documentationhttps://dottxt-ai.github.io/outlines/latest/} provides a high-level Python interface for structured generation. In current releases, you build an Outlines model once, then create a reusable Generator for a hot target type. That keeps the model wrapper, tokenizer, and schema-bound generation machinery off the per-request setup path.

This is an integration example, not a no-dependency local script. It requires outlines, transformers, pydantic, a compatible model backend, and enough local compute to load the chosen model.

Enums and required fields can be enforced during decoding, but business rules still belong in post-validation. For example, if level must be between 1 and 100, validate that in your application even if the grammar already narrows the shape.

What should you build once at startup in an Outlines-style integration?

Answer

Build the model wrapper, tokenizer/backend connection, and reusable generators for hot schemas once where possible. Keep per-request work focused on the prompt so schema compilation and tokenizer setup don't sit on every request's critical path.

llama.cpp

llama.cpp exposes grammar constraints directly through GBNF and can also convert a subset of JSON Schema into grammars for its server and CLI flows.^{[6]Reference 6llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp[7]Reference 7llama.cpp Grammars Documentationhttps://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md} One subtle but important detail from its docs is that the schema is used to constrain decoding, not automatically to teach the model what the fields mean. For plain structured generation, you still want prompt instructions that explain the task and the semantics of the fields you're asking for.^{[7]Reference 7llama.cpp Grammars Documentationhttps://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md}

Why does a constrained decoder still need task instructions?

Answer

The grammar can say a field named status must be a string. It doesn't teach whether "failed" or "passed" is the correct value for RUN-842. The prompt explains the task semantics; the grammar enforces the output shape.

SGLang

SGLang (Structured Generation Language) ^{[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104[4]Reference 4SGLang Structured Outputs Documentationhttps://docs.sglang.io/docs/advanced_features/structured_outputs} goes further by optimizing the runtime for structured workloads. Its paper describes two separate ideas that are easy to conflate:

1. RadixAttention reuses KV cache for shared token prefixes across requests.
2. Compressed finite state machines speed up structured decoding itself.^{[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104}

Under the hood, SGLang stores token-prefix mappings in a radix tree and reuses previously computed KV cache when a new request shares a prompt prefix.^{[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104} That helps Time To First Token (TTFT) when many requests reuse the same system prompt, few-shot examples, or schema instructions. It's a prefix-caching optimization, not a grammar-state cache.

SGLang also provides a domain-specific language (DSL, a specialized syntax for a particular application domain) for interleaving Python control flow with LLM generation. Its structured-output documentation exposes JSON Schema, regex, and grammar constraints; its runtime design separately addresses cache reuse.^{[4]Reference 4SGLang Structured Outputs Documentationhttps://docs.sglang.io/docs/advanced_features/structured_outputs[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104}

Why should you separate prefix caching from grammar-state optimization?

Answer

Prefix caching saves prefill work when prompts share text. Grammar-state optimization makes valid-token computation cheaper during decoding. Both reduce latency, but they act on different parts of the serving path.

XGrammar

XGrammar ^{[8]Reference 8XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelshttps://arxiv.org/abs/2411.15100} is a grammar engine built specifically to make context-free-grammar decoding cheap enough for production serving. It attacks the tokenizer-alignment cost from the previous section head on. It splits the vocabulary into context-independent tokens, whose validity can be precomputed regardless of the parser stack, and context-dependent tokens, which must be checked at runtime against the current stack. A persistent stack and overlap with GPU execution then shrink the per-token mask cost further.^{[8]Reference 8XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelshttps://arxiv.org/abs/2411.15100}

The paper reports up to 100x faster grammar processing than evaluated prior approaches and near-zero structured-generation overhead in its end-to-end serving experiments.^{[8]Reference 8XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelshttps://arxiv.org/abs/2411.15100} Serving stacks can expose engines such as XGrammar behind a higher-level structured-output interface; verify which backend and benchmark settings your deployment uses.

How does XGrammar make grammar-guided decoding cheaper?

Answer

It precomputes validity for context-independent tokens, checks only context-dependent tokens at runtime against a persistent parser stack, and overlaps grammar computation with GPU work. That keeps per-token masking close to free in end-to-end serving.

Function calling vs. structured outputs

Both techniques constrain model outputs, but they solve different orchestration problems in an AI system. It's common to blur them together because both can involve JSON schemas. Some APIs also allow strict tool-argument schemas, but the core distinction still holds: function calling is about selecting actions, while structured outputs are about returning data in a fixed contract.

When should you choose function calling instead of structured outputs?

Answer

Choose function calling when the application needs an action request and structured arguments, whether tool choice is automatic or forced. Choose structured outputs when every response should be a terminal data record for your code to parse.

When to use function calling

Use function calling when the model is an agent that needs to interact with the world. In a typical tool loop, the application exposes allowed tools and may let the model choose one or require a specific call. The application executes an accepted call, then returns its result to the model in a subsequent turn. This allows the agent to formulate a final user-facing response based on fetched data.

This example defines a tool for an agent to fetch CI run status. If tool choice is left automatic, the model can either call the function with structured arguments or return a normal text response:

If your provider supports strict tool schemas, enable them. That improves argument reliability, but the control flow is still tool invocation rather than terminal data extraction. On OpenAI, manual strict tool schemas should set strict: true, list every parameter in required, and close objects with additionalProperties: false. parallel_tool_calls=false is useful when your application expects zero or one call, but that's an orchestration choice, not the definition of strictness.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Why does a strict tool schema not replace authorization checks?

Answer

It can make the tool arguments well shaped. It doesn't know whether the user can access the repository, whether the action is allowed, or whether the tool should run. Authorization, limits, and approval gates still belong in application code.

When to use structured outputs

Use structured outputs when you need to extract data or create a reliable interface between the LLM and your code. Unlike the multi-turn loop of function calling, structured outputs are typically terminal, single-turn actions used purely for data formatting and strict schema adherence.

This example defines a Pydantic model for a run-status update and passes it directly to OpenAI's parsing helper. The SDK handles the JSON schema conversion for you, and output_parsed gives you a typed object back when the model succeeds.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} This snippet requires the OpenAI Python SDK and an OPENAI_API_KEY.

One provider-specific gotcha: hosted structured-output APIs usually implement a subset of JSON Schema, not the full spec. On OpenAI, that means root-level anyOf isn't allowed, every field must be required, and every object must opt into closed-world generation with additionalProperties: false.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} Treat schema design as part of your API integration, not as a prompt-writing detail alone.

What does additionalProperties: false protect you from?

Answer

It closes the object shape so the model can't add surprise fields like admin_note, debug_prompt, or raw_user_text. That makes downstream code safer because unknown keys are rejected instead of silently accepted.

The next example keeps these gates separate. Pydantic checks shape and enum values; application code checks whether the authenticated engineer can access the run's repository.

Production patterns

Moving from a prototype that occasionally outputs valid JSON to a production system that processes thousands of requests reliably requires defensive engineering. These patterns address the most common failure modes and lifecycle challenges associated with structured output generation.

1. Preserve useful intermediate evidence for hard tasks

A subtle but important pattern: strict structured output can hurt task quality on some hard problems when the schema is too tight. A controlled study found measurable reasoning-accuracy reductions under format restrictions on evaluated tasks such as GSM8K, with outcomes affected by format and field ordering.^{[9]Reference 9Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelshttps://arxiv.org/abs/2408.02442}

Don't respond by storing unrestricted chain-of-thought. Instead, design product-visible intermediate fields that can be checked: cited evidence, extracted quantities, calculation steps needed by a tutor, or a reason code used by a reviewer. If a workflow genuinely needs those fields, place them before the final decision and validate them independently.

This isn't about recovering a hidden "true" chain of thought. The contract exposes evidence that a product or reviewer can inspect when the final decision is wrong.

Why are visible evidence fields not the same as asking for hidden chain of thought?

Answer

You are asking for product-visible fields that your application can check: evidence quote, calculation output, reason code, or final decision. The model's private reasoning is not part of the contract.

2. Explicit intermediate fields for complex extraction

A common pitfall is forcing a difficult decision into a final field with no checkable support. For extraction, tutoring, or multi-step workflows, explicit evidence or bounded calculation fields can make the result easier to validate before code acts on it.

The schema below follows the same pattern as OpenAI's math-tutor examples: return a bounded list of explicit steps plus the final answer.

When should you add intermediate fields to a schema?

Answer

Add them when they make the task easier to validate or debug, such as extracted quote, evidence span, confidence reason, or calculation step. Don't add them as generic verbose text that nobody checks.

3. Handling schema evolution

Schemas change. If producers and consumers disagree about a payload version, a rollout can break downstream processing even when each individual response is valid JSON.

Production tip: Version your schemas in application code. Choose the validator before sending the request, then attach the corresponding version to the validated record or require a fixed version literal and verify it. Don't let the model choose which contract it claims to satisfy.

Why should a schema version travel with the validated record?

Answer

They let the backend route old and new records through the correct validator during rollouts. The application should assign or verify the version, because a model-generated version label isn't proof that the payload matches that contract.

4. Recover by failure class, without weakening the contract

Even with structured outputs, failures can happen. The important distinction is between recoverable failures (for example, truncation from max_output_tokens) and policy outcomes such as refusals or content filtering. Don't treat a refusal as an ordinary parse failure and then retry with a looser mode.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Truncation doesn't justify switching from a schema-enforced response to JSON mode: the missing content is still missing and the fallback loses schema enforcement. Prefer a bounded retry with more output budget, a smaller contract, or chunked input. The fake client below mirrors response states so you can unit-test that control flow without an API call:

Which structured-output failures are retryable?

Answer

Recoverable transport or length failures can use a bounded schema-preserving retry, such as increasing max_output_tokens after truncation or chunking the task. Refusals and content-filter stops are policy outcomes, so they should route to policy handling, not a looser parser.

5. Flatten nested structures

Schemas with many nesting levels or recursive shapes can increase grammar state, output length, and debugging complexity. Some hosted APIs support recursive schemas, but support doesn't establish acceptable latency for your workload.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} Benchmark nested contracts on the target runtime.

Production tip: Keep nesting as shallow as your interface allows, bound recursive outputs, and benchmark the exact schema on your target runtime. If you need tree-shaped data, a flat list of nodes with parent_id references is often easier to generate, validate, and evolve than a recursive JSON object.

Why is a flat list with parent_id often better than recursive JSON?

Answer

It reduces grammar state, output length, validation complexity, and schema-evolution risk. You can still reconstruct the tree in code after parsing the flatter contract.

Performance considerations

Grammar-guided decoding performs valid-token work during generation, although optimized engines may hide or greatly reduce observable overhead for particular workloads. Schema compliance doesn't make latency irrelevant. Measure on the model, schema, batch shape, and runtime you plan to ship.

Where the latency comes from

The latency impact depends on the schema, tokenizer, and runtime design. The main cost centers are:

Interviewers often ask about TTFT versus TPOT for this reason. Compilation, hosted schema preprocessing, and large prompt prefixes mostly affect TTFT. Token masking affects TPOT. Systems such as Outlines, SGLang, and XGrammar focus on reducing those costs with precomputation, token categorization, and cache reuse rather than ignoring the cost.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104[8]Reference 8XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelshttps://arxiv.org/abs/2411.15100[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Which constrained-generation costs affect TTFT versus TPOT?

Answer

Schema compilation, provider preprocessing, and long prompt prefixes usually affect Time To First Token. Per-token valid-token masking affects Time Per Output Token. Measure both because one chart can hide the bottleneck.

The quality-compliance tradeoff

Constraining the output space can affect generation quality because a grammar eliminates paths outside the contract. The effect depends on task, model, and schema; format-restriction studies show it can be measurable on reasoning tasks.^{[9]Reference 9Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelshttps://arxiv.org/abs/2408.02442}

Production tip: Keep your schemas semantically permissive, but structurally stable. Prefer a stable field set with nullable values or bounded enums over a maze of branching object variants. If your provider uses strict mode, supported-schema limits and closed-object requirements become part of the interface contract.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

What is the practical schema-design tradeoff?

Answer

Make structure strict enough for code to trust, but leave values flexible enough for the model to express real cases. Overly tight enums and deep branching schemas can produce brittle or meaningless outputs.

Batched constrained generation

When processing multiple requests with the same schema (common in data extraction pipelines), the FSM can be compiled once and reused across requests. This avoids repeated per-request compilation work when the runtime supports that reuse.

Current Outlines calls support the same batching pattern. Build the model wrapper and schema-bound generator once at startup, then reuse that generator across prompts:

SGLang takes this further with prefix-aware KV cache reuse. If multiple requests share the same prompt prefix, the runtime can reuse prefetched state instead of rebuilding it from scratch.^{[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104}

When does batching or cache reuse pay off for structured generation?

Answer

It pays off when many requests share the same model, schema, prompt prefix, or target type. One-off schemas get less benefit because compilation and prefill work can't be amortized across traffic.

Debugging common failures

Even with structured outputs, things go wrong. The difference between a prototype and a production system is knowing what failure looks like, why it happens, and how to fix it. This table turns common misconceptions into a debugging guide.

"The model returned valid JSON, so the data must be correct"

• Symptom: Your pipeline parses the output successfully, but the values are nonsense. A CI status reads "passed" for a run that failed. A run ID doesn't exist in your build system.

• Cause: Structured outputs enforce format, not accuracy. The model can produce valid JSON with correct types while the values are still fabricated. A {"status": "passed"} value satisfies the schema but is wrong for a failed run.

• Fix: Add application-layer semantic validation. Check that run IDs exist in CI, that status values match source-of-truth logs, and that enum values match your known set. Use Pydantic validators or plain Python assertions after parsing.

Never put authorization decisions in model-owned schema fields

Wrong CI facts are one class of failure. A higher-risk class is schema-valid false authority: boolean or enum fields that look like grants.

Examples a constrained decoder will happily emit if they appear in the schema:

• authorized: true
• requires_approval: false
• risk_tier: "low"
• is_admin: true
• approved: true

Those values are still model claims. They are not session identity, not a stored approval row, and not policy. Put only descriptive fields in the model contract (what happened, what the model proposes). Resolve grants from trusted storage: session actor, RBAC, approval records with action hashes, environment flags your host owns.

What is the difference between format validation and semantic validation?

Answer

Format validation checks shape: JSON parses, fields exist, types match, and enums are legal. Semantic validation checks truth and operational rules: the run exists, the status matches CI, the user can access the repo, and the action is allowed. Self-attested flags such as authorized: true are still format-only; grants come from trusted host state.

"The Markdown wrapper trap"

• Symptom: Your JSON parser throws a JSONDecodeError even though the output looks like JSON at a glance.

• Cause: The model wrapped the JSON in triple backticks with a json label, or added a preamble like "Here is the result:". When you feed the raw output into json.loads(), the extra characters break parsing.

• Fix: Treat unparseable or schema-invalid text as a contract failure. For a legacy prompt-only integration, make a bounded retry through a stronger interface or send the item to review; don't silently slice arbitrary text between braces and trust it as the record.

Use structured outputs or a grammar-guided runtime when the system must produce the contract directly rather than repair free-form text.

Why should a machine-consumed pipeline reject Markdown wrappers?

Answer

Text cleanup adds ambiguous edge cases and can hide contract drift. A schema-aware API or grammar-guided runtime produces a checked record or a failure state that application code can handle explicitly.

"A refusal isn't a parse error"

• Symptom: Your fallback cascade retries a refused request with a looser mode, and the model still refuses. You've spent extra tokens and latency for no gain.

• Cause: A refusal or content-filter stop is a policy outcome, not a decoding bug. The model (or the safety layer) has decided not to answer. Loosening the schema doesn't change that decision.

• Fix: Surface the refusal to your application layer. Route it to a human reviewer, change the input, or return a polite error to the user. Don't treat refusals as retryable parse failures.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Why should refusals not fall back to looser generation?

Answer

A refusal is a policy result, not a grammar failure. Retrying with fewer constraints can turn a safety decision into a bypass attempt and wastes latency without fixing the underlying request.

"Grammar-guided decoding is always slow"

• Symptom: You've heard constrained decoding adds overhead and assume it's too slow for your use case.

• Cause: Naive implementations can be slow, but optimized runtimes reduce the cost a lot. The overhead depends on tokenizer alignment, grammar complexity, and whether you get cache hits.

• Fix: Benchmark before deciding. The right question isn't "is there overhead?" It's "where is the overhead, and can I amortize it?" If you process many requests with the same schema, compilation cost may be reused. Compare hosted schema APIs and optimized self-hosted runtimes on your latency and compliance targets; provider-managed enforcement hides implementation work but doesn't guarantee lower latency.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702[3]Reference 3SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

What benchmark should you run before rejecting constrained decoding?

Answer

Measure the exact schema, model, tokenizer, batch shape, and traffic pattern you plan to use. Compare TTFT, TPOT, parse failure rate, retry count, and semantic validation failures against your baseline.

"JSON mode and structured outputs are the same thing"

• Symptom: You enabled JSON mode and assumed the output would match your schema. It returned {"foo": "bar"} when you expected {"name": "string", "age": "integer"}.

• Cause: JSON mode enforces valid JSON syntax on successful completion, but not schema compliance. The model might return any valid JSON object.

• Fix: Use structured outputs or grammar-guided decoding when you need schema adherence. Use JSON mode only when you need syntactic validity and plan to validate the shape yourself.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

What should still happen after JSON mode?

Answer

Parse the JSON, validate the schema, reject unknown fields, check business rules, and handle truncation or refusal states. JSON mode is not the end of the pipeline.

"I should use structured outputs for every LLM call"

• Symptom: You're wrapping every prompt in a Pydantic model, even for creative writing or open-ended Q&A.

• Cause: Over-application of a useful technique. Structured outputs shine when the output feeds into code (APIs, databases, downstream processing). For user-facing text responses, free-form generation is often better.

• Fix: Use structured outputs when the consumer is code. Use free-form generation when the consumer is a human. Forcing unnecessary structure wastes tokens on syntax characters and may constrain the model's expressiveness.

What is the simplest decision rule for using structured outputs?

Answer

If code consumes the answer, use a structured contract. If a person consumes the answer and you don't need machine routing, let the model write normally.

Practice: build an incident-digest parser

Here's a concrete exercise to test your understanding. Try it before looking at the solution sketch.

Task: You receive a 500-word incident digest about engineering services. Build a tool that extracts every mentioned service and classifies its operational status as healthy, degraded, outage, or unknown.

Requirements

1. Define a Pydantic model with two fields: service (str) and status (Literal["healthy", "degraded", "outage", "unknown"]).
2. Use structured outputs or grammar-guided decoding to enforce the schema.
3. Handle the case where the model returns no mentions (return an empty list, not a null).
4. Add a post-validation step that checks the service name against a known service catalog (e.g., auth-api, vector-indexer, billing-export). Flag unknown services for review.

Input example

"auth-api recovered after elevated 5xx errors. vector-indexer remains degraded. report-worker has no known impact."

Expected output shape

Solution sketch

Click to expand solution sketch

solution-sketch.py

1from typing import Literal 2from pydantic import BaseModel 3 4class Mention(BaseModel): 5 service: str 6 status: Literal["healthy", "degraded", "outage", "unknown"] 7 8class IncidentDigest(BaseModel): 9 mentions: list[Mention] 10 11KNOWN_SERVICES = {"auth-api", "vector-indexer", "billing-export"} 12 13def unknown_services(digest: IncidentDigest) -> list[str]: 14 return [ 15 mention.service 16 for mention in digest.mentions 17 if mention.service not in KNOWN_SERVICES 18 ] 19 20digest = IncidentDigest.model_validate({ 21 "mentions": [ 22 {"service": "auth-api", "status": "healthy"}, 23 {"service": "vector-indexer", "status": "degraded"}, 24 {"service": "report-worker", "status": "unknown"}, 25 ] 26}) 27 28empty = IncidentDigest(mentions=[]) 29print("mentions:", len(digest.mentions)) 30print("unknown services:", unknown_services(digest)) 31print("empty mentions:", empty.mentions)

Output

1mentions: 3 2unknown services: ['report-worker'] 3empty mentions: []

Key design decisions:

• Require a list, not a nullable or defaulted field: mentions: list[Mention] makes mentions required in generated JSON Schema and represents no matches as {"mentions": []}. That shape is directly compatible with OpenAI strict mode, which requires every field.
• Post-validate status: The schema restricts the value to one of the three literals, but it can't prove the service status is factually correct. Add a second-pass check for high-stakes classifications.
• Handle truncation: If the digest is long and the output hits max_output_tokens, use a bounded schema-preserving retry, chunk the input, or explicitly version a smaller contract.

Why does the incident-digest parser flag report-worker for review?

Answer

The phrase "no known impact" doesn't prove health, so the parser preserves uncertainty with status: "unknown". The schema can represent that value, but it can't know whether report-worker is in the service catalog. The known-service allowlist is a separate semantic check after structured parsing.

Where structured output leads

Schema-constrained generation can enforce structure during decoding. JSON mode enforces JSON syntax only, while prompt-only formatting remains best effort. The right enforcement tier, failure diagnosis, and fallback cascade determine whether truncation, refusals, and malformed output become recoverable errors or wasted tokens.