LearnAdvanced Agents & RetrievalStructured Output Generation

🤖HardLLM Agents & Tool Use

Structured Output Generation

Build reliable LLM interfaces with JSON mode, structured outputs, schema validation, and grammar-guided decoding.

37 min read

Learning path

Step 124 of 196 in the full curriculum

RAG Security & Access Control ReAct & Plan-and-Execute

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

A CI triage pipeline receives one short log: "Run RUN-842 failed in unit-tests after auth-cache timed out." The next component doesn't want a paragraph. It needs run_id, failure_type, and failing_job so it can look up trusted CI state.

The model replies with Extracted data: { 'run_id': 'RUN-842' }. json.loads stops at the preamble and the single quotes. Even a parseable object could omit a required field or invent a value. The real question isn't whether the model can produce JSON. It's where your system draws and checks the boundary.

A secure retrieval path decides which documents may enter the model. Structured output draws the matching output boundary: which shape may leave the model and enter application code.

You already know how decoding turns logits into one token at a time, and how function calling lets a model request an application-owned tool with structured arguments. Here, those ideas meet: structured output constrains the token choices behind a record your code can parse.

Structured output generation makes model text conform to a contract. Reliable integration still needs three distinct checks: a structural contract, explicit completion and refusal handling, and application checks against real systems. The practical range runs from best-effort JSON mode to token-level constraint enforcement with grammar-guided decoding ^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}. Hosted APIs such as OpenAI Structured Outputs, libraries such as Outlines, and serving stacks such as SGLang and vLLM all sit on that idea.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}^{[3]Reference 3Outlines Documentationhttps://dottxt-ai.github.io/outlines/latest/}^{[4]Reference 4SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104}^{[5]Reference 5SGLang Structured Outputs Documentationhttps://docs.sglang.io/docs/advanced_features/structured_outputs}^{[6]Reference 6vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionhttps://github.com/vllm-project/vllm}

At object start, candidate logits are The 0.45, opening brace 0.30, Sure 0.15, and JSON 0.10. The parser mask sets every illegal token to negative infinity, so only the opening brace remains sampleable. — Token The led at 0.45, but the object-start mask assigns illegal tokens -inf. Softmax then puts mass 1.0 on `{`, so the sampler can't emit a preamble.

The figure freezes one decoding step for the RUN-842 extractor. The mask guarantees a legal grammar path. It can't prove that RUN-842 exists or that unit-tests actually failed.

Why "please return JSON" isn't enough

Run the weak version first. Suppose you send this prompt to a chat model:

why-please-return-json-is-not-enough.py

prompt = """
Extract run_id and failure_type from this build log:
"Run RUN-842 failed in unit-tests after auth-cache timed out."
Return JSON.
"""

The request sounds clear to a person. It leaves three different failure paths open:

Failure mode	Example output	Why it breaks
Preamble text	"Here is the JSON: `{ 'run_id': 'RUN-842' }`"	The parser sees `Here` before the brace and throws
Markdown wrapper	"`json\n{ 'run_id': 'RUN-842' }\n`"	The triple backticks and newlines aren't valid JSON
Wrong shape	`{ "run_id": "RUN-842", "failure_type": null }`	Your database expects `failure_type` to be a string, not null

None of these outputs requires a special hallucination story. The model followed an underspecified natural-language request. You asked for JSON, but nothing in the interface enforced a shape.

That gives us the first design move: separate asking from enforcing. The tiers below differ in what the application can safely assume after generation.

Contract choices and enforcement mechanisms

Two decisions sit behind every machine-consumed response: which contract does application code receive, and where does enforcement run? They aren't a strict ladder. A hosted structured-output API may enforce a JSON Schema with constrained decoding hidden inside the service, while a self-hosted runtime may expose grammar-guided decoding directly. The consumer still needs a checked contract either way.

Choice	Structural contract	Typical implementation
Prompt-based ("respond in JSON")	None; best-effort formatting only	Most chat APIs
JSON mode	Valid JSON syntax on successful completion, with edge cases to handle	OpenAI JSON mode, similar API flags
Strict function calling	Schema-constrained arguments for an action request	Tool-calling APIs and agent runtimes
Structured-output API	Schema adherence for supported schemas	OpenAI Structured Outputs, schema-aware SDKs
Grammar-guided runtime	Valid path through a supplied grammar or schema translation	Outlines, SGLang, vLLM, llama.cpp

The same CI log makes the distinction concrete. First choose the guarantee the consumer needs. Then ask where each tier enforces it:

Prompt-based output is a hope, not a boundary. The model might return valid JSON, or it might return a monologue. A cleanup regex, parser, schema validator, and retry loop can reduce damage, but the interface remains best effort.

JSON mode moves one check into the API. A successful completion has valid JSON syntax, but you still need to request JSON, detect incomplete output, and validate the shape. It could return {"foo": "bar"} when the consumer expects {"run_id": "string", "failure_type": "string"}. JSON mode enforces syntax, not schema.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Structured-output APIs accept a JSON Schema or Pydantic model. When generation completes without refusal or truncation, the returned output matches the supported schema features, including required properties and field types.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Grammar-guided decoding moves enforcement into generation. The runtime masks tokens that would leave its compiled grammar, so invalid continuations receive zero probability at each step. That controls structure, not meaning. A grammar-valid payload can still contain the wrong value.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}

Parsing is only the first gate. The object below is valid JSON and still fails the application contract. The small standard-library validator makes that boundary visible; a deployed service would usually express the same rules with Pydantic or generated JSON Schema.

json-syntax-versus-schema-contract.py

import json

REQUIRED = ("run_id", "failure_type")

def validate_triage(payload: object) -> dict[str, str]:
    if not isinstance(payload, dict):
        raise TypeError("root must be an object")
    extra = sorted(set(payload) - set(REQUIRED))
    missing = [key for key in REQUIRED if key not in payload]
    if extra or missing:
        raise ValueError(f"shape mismatch extra={extra} missing={missing}")
    record: dict[str, str] = {}
    for key in REQUIRED:
        value = payload[key]
        if not isinstance(value, str):
            raise TypeError(f"{key} must be a string")
        record[key] = value
    return record

syntax_valid_but_wrong_shape = json.loads('{"foo": "bar"}')
schema_valid = json.loads('{"run_id": "RUN-842", "failure_type": "test_failure"}')

try:
    validate_triage(syntax_valid_but_wrong_shape)
except ValueError:
    print("JSON-only result: rejected by schema")

print("structured record:", validate_triage(schema_valid))

Output

JSON-only result: rejected by schema
structured record: {'run_id': 'RUN-842', 'failure_type': 'test_failure'}

How grammar-guided decoding works

The contract describes acceptable completed strings. A constrained decoder keeps that promise during generation by compiling a guide, tracking parser state, finding legal continuations, masking the rest, sampling one token, and updating the state. The model still supplies logits; the sampler simply can't select a token that leaves the accepted language.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}

Diagram showing logits + parser state, mask illegal IDs, sample token, and complete?. — logits + parser state, mask illegal IDs, sample token, and complete?.

Read the diagram from left to right. The guide is compiled once for a schema; parser state and token masking are revisited for every generated token. Only after the string is complete can application code validate the record's meaning.

From schema to logit masking

Schema compilation: For regular patterns, the runtime can compile allowed strings into a Deterministic Finite Automaton (DFA), a state machine whose next state is fixed by the current state and input symbol. Nested or recursive JSON needs a context-free grammar or another stack-aware parser representation. Each runtime supports only some schema keywords, so it must reject unsupported constraints or document how it handles them.
- Example schema: {"type": "object", "properties": {"age": {"type": "integer", "minimum": 0}}}
- One simplified accepted form, if minimum is supported: \{"age":\s*(0|[1-9][0-9]*)\}
State tracking: As the model generates tokens, the engine tracks the current DFA state or parser stack for the active grammar.
Logit masking: At each step, the engine identifies which tokens are valid transitions.
- Scenario: The model has generated {"age": .
- Valid next characters or bytes: 1, 2, 3, ... 9, 0, (space).
- Invalid next characters or bytes: ", a, b, {, [, etc.
- Action: The runtime maps those allowed prefixes back to token IDs, then sets the logits (unnormalized probabilities) of all invalid token IDs to $-\infty$ . When softmax is applied, their probability becomes 0.

The figure's object-start step is small enough to run by hand. The has the highest raw logit, but the grammar state says only { is legal. The toy mask below assigns -inf to every other candidate before softmax:

from-schema-to-logit-masking.py

import math

NEG_INF = float("-inf")

def mask_invalid_tokens(logits: dict[str, float], valid_tokens: set[str]) -> dict[str, float]:
    return {
        token: score if token in valid_tokens else NEG_INF
        for token, score in logits.items()
    }

def softmax(logits: dict[str, float]) -> dict[str, float]:
    finite = {token: score for token, score in logits.items() if score != NEG_INF}
    peak = max(finite.values())
    weights = {token: math.exp(score - peak) for token, score in finite.items()}
    total = sum(weights.values())
    return {token: (weights[token] / total if token in weights else 0.0) for token in logits}

next_token_logits = {
    "The": 0.45,
    "{": 0.30,
    "Sure": 0.15,
    "JSON": 0.10,
}

masked = mask_invalid_tokens(next_token_logits, valid_tokens={"{"} )
probs = softmax(masked)
chosen = max(probs, key=probs.get)
print("masked logits:", {token: masked[token] for token in masked})
print("softmax:", {token: round(probs[token], 3) for token in probs})
print("chosen token:", chosen)
assert chosen == "{"
assert probs["{"] == 1.0

Output

masked logits: {'The': -inf, '{': 0.3, 'Sure': -inf, 'JSON': -inf}
softmax: {'The': 0.0, '{': 1.0, 'Sure': 0.0, 'JSON': 0.0}
chosen token: {

The result enforces the grammar implemented by the runtime, assuming generation completes. It doesn't turn every JSON Schema keyword into a decoder rule, and support varies by runtime. Application code still has to parse and validate the finished record, then route refusals, truncation, and semantic mistakes.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}^{[7]Reference 7llama.cpp Grammars Documentationhttps://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md}

The tokenizer alignment problem

The DFA story leaves out the awkward boundary between grammar and model. Grammars usually speak in characters or bytes. LLMs sample subword tokens instead. At a state such as {"age": , the runtime has to ask more than "is } valid here?" It must find which of the 100,000+ vocabulary items could extend a valid character prefix from that state.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}

That token-to-grammar mapping is where much of the implementation cost appears. Outlines precomputes an index for guided generation, and SGLang's paper describes compressed finite state machines for faster structured decoding.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}^{[4]Reference 4SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104} A naive implementation can make masking a noticeable part of per-token latency.

Hosted APIs hide some of this machinery, but not every cost or limitation. OpenAI's current docs call out extra processing on the first Structured Outputs request with a schema for fine-tuned models; later requests with that schema avoid the extra latency, while other models don't have that limitation. Fine-tuned models also support a smaller JSON Schema subset, so pattern or numeric bounds can fail there even when they work on the base Structured Outputs path.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Regex vs. context-free grammars

Flat patterns such as a run ID, a closed enum, or one shallow object fit regular expressions and DFAs. Nested JSON asks a different question: regular constraints can't represent arbitrarily nested structures. Arrays of objects, recursive trees, and balanced { / [ pairs need a context-free grammar (CFG) or another stack-aware parser. Libraries usually translate the schema, but nested contracts still add parser-state work. Benchmark the shape on the runtime you plan to ship.

Once the mask is clear, the remaining choice is operational: which runtime should own this loop?

Libraries and serving stacks

Hosted Structured Outputs APIs are convenient when their supported schema subset fits. With a self-hosted model, you choose a library or serving stack that compiles a grammar and can reuse it across requests.

Outlines

If you own the Python inference process, Outlines ^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}^{[3]Reference 3Outlines Documentationhttps://dottxt-ai.github.io/outlines/latest/} gives you an interface over that loop. In Outlines v1, you wrap a model, bind an output type to a reusable Generator, receive a raw string, and validate it into your application type. Keep the wrapper and generator alive across requests. The selected backend still determines how grammar compilation and caching work.

This integration boundary needs outlines, transformers, pydantic, a compatible backend, and enough compute to load the model. It isn't a no-dependency local script.

outlines.py

from typing import Literal

import outlines
from outlines import Generator
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

class RunTriage(BaseModel):
    run_id: str
    failure_type: Literal["test_failure", "build_failure", "timeout"]
    failing_job: str

model_name = "HuggingFaceTB/SmolLM2-135M-Instruct"
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained(model_name),
    AutoTokenizer.from_pretrained(model_name),
)

triage_generator = Generator(model, RunTriage)
raw = triage_generator(
    "Extract the run ID, failure type, and failing job: "
    "Run RUN-842 failed in unit-tests after auth-cache timed out.",
    max_new_tokens=120,
)
triage = RunTriage.model_validate_json(raw)

The enum and required fields constrain the generated shape. RunTriage.model_validate_json(raw) remains a useful second boundary: application code receives one typed object and can catch backend mismatches. CI still has to verify that RUN-842 exists and that unit-tests truly failed.

llama.cpp

If you run a local or embedded model, llama.cpp exposes grammar constraints through GBNF and can convert a subset of JSON Schema into grammars for its server and CLI.^{[8]Reference 8llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}^{[7]Reference 7llama.cpp Grammars Documentationhttps://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md} The important boundary is easy to miss: a schema constrains decoding, but it doesn't teach the model what the fields mean. Plain structured generation still needs task instructions and field semantics in the prompt.^{[7]Reference 7llama.cpp Grammars Documentationhttps://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md}

When the required next token is far from the model's unconstrained preference, generation can stall or spend its remaining budget on whitespace. Prompt the task and field meanings; don't attach a schema and hope the mask will supply semantics.

Serving stacks: SGLang, vLLM, and XGrammar

At serving scale, request handlers usually call a stack that already knows how to compile and reuse the grammar instead of importing Outlines for every request.

SGLang's structured-output docs expose JSON Schema, regex, and EBNF, with XGrammar as the default grammar backend. Outlines and llguidance are launch-time alternatives. The same docs still ask you to put format instructions in the prompt because the mask blocks illegal tokens without teaching the task.^{[5]Reference 5SGLang Structured Outputs Documentationhttps://docs.sglang.io/docs/advanced_features/structured_outputs}

SGLang's paper puts two different optimizations next to each other. They solve different bottlenecks:

RadixAttention reuses KV cache for shared token prefixes across requests.
Compressed finite state machines speed up structured decoding itself.^{[4]Reference 4SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104}

Prefix caching saves prefill work when prompts share text, so it can improve Time To First Token (TTFT) for many RUN-842 requests that reuse system instructions or a schema prefix. Grammar-state work happens during decode. Both can cut latency, but they aren't the same cache.

XGrammar takes a different route through the vocabulary. It separates context-independent tokens, whose validity can be precomputed regardless of the parser stack, from context-dependent tokens, which must be checked against the current stack. A persistent stack and overlap with GPU execution then shrink per-token mask cost.^{[9]Reference 9XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelshttps://arxiv.org/abs/2411.15100}

The XGrammar paper reports up to 100x lower per-token grammar latency than the methods it compared, plus near-zero extra cost in some end-to-end serving setups. vLLM is another open-source server in this family: it serves batched generation and can attach a grammar engine at decode time.^{[6]Reference 6vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionhttps://github.com/vllm-project/vllm}^{[9]Reference 9XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelshttps://arxiv.org/abs/2411.15100} Check the backend and schema subset your service actually runs. Keyword support isn't universal.

A serving engine can enforce a schema. One orchestration question remains: is the record a terminal answer, or is it an action request?

Function calling vs. structured outputs

The same JSON Schema can appear in two very different paths. Function calling proposes an action for the host to consider; structured outputs return a terminal record for code to consume. Some APIs support strict schemas for both, but the control-flow distinction still matters.

Aspect	Function Calling	Structured Outputs
Primary Goal	Action Selection (Tool Use)	Data Extraction / Formatting
Trigger	Application may allow or require a tool call; model supplies arguments	Application requests a format for the returned record
Output	Function name + arguments	Arbitrary JSON object
Control Flow	Loop: Model -> Code -> Model	Linear: Model -> Parser -> App

When to use function calling

Function calling fits an agent that needs to affect or inspect the world. The application exposes allowed tools, lets the model choose one or requires a specific call, executes an accepted request, and sends the result back in a later turn. The agent can then write a user-facing response from fetched data instead of guessing.

The small example leaves tool choice automatic. A question about RUN-842 can become a structured call, while a greeting can remain ordinary text:

when-to-use-function-calling.py

tools = [
    {
        "type": "function",
        "name": "get_run_status",
        "description": "Fetch current CI status for a run.",
        "strict": True,
        "parameters": {
            "type": "object",
            "properties": {
                "run_id": {"type": "string"}
            },
            "required": ["run_id"],
            "additionalProperties": False
        }
    }
]

tool_schema = tools[0]
example_routes = {
    "What happened to RUN-842?": "get_run_status",
    "Hi!": "text_response",
}
print("tool:", tool_schema["name"])
print("strict:", tool_schema["strict"])
print("routes:", example_routes)

Output

tool: get_run_status
strict: True
routes: {'What happened to RUN-842?': 'get_run_status', 'Hi!': 'text_response'}

If your provider supports strict tool schemas, enable them for argument reliability. The call still invokes a tool rather than returning terminal data. On OpenAI, manual strict schemas should set strict: true, list every parameter in required, and close objects with additionalProperties: false. parallel_tool_calls=false can fit an application that expects zero or one call, but that setting controls orchestration, not strictness.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

When to use structured outputs

Structured outputs fit data extraction and other terminal interfaces between the model and code. Unlike function calling's multi-turn loop, the usual path is one request, one schema-shaped record, and then application validation.

Here a Pydantic model defines the run-status record and goes directly to OpenAI's current Responses API parsing helper. The SDK converts the model to JSON Schema, and output_parsed exposes a typed object when generation succeeds.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} Running the snippet requires the OpenAI Python SDK, OPENAI_API_KEY, and an OPENAI_STRUCTURED_MODEL value for a model that currently supports Structured Outputs.

when-to-use-structured-outputs.py

import os

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class RunStatusUpdate(BaseModel):
    run_id: str
    status: str
    failing_job: str

response = client.responses.parse(
    model=os.environ["OPENAI_STRUCTURED_MODEL"],
    input="Run RUN-842 failed in unit-tests.",
    text_format=RunStatusUpdate,
)

update = response.output_parsed

Hosted structured-output APIs usually implement a subset of JSON Schema rather than the full specification. OpenAI's current subset requires an object at the root, disallows root-level anyOf, requires every field, and requires additionalProperties: false on every object. An optional value is represented as a required field whose type includes null.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} Schema design belongs to API integration, not only prompt writing.

The next example keeps the two gates visible. A small validator checks shape and enum values; application code then checks whether the authenticated engineer can access the run's repository.

RUN-842 with status failed passes a schema gate, then a semantic gate. Project-alpha is allowed to see the run. Project-beta is rejected even though the JSON shape is valid, because ownership is not a schema property. — Schema-valid JSON is only the first gate. RUN-842 parses, then application code checks that the caller may see project-alpha. Project-beta is held even though the shape is legal.

separate-format-from-semantic-validation.py

ALLOWED_STATUS = {"queued", "passed", "failed"}
PROJECT_RUNS = {"project-alpha": {"RUN-842"}}

def validate_update(payload: object) -> dict[str, str]:
    if not isinstance(payload, dict):
        raise TypeError("root must be an object")
    required = ("run_id", "status")
    extra = sorted(set(payload) - set(required))
    missing = [key for key in required if key not in payload]
    if extra or missing:
        raise ValueError(f"shape mismatch extra={extra} missing={missing}")
    run_id = payload["run_id"]
    status = payload["status"]
    if not isinstance(run_id, str) or status not in ALLOWED_STATUS:
        raise TypeError("run_id must be a string and status must be a known enum")
    return {"run_id": run_id, "status": status}

def can_show_update(project_id: str, update: dict[str, str]) -> bool:
    return update["run_id"] in PROJECT_RUNS.get(project_id, set())

parsed = validate_update({"run_id": "RUN-842", "status": "failed"})
print("format gate:", parsed)
print("project allowed:", can_show_update("project-alpha", parsed))
print("other project allowed:", can_show_update("project-beta", parsed))

Output

format gate: {'run_id': 'RUN-842', 'status': 'failed'}
project allowed: True
other project allowed: False

The two gates catch different failures: malformed data and untrusted authority. A deployed system still has to handle hard tasks, schema rollouts, truncation, and refusals without weakening either contract.

Production patterns

Put checkable evidence in the contract

A 2024 study found that format restrictions changed accuracy on its evaluated reasoning and classification tasks, using the model versions available then. The direction wasn't universal: stricter formats often hurt reasoning, sometimes helped classification, and field order caused a large failure on one symbolic-reasoning task.^{[10]Reference 10Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelshttps://arxiv.org/abs/2408.02442} Use that result to motivate task-specific evaluation, not to conclude that every schema lowers quality.

Don't answer that trade-off by storing unrestricted chain-of-thought. Add product-visible evidence a reviewer can check: a cited quote, extracted quantity, calculation step, or reason code. Put it before the final decision and validate it separately. Generic "thoughts" that nobody reads add noise, not a useful contract.

validate-visible-evidence-before-decision.py

ALLOWED_DECISIONS = {"retry_tests", "rollback", "escalate", "unknown"}

def validate_decision(payload: object) -> dict[str, str]:
    if not isinstance(payload, dict):
        raise TypeError("root must be an object")
    required = ("evidence_quote", "run_id", "decision")
    extra = sorted(set(payload) - set(required))
    missing = [key for key in required if key not in payload]
    if extra or missing:
        raise ValueError(f"shape mismatch extra={extra} missing={missing}")
    quote = payload["evidence_quote"]
    run_id = payload["run_id"]
    decision = payload["decision"]
    if not isinstance(quote, str) or not isinstance(run_id, str):
        raise TypeError("evidence_quote and run_id must be strings")
    if decision not in ALLOWED_DECISIONS:
        raise ValueError("unknown decision")
    if run_id not in quote:
        raise ValueError("run ID does not appear in evidence")
    return {"evidence_quote": quote, "run_id": run_id, "decision": decision}

decision = validate_decision(
    {
        "evidence_quote": "Run RUN-842 failed in unit-tests after auth-cache timed out.",
        "run_id": "RUN-842",
        "decision": "retry_tests",
    }
)
print("decision:", decision["decision"])
print("evidence checked:", decision["run_id"] in decision["evidence_quote"])

Output

decision: retry_tests
evidence checked: True

Version the schema in application code

Schemas change while producers and consumers are still running. If they disagree about a payload version, a rollout can break downstream processing even when every response is valid JSON.

🎯 Production tip: Version schemas in application code. Choose the validator before sending the request, then attach that version to the validated record. Don't let the model choose which contract it claims to satisfy.

route-versioned-records-with-code-owned-contracts.py

def validate_run_status_v2(payload: object) -> dict[str, str]:
    if not isinstance(payload, dict):
        raise TypeError("root must be an object")
    required = ("run_id", "status", "failing_job")
    extra = sorted(set(payload) - set(required))
    missing = [key for key in required if key not in payload]
    if extra or missing:
        raise ValueError(f"shape mismatch extra={extra} missing={missing}")
    record = {}
    for key in required:
        value = payload[key]
        if not isinstance(value, str):
            raise TypeError(f"{key} must be a string")
        record[key] = value
    return record

generated_payload = {"run_id": "RUN-842", "status": "failed", "failing_job": "unit-tests"}
validated = validate_run_status_v2(generated_payload)
wire_record = {"schema_version": "2", "payload": validated}
print("version:", wire_record["schema_version"])
print("failing job:", wire_record["payload"]["failing_job"])

Output

version: 2
failing job: unit-tests

Recover by failure class, without weakening the contract

Structured outputs still fail. Separate recoverable failures, such as truncation from max_output_tokens, from policy outcomes such as refusals or content filtering. A refusal isn't an ordinary parse failure, so don't retry it with a looser mode.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Truncation also doesn't justify switching from schema enforcement to JSON mode. Missing content remains missing, and the fallback throws away the stronger contract. Prefer a bounded retry with more output budget, a smaller schema, or chunked input. The fake client below mirrors those response states so you can test the control flow without an API call:

bounded-retry-preserves-schema-contract.py

from dataclasses import asdict, dataclass

@dataclass(frozen=True)
class RunStatusUpdate:
    run_id: str
    status: str
    failing_job: str

@dataclass
class ContentItem:
    type: str
    refusal: str | None = None

@dataclass
class MessageOutput:
    content: list[ContentItem]

@dataclass
class IncompleteDetails:
    reason: str

@dataclass
class FakeResponse:
    status: str
    output: list[MessageOutput]
    output_parsed: RunStatusUpdate | None = None
    incomplete_details: IncompleteDetails | None = None

class FakeResponsesApi:
    def __init__(self, responses: list[FakeResponse]) -> None:
        self.responses = iter(responses)

    def parse(self, **kwargs) -> FakeResponse:
        return next(self.responses)

class FakeClient:
    def __init__(self, responses: list[FakeResponse]) -> None:
        self.responses = FakeResponsesApi(responses)

def generate_with_bounded_retry(client: FakeClient, input_text: str) -> RunStatusUpdate:
    response = client.responses.parse(
        model="structured-output-model",
        input=input_text,
        text_format=RunStatusUpdate,
        max_output_tokens=120,
    )

    first_content = next(
        (content for item in response.output for content in item.content),
        None,
    )

    if first_content is not None and first_content.type == "refusal":
        raise RuntimeError(f"policy refusal: {first_content.refusal}")

    if response.status == "completed" and response.output_parsed is not None:
        return response.output_parsed

    if response.status != "incomplete":
        raise RuntimeError(f"Unexpected response status: {response.status}")

    if response.incomplete_details is None:
        raise RuntimeError("Incomplete response did not include a reason")

    if response.incomplete_details.reason != "max_output_tokens":
        raise RuntimeError(
            f"Structured output halted: {response.incomplete_details.reason}"
        )

    retry = client.responses.parse(
        model="structured-output-model",
        input=input_text,
        text_format=RunStatusUpdate,
        max_output_tokens=400,
    )
    retry_content = next(
        (content for item in retry.output for content in item.content),
        None,
    )
    if retry_content is not None and retry_content.type == "refusal":
        raise RuntimeError(f"policy refusal: {retry_content.refusal}")
    if retry.status != "completed" or retry.output_parsed is None:
        raise RuntimeError("bounded retry did not produce a structured result")
    return retry.output_parsed

output_item = MessageOutput(content=[ContentItem(type="output_text")])
incomplete = FakeResponse(
    status="incomplete",
    output=[output_item],
    incomplete_details=IncompleteDetails(reason="max_output_tokens"),
)
completed = FakeResponse(
    status="completed",
    output=[output_item],
    output_parsed=RunStatusUpdate(run_id="RUN-842", status="failed", failing_job="unit-tests"),
)
update = generate_with_bounded_retry(
    FakeClient([incomplete, completed]),
    "Run RUN-842 failed in unit-tests.",
)
print("retry preserved contract:", asdict(update))

refused = FakeResponse(
    status="completed",
    output=[MessageOutput(content=[ContentItem(type="refusal", refusal="blocked")])],
)
try:
    generate_with_bounded_retry(FakeClient([refused]), "disallowed request")
except RuntimeError as exc:
    print("refusal routed:", str(exc))

Output

retry preserved contract: {'run_id': 'RUN-842', 'status': 'failed', 'failing_job': 'unit-tests'}
refusal routed: policy refusal: blocked

Flatten nested structures

Deep nesting and recursive shapes increase parser-state and debugging complexity. Some hosted APIs support recursive schemas, but support alone says nothing about acceptable latency or output size for your workload.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} Benchmark the exact contract on the target runtime.

🎯 Production tip: Keep nesting as shallow as the interface allows, bound recursive outputs, and benchmark the exact schema on the target runtime. For tree-shaped data, a flat list of nodes with parent_id references is often easier to generate, validate, and evolve than a recursive JSON object.

reconstruct-a-flat-tree-after-validation.py

from dataclasses import dataclass

@dataclass(frozen=True)
class FlatNode:
    node_id: str
    parent_id: str | None
    label: str

nodes = [
    FlatNode(node_id="root", parent_id=None, label="ci_run"),
    FlatNode(node_id="n1", parent_id="root", label="failing_job"),
    FlatNode(node_id="n2", parent_id="root", label="log_url"),
]
children: dict[str | None, list[str]] = {}
for node in nodes:
    children.setdefault(node.parent_id, []).append(node.label)

print("root nodes:", children[None])
print("run fields:", children["root"])

Output

root nodes: ['ci_run']
run fields: ['failing_job', 'log_url']

Constrained decoding still costs time

Grammar-guided decoding does extra valid-token work while the model generates. Optimized engines can hide much of it for particular workloads, but schema compliance doesn't make latency disappear. Measure the model, schema, batch shape, and runtime you plan to ship.

Where the latency comes from

Start by separating setup work from per-token work. Schema shape, tokenizer, and runtime design decide how much each path costs:

Cost source	Why it appears	Common mitigation
Grammar compilation	The runtime has to convert a schema or grammar into an indexable guide	Compile once and reuse it across requests
Per-token masking	Each generation step must compute the valid next-token set	Precompute token-prefix tables, compressed FSMs, or categorize context-independent tokens (XGrammar)
First use of a hosted schema	Some providers or model paths may preprocess and cache a new schema before generation starts	Reuse stable schemas and warm hot paths when your provider documents or measurements justify it
Large prompt prefixes	Long schema instructions still consume prefill work and context	Use server-side structured outputs or prefix caching

Compilation, hosted schema preprocessing, and long prompt prefixes mostly delay the first token, or TTFT. Token masking repeats during decode and affects TPOT. Outlines, SGLang, XGrammar, and hosted schema caches reduce these costs with precomputation, token categorization, and reuse. None makes the work free.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}^{[4]Reference 4SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104}^{[9]Reference 9XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelshttps://arxiv.org/abs/2411.15100}^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

The quality-compliance tradeoff

Constraining the output space can also change quality: the grammar removes paths outside the contract. The effect depends on task, model, and schema, and format-restriction studies show measurable changes on reasoning tasks.^{[10]Reference 10Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelshttps://arxiv.org/abs/2408.02442}

🎯 Production tip: Keep schemas semantically permissive but structurally stable. Prefer a stable field set with nullable values or bounded enums over a maze of branching object variants. With strict mode, supported-schema limits and closed-object requirements become part of the interface contract.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Compile once, then reuse the guide

When requests share a schema, compile the guide once and reuse it. The toy mask table below repeats the figure's object-start rule across a batch. A serving stack does the same with a compiled grammar and can also reuse the KV cache when prompts share a prefix.^{[4]Reference 4SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104}

reuse-compiled-object-start-mask.py

LEGAL_BY_STATE = {
    "object_start": {"{"},
}

def sample(state: str, logits: dict[str, float]) -> str:
    legal = LEGAL_BY_STATE[state]
    masked = {
        token: score if token in legal else float("-inf")
        for token, score in logits.items()
    }
    return max(masked, key=masked.get)

batch_logits = [
    {"The": 0.45, "{": 0.30, "Sure": 0.15},
    {"Sure": 0.50, "{": 0.20, "The": 0.10},
]
print([sample("object_start", logits) for logits in batch_logits])

Output

['{', '{']

Debugging common failures

Structured output narrows the failure surface, but it doesn't remove it. Debugging gets easier when you can name the observed symptom, connect it to the boundary that failed, and choose a recovery that preserves the contract. The cases below use the RUN-842 path to make those decisions concrete.

"The model returned valid JSON, so the data must be correct"

Your parser can succeed while the values are nonsense: a failed CI run reads "passed", or a run ID doesn't exist in the build system. Structured outputs enforce format, not accuracy. A {"status": "passed"} value satisfies the schema while contradicting the source log.

Keep the format gate, then add application-layer semantic validation. Check that run IDs exist in CI, that status values match source-of-truth logs, and that enum values match your known set. A schema library or plain assertions after parsing can do the work; the host has to own the check.

Never put authorization decisions in model-owned schema fields

Wrong CI facts are one class of failure. A more dangerous class is schema-valid false authority: boolean or enum fields that look like grants.

Examples a constrained decoder will happily emit if they appear in the schema:

authorized: true
requires_approval: false
risk_tier: "low"
is_admin: true
approved: true

Those values remain model claims. They aren't session identity, a stored approval row, or policy. Keep the model contract descriptive: what happened and what the model proposes. Resolve grants from trusted storage such as the session actor, RBAC, approval records with action hashes, and environment flags owned by the host.

ignore-self-attested-approval.py

from dataclasses import dataclass

@dataclass(frozen=True)
class ModelDeployProposal:
    service: str
    environment: str
    approved: bool  # never treat as a grant

def authorize(proposal: ModelDeployProposal, stored_approval: bool) -> str:
    if proposal.environment == "prod" and not stored_approval:
        return "deny: missing trusted approval"
    return "execute"

proposal = ModelDeployProposal("payment-api", "prod", approved=True)
print(authorize(proposal, stored_approval=False))
print("model approved field was", proposal.approved, "and was ignored")

Output

deny: missing trusted approval
model approved field was True and was ignored

"The Markdown wrapper trap"

The next failure is easier to reproduce: json.loads() raises JSONDecodeError even though the output looks like JSON at a glance. The model may have wrapped it in triple backticks with a json label or added "Here is the result:" before the object. Those extra characters break the raw parse.

Treat unparseable or schema-invalid text as a contract failure. A legacy prompt-only integration can make a bounded retry through a stronger interface or send the item to review. It shouldn't silently slice between braces and promote arbitrary text to a record.

the-markdown-wrapper-trap.py

import json

REQUIRED = ("run_id",)

def accept_typed_record(raw: str) -> str:
    try:
        payload = json.loads(raw)
    except json.JSONDecodeError:
        return "reject: contract not satisfied"
    if not isinstance(payload, dict):
        return "reject: contract not satisfied"
    extra = set(payload) - set(REQUIRED)
    missing = [key for key in REQUIRED if key not in payload]
    if extra or missing:
        return "reject: contract not satisfied"
    if not isinstance(payload["run_id"], str):
        return "reject: contract not satisfied"
    return "accept: typed record"

print(accept_typed_record('{"run_id": "RUN-842"}'))
print(accept_typed_record('Here is the JSON: {"run_id": "RUN-842"}'))
print(accept_typed_record("```json\n{\"run_id\": \"RUN-842\"}\n```"))

Output

accept: typed record
reject: contract not satisfied
reject: contract not satisfied

If the system must produce the contract directly, use structured outputs or a grammar-guided runtime instead of repairing free-form text.

"A refusal isn't a parse error"

Suppose a fallback cascade retries a refused request with a looser mode and receives another refusal. The extra tokens and latency haven't changed the outcome. A refusal or content-filter stop is a policy result, not a decoding bug: the model or safety layer has decided not to answer, and loosening the schema doesn't change that decision.

Surface the refusal to the application layer. Route it to a reviewer, change the input, or return a clear error to the user. Don't treat it as a retryable parse failure.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

"Grammar-guided decoding is always slow"

You may hear that constrained decoding adds overhead and conclude that it's too slow. Naive implementations can be slow, while optimized runtimes reduce the cost substantially. The overhead depends on tokenizer alignment, grammar complexity, and cache hits.

Benchmark before deciding. Ask where the overhead appears and whether traffic can amortize it. If many requests share a schema, compilation can be reused. Compare hosted schema APIs with optimized self-hosted runtimes on your latency and compliance targets; provider-managed enforcement hides implementation work but doesn't guarantee lower latency.^{[1]Reference 1Efficient Guided Generation for Large Language Models.https://arxiv.org/abs/2307.09702}^{[4]Reference 4SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104}^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

"The grammar is fighting the model"

Sometimes the response has the right shape but is empty, padded with whitespace, or truncated at max_output_tokens. The mask may be forcing a token that the model assigned very little probability. Both llama.cpp and SGLang warn that a grammar doesn't teach field meaning, so the prompt still needs to describe the desired format.^{[7]Reference 7llama.cpp Grammars Documentationhttps://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md}^{[5]Reference 5SGLang Structured Outputs Documentationhttps://docs.sglang.io/docs/advanced_features/structured_outputs}

Keep the schema and strengthen the task instructions. If generation still stalls, simplify or bound the contract, or split the job. Raw text shouldn't be the first recovery.

"JSON mode and structured outputs are the same thing"

JSON mode can return {"foo": "bar"} when you expect {"name": "string", "age": "integer"}. It enforces valid JSON syntax on successful completion, not schema compliance, so any valid JSON object may pass.

Use structured outputs or grammar-guided decoding when you need schema adherence. JSON mode fits cases where syntactic validity is enough and your application will validate the shape itself.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

"I should use structured outputs for every LLM call"

The opposite mistake is wrapping every prompt in a Pydantic model, including creative writing and open-ended Q&A. Structured outputs shine when the result feeds code such as an API, database, or downstream processor. User-facing text often benefits from free-form generation.

Match the contract to the consumer. Use structured outputs when code consumes the answer and free-form generation when a person does. Unnecessary structure spends tokens on syntax and can narrow the model's expressive range.

Practice: build an incident-digest parser

Now carry the boundary into a small incident parser. Read the requirements, sketch the contract, and predict the empty and unknown cases before opening the solution.

You receive a 500-word incident digest about engineering services. Build a tool that extracts every mentioned service and classifies its operational status as healthy, degraded, outage, or unknown.

Requirements

Return a mentions list. Each item has service and status.
Use structured outputs or grammar-guided decoding to enforce the schema.
Handle no mentions as {"mentions": []}, not null or a missing field.
Add a post-validation step that checks the service name against a known service catalog (e.g., auth-api, vector-indexer, billing-export). Flag unknown services for review.

Input example

"auth-api recovered after elevated 5xx errors. vector-indexer remains degraded. report-worker has no known impact."

Expected output shape

expected-output-shape.json

{
  "mentions": [
    {"service": "auth-api", "status": "healthy"},
    {"service": "vector-indexer", "status": "degraded"},
    {"service": "report-worker", "status": "unknown"}
  ]
}

One possible implementation

Reveal one implementation

solution-sketch.py

ALLOWED_STATUS = {"healthy", "degraded", "outage", "unknown"}
KNOWN_SERVICES = {"auth-api", "vector-indexer", "billing-export"}

def validate_digest(payload: object) -> dict[str, list[dict[str, str]]]:
    if not isinstance(payload, dict) or set(payload) != {"mentions"}:
        raise ValueError("root must be an object with only mentions")
    mentions = payload["mentions"]
    if not isinstance(mentions, list):
        raise TypeError("mentions must be a list")
    checked: list[dict[str, str]] = []
    for item in mentions:
        if not isinstance(item, dict):
            raise TypeError("each mention must be an object")
        extra = sorted(set(item) - {"service", "status"})
        missing = [key for key in ("service", "status") if key not in item]
        if extra or missing:
            raise ValueError(f"mention shape extra={extra} missing={missing}")
        service = item["service"]
        status = item["status"]
        if not isinstance(service, str) or status not in ALLOWED_STATUS:
            raise TypeError("service must be a string and status must be a known enum")
        checked.append({"service": service, "status": status})
    return {"mentions": checked}

def unknown_services(digest: dict[str, list[dict[str, str]]]) -> list[str]:
    return [
        mention["service"]
        for mention in digest["mentions"]
        if mention["service"] not in KNOWN_SERVICES
    ]

digest = validate_digest(
    {
        "mentions": [
            {"service": "auth-api", "status": "healthy"},
            {"service": "vector-indexer", "status": "degraded"},
            {"service": "report-worker", "status": "unknown"},
        ]
    }
)
empty = validate_digest({"mentions": []})
print("mentions:", len(digest["mentions"]))
print("unknown services:", unknown_services(digest))
print("empty mentions:", empty["mentions"])

Output

mentions: 3
unknown services: ['report-worker']
empty mentions: []

The implementation makes three deliberate choices:

Require a list, not a nullable field: {"mentions": []} is the empty case. That shape matches OpenAI strict mode, which requires every field.
Post-validate the catalog: the schema can restrict status to four literals, but it can't know whether report-worker is a real service.
Handle truncation: if the digest is long and output hits max_output_tokens, retry with the same schema, chunk the input, or version a smaller contract.

Where structured output leads

The incident parser leaves a useful boundary in view. Schema-constrained generation can enforce structure during decoding; JSON mode enforces syntax only; prompt-only formatting remains best effort. From there, the host owns the meaning: it validates evidence and authority, versions contracts, and routes truncation separately from refusals. The next step is to place these reliable records inside an agent loop and decide which one should trigger a tool.

Next Step

Continue to ReAct & Plan-and-Execute

A schema gives an agent loop reliable fields to inspect, including tool arguments that your runtime can validate before any side effect. The next chapter compares <span data-glossary="react">ReAct</span> for tightly coupled tool use with Plan-and-Execute for longer workflows, where those contracts make each step checkable.

PreviousRAG Security & Access Control

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Efficient Guided Generation for Large Language Models.

Willard, B. T. & Louf, R. · 2023 · arXiv preprint

https://arxiv.org/abs/2307.09702

Structured outputs

OpenAI · 2024

https://developers.openai.com/api/docs/guides/structured-outputs

Outlines Documentation

Outlines Developers · 2026

https://dottxt-ai.github.io/outlines/latest/

SGLang: Efficient Execution of Structured Language Model Programs.

Zheng, L., et al. · 2023

https://arxiv.org/abs/2312.07104

SGLang Structured Outputs Documentation

SGLang Project · 2026

https://docs.sglang.io/docs/advanced_features/structured_outputs

vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

vLLM Team · 2024

https://github.com/vllm-project/vllm

llama.cpp Grammars Documentation

ggml-org · 2026

https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md

llama.cpp: Inference of LLaMA model in pure C/C++

Gerganov, G. · 2023

https://github.com/ggml-org/llama.cpp

XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models

Dong, Y., Ruan, C. F., Cai, Y., et al. · 2024

https://arxiv.org/abs/2411.15100

Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models

Tam, Z. R., Wu, C.-K., Tsai, Y.-L., et al. · 2024

https://arxiv.org/abs/2408.02442

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Where does structured output earn its keep?

Where does structured output earn its keep?

Why isn't "return JSON" an interface contract?

Why isn't "return JSON" an interface contract?

What can you assume after JSON mode, and what changes with structured outputs?

What can you assume after JSON mode, and what changes with structured outputs?

What promise does grammar-guided decoding actually make?

What promise does grammar-guided decoding actually make?

Why can a schema-valid result still be wrong?

Why can a schema-valid result still be wrong?

Why is tokenizer alignment harder than parsing characters?

Why is tokenizer alignment harder than parsing characters?

When is a regular expression enough, and when do you need a CFG?

When is a regular expression enough, and when do you need a CFG?

Which state should an Outlines-style integration keep warm?

Which state should an Outlines-style integration keep warm?

What work remains for the prompt after a grammar is attached?

What work remains for the prompt after a grammar is attached?

Which latency work belongs to prefix caching, and which belongs to grammar state?

Which latency work belongs to prefix caching, and which belongs to grammar state?

Where does XGrammar save work?

Where does XGrammar save work?

Which control flow does each technique create?

Which control flow does each technique create?

What does a strict tool schema leave for the host?

What does a strict tool schema leave for the host?

What boundary does additionalProperties: false close?

What boundary does additionalProperties: false close?

How do visible evidence fields differ from hidden chain of thought?

How do visible evidence fields differ from hidden chain of thought?

When does an intermediate field earn its place?

When does an intermediate field earn its place?

Why should the validated record carry its schema version?

Why should the validated record carry its schema version?

Which failure path is safe to retry?

Which failure path is safe to retry?

Why can a flat list with parent_id beat recursive JSON?

Why can a flat list with parent_id beat recursive JSON?

Which costs delay the first token, and which repeat during decode?

Which costs delay the first token, and which repeat during decode?

Where should schema strictness stop?

Where should schema strictness stop?

When does reuse pay for itself?

When does reuse pay for itself?

Which gate can catch a schema-valid false authority?

Which gate can catch a schema-valid false authority?

Why should a machine-consuming pipeline reject Markdown wrappers?

Why should a machine-consuming pipeline reject Markdown wrappers?

Why shouldn't a refusal fall back to looser generation?

Why shouldn't a refusal fall back to looser generation?

What should a constrained-decoding benchmark hold constant?

What should a constrained-decoding benchmark hold constant?

What work remains after JSON mode?

What work remains after JSON mode?

What simple rule picks the interface?

What simple rule picks the interface?

Why does the incident-digest parser flag report-worker for review?

Why does the incident-digest parser flag report-worker for review?

Mastery Check

Discussion

What boundary does `additionalProperties: false` close?

Why can a flat list with `parent_id` beat recursive JSON?

Why does the incident-digest parser flag `report-worker` for review?