Skip to content
Skip to lesson
Back to Topics
LearnApplied LLM EngineeringModel Gateways, Routing, and Fallbacks
⚙️MediumMLOps & Deployment

Model Gateways, Routing, and Fallbacks

The healthy backup cannot return the cited answer access-R900 needs. A faster reply would still be the wrong route.

32 min read
Learning path
Step 84 of 199 in the full curriculum

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

The primary model times out on access-R900. A cheap backup is healthy, but it can't return the cited, structured answer that this private access request needs. Returning something quickly would turn a provider outage into an application error.

The cost engineering lesson supplied a fictional release support-release-2026-09-cost-v1, schema cited-support-answer-v3, and a 0.005000 USD single-generation ceiling. A can apply these requirements when choosing a large language model (LLM) path. The app sends a stable internal request; the gateway filters approved paths, invokes an adapter, checks the result, and records the decision. Privacy and human review also require authorization and workflow controls around that model call.

Compatibility is the constraint that makes every later choice meaningful. The gateway must enforce the full contract so application code can't satisfy one field by dropping another.

What belongs at the gateway boundary

An adapter gives application code a stable internal schema while translating provider wire formats. You can put that adapter inside a service or behind a shared gateway. A separate gateway centralizes credentials, routing, accounting, and telemetry, but also adds a dependency, latency, and an operational failure boundary.

For example, native schema configuration differs across APIs. These are request fields checked September 21, 2026, not interchangeable SDK helper arguments:

APINative JSON-schema output configuration
OpenAI Responsestext.format with type: "json_schema"
OpenAI Chat Completionsresponse_format with a json_schema object
Claude Messagesoutput_config.format with type: "json_schema"

Claude also offers strict tool inputs; tools aren't the only route to structured output. Both providers document supported schema subsets and exceptions such as refusals and incomplete output. Keep application validation even when constrained decoding is enabled.[1][2]

Cost engineering separated generation cost from cache decisions. Now the gateway consumes that alongside product and safety requirements. A stored semantic-cache hit ends before this decision, but still needs its own authorization and freshness checks. The gateway decides where one generated support request may run.

Keep three terms separate:

TermJobExample here
AdapterTranslates a stable application request into one provider's API shapeSend messages and parse a structured reply
LaneNames an approved model-plus-adapter path and its surrounding controlslocal-private-cited-review
GatewayCompiles requirements, filters lanes, selects or falls back, and logs the decisionReject a cheap lane that drops citations

Request translation leaves contract compatibility unresolved. Anthropic describes its OpenAI SDK compatibility layer as a way to test and compare Claude, not as a long-term production path. In that layer, prompt isn't supported, response_format is ignored, and the strict tool-calling flag is ignored too. For schema conformance, Anthropic points you to native structured outputs on the Claude API.[3]

Record capabilities on each lane. Don't infer them from a familiar URL shape. Here, citation support preserves application evidence identifiers in JSON for subsequent checking; reviewers still need to verify each claim against its source. Claude currently rejects native citations combined with output_config.format.[2] Test capability combinations as a set. Human review belongs to the surrounding workflow. A lane can prepare a reviewable answer; an authorized reviewer approves access grants.

Before routing access-R900, inventory the cost artifact, trusted request facts, a compiled contract, and lane capability records. The lanes, capabilities, costs, and timings below are teaching fixtures. Replace them with measured behavior and the effective provider rate card for each lane. They don't use the cost lesson's rate card: at those dated rates, 24,000 input tokens alone would cost 0.060000 USD, so these lane costs stand in for a much cheaper model. This CPU lab has no authentication, provider adapters, answer review, human queue, or live-spend controls. A deployed registry needs retained evaluations tied to model, adapter, schema, policy, and registry revisions.

Run the numbered Python cells in order using Python 3.10 or later; they share state and need no network or third-party packages. The first examples select routes. Later cells simulate attempts, advance a clock, and reject invalid replies. No example grants real access or calls a model.

01-gateway-types-and-lanes.py
from dataclasses import dataclass
from decimal import Decimal
from enum import Enum
import json
import math

COST_RELEASE_ID = "support-release-2026-09-cost-v1"
GATEWAY_POLICY_ID = "gateway-policy-v1"
REQUIRED_ANSWER_SCHEMA = "cited-support-answer-v3"
MAX_GENERATED_ANSWER_USD = Decimal("0.005000")
MAX_GENERATION_ATTEMPTS = 2
REQUEST_DEADLINE_MS = 2_500
MAX_REQUEST_SPEND_USD = Decimal("0.010000")
ATTEMPT_RESERVATION_USD = Decimal("0.005000")

def owned_text(value: object) -> bool:
    return type(value) is str and bool(value.strip())

def integer_count(value: object) -> bool:
    return type(value) is int and value >= 0

def money(value: object) -> bool:
    return type(value) is Decimal and value.is_finite() and value >= 0

def seconds(value: object) -> bool:
    if type(value) not in (int, float):
        return False
    try:
        return math.isfinite(value) and value >= 0
    except OverflowError:
        return False

class DataClass(str, Enum):
    PUBLIC = "public"
    TENANT_PRIVATE = "tenant_private"

@dataclass(frozen=True)
class GatewayRequest:
    request_id: str
    task: str
    data_class: DataClass
    context_tokens: int
    risk_amount_cents: int = 0
    requires_citations: bool = False
    requires_schema: bool = True

    def __post_init__(self) -> None:
        if not owned_text(self.request_id) or not owned_text(self.task) or type(self.data_class) is not DataClass:
            raise ValueError("need owned request/task identifiers and a known data class")
        if not integer_count(self.context_tokens) or self.context_tokens == 0 or not integer_count(self.risk_amount_cents):
            raise ValueError("need positive context and nonnegative integer risk")
        if type(self.requires_citations) is not bool or type(self.requires_schema) is not bool:
            raise ValueError("requirement flags must be known Booleans")

@dataclass(frozen=True)
class RouteContract:
    request_id: str
    data_class: DataClass
    context_tokens: int
    needs_citations: bool
    needs_schema: bool
    needs_human_review: bool
    max_answer_cost_usd: Decimal

    def __post_init__(self) -> None:
        if not owned_text(self.request_id) or type(self.data_class) is not DataClass or not integer_count(self.context_tokens) or self.context_tokens == 0:
            raise ValueError("need a named contract, known data class, and positive context")
        if any(type(value) is not bool for value in (self.needs_citations, self.needs_schema, self.needs_human_review)) or self.needs_schema is not True:
            raise ValueError("known requirements and the inherited schema are required")
        if not money(self.max_answer_cost_usd) or self.max_answer_cost_usd > MAX_GENERATED_ANSWER_USD:
            raise ValueError("cost ceiling must fit the inherited policy")

@dataclass(frozen=True)
class Lane:
    name: str
    provider: str
    allowed_data_classes: frozenset[DataClass]
    max_context_tokens: int
    supports_schema: bool
    supports_citations: bool
    supports_human_review: bool
    evaluated_answer_cost_usd: Decimal
    expected_latency_ms: int

    def __post_init__(self) -> None:
        if not owned_text(self.name) or not owned_text(self.provider):
            raise ValueError("lane and provider must be named")
        if type(self.allowed_data_classes) is not frozenset or not self.allowed_data_classes or any(type(value) is not DataClass for value in self.allowed_data_classes):
            raise ValueError("need a nonempty set of known data classes")
        if not integer_count(self.max_context_tokens) or self.max_context_tokens == 0 or not integer_count(self.expected_latency_ms):
            raise ValueError("invalid context capacity or latency")
        if any(type(value) is not bool for value in (self.supports_schema, self.supports_citations, self.supports_human_review)):
            raise ValueError("capability flags must be known Booleans")
        if not money(self.evaluated_answer_cost_usd):
            raise ValueError("evaluated cost must be a finite nonnegative Decimal")

LANES = [
    Lane(
        "fast-public-json", "hosted-fast", frozenset({DataClass.PUBLIC}),
        16_000, True, False, False, Decimal("0.001100"), 260,
    ),
    Lane(
        "public-cited-review", "hosted-cited", frozenset({DataClass.PUBLIC}),
        64_000, True, True, True, Decimal("0.003800"), 860,
    ),
    Lane(
        "primary-private-cited-review", "hosted-private", frozenset({DataClass.TENANT_PRIVATE}),
        64_000, True, True, True, Decimal("0.004200"), 940,
    ),
    Lane(
        "local-private-cited-review", "local-private", frozenset({DataClass.TENANT_PRIVATE}),
        32_000, True, True, True, Decimal("0.004500"), 1_100,
    ),
    Lane(
        "regional-private-cited-review", "regional-private", frozenset({DataClass.TENANT_PRIVATE}),
        64_000, True, True, True, Decimal("0.004560"), 1_300,
    ),
    Lane(
        "cheap-text-fallback", "hosted-cheap", frozenset({DataClass.PUBLIC, DataClass.TENANT_PRIVATE}),
        32_000, False, False, False, Decimal("0.001500"), 420,
    ),
]

print(f"cost_release={COST_RELEASE_ID}")
print(f"gateway_policy={GATEWAY_POLICY_ID}")
print(f"required_answer_schema={REQUIRED_ANSWER_SCHEMA}")
print(f"max_generated_answer_usd={MAX_GENERATED_ANSWER_USD}")
print(f"max_generation_attempts={MAX_GENERATION_ATTEMPTS}")
print(f"request_deadline_ms={REQUEST_DEADLINE_MS}")
print(f"registered_lanes={len(LANES)}")
Output
cost_release=support-release-2026-09-cost-v1
gateway_policy=gateway-policy-v1
required_answer_schema=cited-support-answer-v3
max_generated_answer_usd=0.005000
max_generation_attempts=2
request_deadline_ms=2500
registered_lanes=6

Compile requirements before choosing a lane

The bug fits in two early returns:

unsafe-first-match.py
if request.data_class == "tenant_private":
    return "local-private"
if request.risk_amount_cents >= 50_000:
    return "human-review"

For a private break-glass access request with 900 USD of risk, the first return silently drops the second requirement. Give each stage one job:

  1. Convert every request fact into one contract.
  2. Drop every lane that violates any contract field.
  3. Rank only the remaining lanes by measured preferences such as cost or latency.

Predict access-R900 before running the compiler: its risk requires human review, its private notes require a private lane, and its answer still needs cited-support-answer-v3. The next cell compiles those requirements. Neither check erases the other, and needs_schema means that exact schema, not "any JSON."

These records come from a trusted application boundary, not unchecked JSON from a client. Authenticate the caller, authorize the task and tenant, and derive classification and risk from server-side policy before constructing one. A client-supplied data_class="public" can't declassify private notes. The compiler below also forces private handling and citations for the access task, even if those flags are weakened accidentally.

The single context_tokens count is a fixture simplification. Tokenizers and prompt wrappers differ by lane. At dispatch, tokenize each candidate's complete input, including tools and evidence, using that lane's tokenizer; reserve output and other tokens against its actual limits. Recheck region, retention policy, credential scope, and registry revision too. A private-data Boolean can't express all those conditions.

02-compile-route-contract.py
HIGH_RISK_ACCESS_CENTS = 50_000

def compile_contract(request: GatewayRequest) -> RouteContract:
    if type(request) is not GatewayRequest:
        raise ValueError("need a validated gateway request")
    if request.task not in {"prod_access_decision", "deploy_policy"}:
        raise ValueError("unknown task")
    access_task = request.task == "prod_access_decision"
    return RouteContract(
        request_id=request.request_id,
        data_class=DataClass.TENANT_PRIVATE if access_task else request.data_class,
        context_tokens=request.context_tokens,
        needs_citations=access_task or request.requires_citations,
        # The inherited release contract requires its schema on every route.
        # A caller can't opt out by setting requires_schema=False.
        needs_schema=True,
        needs_human_review=request.risk_amount_cents >= HIGH_RISK_ACCESS_CENTS,
        max_answer_cost_usd=MAX_GENERATED_ANSWER_USD,
    )

private_access = GatewayRequest(
    request_id="access-R900",
    task="prod_access_decision",
    data_class=DataClass.TENANT_PRIVATE,
    context_tokens=24_000,
    risk_amount_cents=90_000,
    requires_citations=True,
)
private_contract = compile_contract(private_access)
schema_downgrade = GatewayRequest(
    "schema-bypass",
    "prod_access_decision",
    DataClass.TENANT_PRIVATE,
    2_000,
    requires_schema=False,
)
assert compile_contract(schema_downgrade).needs_schema is True

print(f"request={private_contract.request_id}")
print(f"data_class={private_contract.data_class.value}")
print(f"needs_citations={private_contract.needs_citations}")
print(f"needs_human_review={private_contract.needs_human_review}")
print(f"max_answer_cost_usd={private_contract.max_answer_cost_usd}")
Output
request=access-R900
data_class=tenant_private
needs_citations=True
needs_human_review=True
max_answer_cost_usd=0.005000

The output preserves every requirement. Now make each lane explain its rejection instead of returning a bare Boolean. If no lane survives, an operator needs to know whether the missing capability is private-data handling, context length, citations, review, or budget.

Before reading the rows, predict the result for access-R900: three private cited-review lanes should survive; cheap text should fail schema, citations, and review.

03-filter-compatible-lanes.py
def contract_violations(lane: Lane, contract: RouteContract) -> list[str]:
    if type(lane) is not Lane or type(contract) is not RouteContract:
        raise ValueError("need a validated lane and contract")
    failures: list[str] = []
    if contract.data_class not in lane.allowed_data_classes:
        failures.append("data_boundary")
    if lane.max_context_tokens < contract.context_tokens:
        failures.append("context_length")
    if contract.needs_schema and not lane.supports_schema:
        failures.append("schema")
    if contract.needs_citations and not lane.supports_citations:
        failures.append("citations")
    if contract.needs_human_review and not lane.supports_human_review:
        failures.append("human_review")
    if lane.evaluated_answer_cost_usd > contract.max_answer_cost_usd:
        failures.append("budget")
    return failures

def compatible_lanes(contract: RouteContract) -> list[Lane]:
    return [lane for lane in LANES if not contract_violations(lane, contract)]

for lane in LANES:
    failures = contract_violations(lane, private_contract)
    status = "compatible" if not failures else "reject=" + ",".join(failures)
    print(f"{lane.name}: {status}")
Output
fast-public-json: reject=data_boundary,context_length,citations,human_review
public-cited-review: reject=data_boundary
primary-private-cited-review: compatible
local-private-cited-review: compatible
regional-private-cited-review: compatible
cheap-text-fallback: reject=schema,citations,human_review

Three private cited-review lanes survive the supplied fields. Cheap text looks tempting: its declared data classes include tenant-private and its cost estimate is below 0.005000 USD. It still fails schema, citations, and review. An estimated cost isn't a charge bound; the later executor demonstrates reservations separately.

💡 Key insight: Budget and a private-data flag aren't a contract. cheap-text-fallback fails three hard fields at once.

The gateway can rank survivors by a soft preference without weakening any requirement. Here it chooses lower fixture answer cost, then latency as a deterministic tie-breaker.

Expect docs-Q102 to take the fast public lane and access-R900 to take the lowest-cost private cited-review lane. The next cell makes that choice explicit and records its reasons.

04-choose-primary-lane.py
@dataclass(frozen=True)
class RouteDecision:
    request_id: str
    lane: str | None
    action: str
    reasons: tuple[str, ...]

def choose_primary(contract: RouteContract) -> RouteDecision:
    feasible = compatible_lanes(contract)
    if not feasible:
        return RouteDecision(contract.request_id, None, "escalate", ("no_compatible_lane",))
    lane = min(feasible, key=lambda candidate: (candidate.evaluated_answer_cost_usd, candidate.expected_latency_ms, candidate.name))
    reasons = (
        f"data={contract.data_class.value}",
        f"review={str(contract.needs_human_review).lower()}",
        f"citations={str(contract.needs_citations).lower()}",
        f"budget<={contract.max_answer_cost_usd}",
    )
    return RouteDecision(contract.request_id, lane.name, "generate", reasons)

status_request = GatewayRequest(
    "docs-Q102", "deploy_policy", DataClass.PUBLIC, 2_000,
    requires_citations=False,
)

for request in (status_request, private_access):
    decision = choose_primary(compile_contract(request))
    print(f"{decision.request_id} -> {decision.lane} action={decision.action}")
    print("  " + " ".join(decision.reasons))
Output
docs-Q102 -> fast-public-json action=generate
  data=public review=false citations=false budget<=0.005000
access-R900 -> primary-private-cited-review action=generate
  data=tenant_private review=true citations=true budget<=0.005000
Two stages. First, a matrix of six lanes against six requirements: private data, 24,000 tokens of context, the cited JSON schema, citations, human review, and the half-cent cost cap. Only primary private, local private, and regional private pass every column. Cited public fails one, cheap text fails three, and fast public fails four. Second, the three lanes that pass are ranked by estimated cost: primary private at $0.004200 and 940 milliseconds is chosen, local private at $0.004500 and 1,100 milliseconds is the fallback, and regional private is at $0.004560 and 1,300 milliseconds.
The matrix checks declared compatibility before ranking by supplied cost. A pre-output fallback runs through these same gates before eligible lanes are ranked; dispatch checks circuit permission, remaining time, and spend reservations. Privacy enforcement, answer quality, and settled charges require their own evidence.

The filter above is deliberately rule-based. Data boundaries, schema requirements, review requirements, and approved cost ceilings are policy constraints. A probabilistic classifier that relaxes any of them would make a plausible prediction more important than authorization.

Learned routing still has a job inside the feasible set. Consider three preferences:

  • Task quality: Evaluate candidate models on your actual lookup, reconciliation, and reasoning cases. RouteLLM learns from preference data and reports more than a twofold cost reduction in certain benchmark cases without reducing measured quality. Re-run its benchmark workflow on this private-access workload and model pair before estimating production savings; choose routes from measured quality, latency, and cost.[4]
  • Latency: Set time-to-first-token (TTFT) and complete-answer targets for the product, then measure realistic load. There's no universal sub-500 ms requirement or hosted-only solution. Batch work has a different completion window; minutes of acceptable delay alone don't make a workload eligible for a 24-hour Batch path.
  • Spend and quotas: Metering helps choose a cheaper approved lane, defer work, or reject it under a declared policy. A 90% budget warning is a policy choice, not a hard cap. Reservations, settlement, and atomic account-level admission are what prevent concurrent calls from overspending an enforceable bound.

Treat learned routers and heuristics as ranking engines over already-legal lanes. They're never a replacement for privacy, context, or schema validation gates.

The control boundary has two paths. A soft router ranks only candidates that pass the contract filter. A pre-output failure returns to that same filter; dispatch then checks circuit state, remaining time, and reserved spend. If no compatible lane remains, the gateway escalates.

Make fallback another contract decision

When a primary lane fails before output, the gateway may get a second chance to generate. It doesn't get a second contract. A cited access answer must remain private, structured, reviewable, and admitted under the same cost policy during an outage.

Gateway products expose fallback mechanisms. Validate each replacement against the semantic contract for the request. LiteLLM documents ordered fallbacks and separate chains for ordinary errors, context-window errors, and content-policy errors. Those knobs decide when another model group may be tried; your contract decides which replacement is acceptable. Its allowed_fails and cooldown_time settings can temporarily remove failing deployments. One documented exception matters: fallback to a specific model ID bypasses cooldown checks for that replacement. Test the exact configured path before relying on its breaker behavior.[5]

⚠️ Common mistake: Copying an ordered fallback list from a proxy config and treating every backup as legal. The proxy will happily fail over. The contract still has to admit that lane.

Classify the failure before the gateway looks for a candidate. A transient or timeout before output is shown may be retried against a compatible lane if attempts, time, and spend remain. A quota-exhausted account isn't the same as a temporary throttle. Authentication failures and policy refusals stop here; switching credentials or providers to bypass them isn't recovery. Once a model has streamed visible text, silently switching models can produce a contradictory continuation. Stop that answer and expose an explicit retry or handoff.

This lab retries answer generation separately from tool writes. Generation isn't generally a pure deterministic function: sampling can vary, and hosted state, storage, or enabled tools can have effects beyond token production. A retry requires a known generation-only path and compatible conversation/input state.

Suppose an access-grant tool commits a write, then the explanation times out. Replaying the whole turn can duplicate the grant. Preserve its verified receipt, recheck authorization, and retry only the explanation when allowed. Write services need scoped operation identities, parameter comparison, retained results, and atomic commit or reconciliation. A pre-output timeout alone says nothing about whether a write committed.[6]

A context rejection also stops here. The filter already checked capacity, so a provider rejection means stale registry data or another mismatch. Investigate it instead of hiding it behind a downgrade.

Some production gateways retry a different schema-capable lane after a validation miss, as long as the caller never saw the bad payload. This lab escalates schema_invalid so a broken supports_schema flag can't hide behind another attempt. Cheap text is illegal either way.

The next cell makes the transparent-retry boundary explicit:

05-classify-fallback-eligible-failures.py
class FailureKind(str, Enum):
    RATE_LIMIT_BEFORE_OUTPUT = "rate_limit_before_output"
    TIMEOUT_BEFORE_OUTPUT = "timeout_before_output"
    QUOTA_EXHAUSTED = "quota_exhausted"
    CONTEXT_REJECTED = "context_rejected"
    MID_STREAM_DROP = "mid_stream_drop"
    SCHEMA_INVALID = "schema_invalid"
    AUTH_DENIED = "auth_denied"
    POLICY_REFUSAL = "policy_refusal"

def may_retry_transparently(failure: FailureKind) -> bool:
    return type(failure) is FailureKind and failure in {
        FailureKind.RATE_LIMIT_BEFORE_OUTPUT,
        FailureKind.TIMEOUT_BEFORE_OUTPUT,
    }

for failure in FailureKind:
    action = "try_compatible_fallback" if may_retry_transparently(failure) else "stop_or_escalate"
    print(f"{failure.value}: {action}")
Output
rate_limit_before_output: try_compatible_fallback
timeout_before_output: try_compatible_fallback
quota_exhausted: stop_or_escalate
context_rejected: stop_or_escalate
mid_stream_drop: stop_or_escalate
schema_invalid: stop_or_escalate
auth_denied: stop_or_escalate
policy_refusal: stop_or_escalate

For access-R900, suppose the lower-cost primary lane times out before emitting output. The fallback selector excludes the failed lane, applies the same contract filter, and chooses from what remains.

Predict the result before looking at the output: local private should serve, regional private should remain available, and cheap text should still be rejected.

06-select-contract-preserving-fallback.py
def choose_fallback(contract: RouteContract, failed_lane: str) -> RouteDecision:
    if not owned_text(failed_lane) or failed_lane not in {lane.name for lane in LANES}:
        raise ValueError("need a known failed lane")
    candidates = [
        lane for lane in compatible_lanes(contract)
        if lane.name != failed_lane
    ]
    if not candidates:
        return RouteDecision(contract.request_id, None, "escalate", ("fallback_contract_unmet",))
    lane = min(candidates, key=lambda candidate: (candidate.evaluated_answer_cost_usd, candidate.expected_latency_ms, candidate.name))
    return RouteDecision(
        contract.request_id,
        lane.name,
        "fallback_generate",
        (f"primary_failed={failed_lane}", "contract_preserved=true"),
    )

primary = choose_primary(private_contract)
fallback = choose_fallback(private_contract, primary.lane or "")
print(f"primary={primary.lane}")
print(f"fallback={fallback.lane} action={fallback.action}")
print("cheap_text_rejected=" + ",".join(contract_violations(LANES[-1], private_contract)))
Output
primary=primary-private-cited-review
fallback=local-private-cited-review action=fallback_generate
cheap_text_rejected=schema,citations,human_review

Both backup estimates fit the single-generation ceiling. That doesn't bound a future call's bill, and a failed primary can still consume work before its timeout reaches the gateway. A maximum for one call also doesn't cap cumulative task spending.

Keep two numbers separate: the admission ceiling used to filter lanes, and the request-total spend that sums usage from every attempt, failed primary plus fallback. Production policy also needs a retry budget: a maximum attempt count, one wall-clock deadline shared across attempts, and post-call pricing for usage reported by every provider attempt.

A critical invariant across sequential attempts is cumulative deadline propagation:

Tremaining=Ttotal−TspentT_{\text{remaining}} = T_{\text{total}} - T_{\text{spent}}

With a 2,500 ms total deadline and 1,800 ms already spent, only 700 ms remain. Giving fallback a fresh 2,500 ms timeout would permit 4,300 ms of attempt time, exceeding the original deadline. Real elapsed time also includes queueing, policy checks, retries, and adapter overhead. Use one monotonic deadline and carry the remaining budget into cancellation and timeout handling. A socket read timeout may reset between chunks, so use a separate total-call deadline. Track remote completion because local cancellation can leave provider work running.

This lab allows at most one fallback (MAX_GENERATION_ATTEMPTS = 2) within a shared 2.5-second deadline. The executor below advances a simulated clock, including retry delay. If the primary consumes 1,800 ms, only 700 ms remain, not another 2,500 ms.

Two timelines on one axis from zero to 3,000 milliseconds, with a deadline marker at 2,500. In the first, a primary that fails at 940 milliseconds is followed by a 1,100 millisecond fallback that finishes at 2,040, leaving 460 milliseconds. In the second, a primary that fails at 1,800 milliseconds leaves 700 milliseconds. The 1,100 millisecond fallback would run past the deadline, shown as a dashed overhang of 400 milliseconds.
With 940 ms spent, the 1,100 ms fallback finishes at 2,040 ms, leaving 460 ms. With 1,800 ms spent, the 700 ms remainder falls 400 ms short. Compare full request latency with the product objective and track provider completion separately.

A hard request-spend cap needs more than estimated lane cost: reserve a conservative charge bound before each attempt, reconcile actual usage, and retain reservations for timed-out calls whose usage is unknown. Concurrent requests need atomic reservations at the tenant/account boundary. The fixture uses a separate 0.010000 USD request cap and a 0.005000 USD reservation per attempt. Real bounds must include all billable token categories and enabled tools; an estimate or typical cost isn't such a bound.

There's another outage cost. If every request still tries the sick primary first, you pay for that failed attempt over and over. How do you stop the next request from paying it too?

Keep an outage from multiplying

The fallback path protects one request. An outage can get worse when every new request still calls the unhealthy primary before trying its backup. Concurrent timeouts and uncoordinated client retries can multiply traffic against the provider.

Two patterns help stop that feedback loop:

  • Jittered exponential backoff: Fixed retry intervals cause clients to synchronize their retry attempts into destructive pulses. Full jittered backoff distributes delays randomly between zero and an exponential ceiling: tsleep=random(0,min⁡(Tmax,Tbase⋅2attempt))t_{\text{sleep}} = \text{random}(0, \min(T_{\text{max}}, T_{\text{base}} \cdot 2^{\text{attempt}})) Jitter spreads retry requests across time and reduces synchronized clusters. Pair it with admission control and load shedding to handle overload. The exponential index starts at zero for the first retry in this convention.[7]
  • Circuit breakers: A suppresses normal calls after chosen health failures cross a threshold. A compatible healthy replacement may exist; otherwise escalate. After cooldown, this policy allows one probe. Other implementations allow a bounded probe set. Probe success can close the circuit; failure reopens it. Define health failures separately from authentication errors, refusals, or unsupported contracts.

This compact implementation models the three states: closed, open, and half_open. Its key is provider because each fixture provider names one upstream target.

A production registry may need provider, endpoint, deployment, or lane keys depending on failure scope. This local breaker rejects malformed clocks and expires an unanswered probe. It has no synchronization or result lease: concurrent workers can race, and an old completion can overwrite newer state. Shared breakers need atomic probe leases, generations that reject stale results, bounded , and actual probe cancellation. After cooldown, probe the upstream and require a healthy response before closing the circuit.

07-circuit-breaker.py
class CircuitStatus(str, Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@dataclass
class CircuitState:
    status: CircuitStatus = CircuitStatus.CLOSED
    failures: int = 0
    opened_until: float = 0.0
    probe_until: float = 0.0

class CircuitBreaker:
    def __init__(self, threshold: int = 2, cooldown_seconds: float = 10.0,
                 probe_timeout_seconds: float = 2.5) -> None:
        if not integer_count(threshold) or threshold == 0 or not seconds(cooldown_seconds) or cooldown_seconds == 0 or not seconds(probe_timeout_seconds) or probe_timeout_seconds == 0:
            raise ValueError("positive finite threshold, cooldown, and probe lifetime required")
        self.threshold = threshold
        self.cooldown_seconds = cooldown_seconds
        self.probe_timeout_seconds = probe_timeout_seconds
        self.states: dict[str, CircuitState] = {}

    def permit(self, provider: str, now: float) -> bool:
        if not owned_text(provider) or not seconds(now) or not seconds(now + self.cooldown_seconds) or not seconds(now + self.probe_timeout_seconds):
            return False
        state = self.states.setdefault(provider, CircuitState())
        if state.status == CircuitStatus.HALF_OPEN:
            if now >= state.probe_until:
                state.status = CircuitStatus.OPEN
                state.opened_until = now + self.cooldown_seconds
            return False
        if state.status == CircuitStatus.OPEN:
            if now < state.opened_until:
                return False
            state.status = CircuitStatus.HALF_OPEN
            state.probe_until = now + self.probe_timeout_seconds
            return True
        return state.status == CircuitStatus.CLOSED

    def failure(self, provider: str, now: float) -> None:
        if not owned_text(provider) or not seconds(now) or not seconds(now + self.cooldown_seconds):
            raise ValueError("need a named target and finite nonnegative clock")
        state = self.states.setdefault(provider, CircuitState())
        state.failures += 1
        if state.status == CircuitStatus.HALF_OPEN or state.failures >= self.threshold:
            state.status = CircuitStatus.OPEN
            state.opened_until = now + self.cooldown_seconds

    def success(self, provider: str) -> None:
        if not owned_text(provider):
            raise ValueError("need a named target")
        self.states[provider] = CircuitState()

breaker = CircuitBreaker()
breaker.failure("hosted-private", 100.0)
breaker.failure("hosted-private", 101.0)
print(f"during_cooldown={breaker.permit('hosted-private', 105.0)}")
print(f"probe_after_cooldown={breaker.permit('hosted-private', 112.0)}")
breaker.success("hosted-private")
print(f"after_success={breaker.states['hosted-private'].status.value}")

unanswered_probe = CircuitBreaker(threshold=1)
unanswered_probe.failure("probe-target", 100.0)
assert unanswered_probe.permit("probe-target", 110.0)
assert not unanswered_probe.permit("probe-target", 110.1)
assert not unanswered_probe.permit("probe-target", 112.5)
assert unanswered_probe.states["probe-target"].status is CircuitStatus.OPEN
print(f"expired_probe={unanswered_probe.states['probe-target'].status.value}")
Output
during_cooldown=False
probe_after_cooldown=True
after_success=closed
expired_probe=open
Two traces on a time axis in seconds, each a colored band showing closed, open, and probing states. In the first, with a threshold of two, failures at 100 and 101 open the circuit until 111, a call at 105 is blocked, the cooldown ends and a probe is allowed from 111, and a probe at 112 succeeds and closes it. In the second, with a threshold of one, a failure at 100 opens the circuit until 110, a probe starts at 110, the next call is denied, and the unanswered probe expires at 112.5, reopening the circuit until 122.5, when a probe is allowed again.
The first trace uses the default threshold of two; the second uses threshold one to show an unanswered probe expiring. Each trace follows its own fixture clock. A successful probe closes the circuit; an expired or failed probe returns it to open.

Execute one outage path and record why

To investigate an incident, retain enough evidence to reconstruct lane selection: policy and registry revisions, approved cost artifact, effective lane/model/adapter, contract fields, attempt/task identities, status, and pricing evidence. Verify the decision against trusted policy and provider records; use IDs and audit rows to correlate the evidence.

That record lets a later review distinguish a model-quality problem from a bad policy decision. The next cell follows one private request through timeout, partial output, and circuit cooldown.

The executor consumes scripted outcomes instead of making HTTP calls. Its defaults assume successful completion with fixed durations and charges; failure overrides let you break that assumption. Each attempted lane advances the simulated clock and contributes its supplied charge. An open circuit consumes no provider attempt or charge in this fixture. served means transport completion only; no payload has yet passed reply validation or human review.

In a real adapter, propagate the remaining deadline to connection, read, and total-call timeouts; disable hidden SDK retries or charge them to the same attempt budget. Track provider completion and billing separately because local cancellation can leave remote generation or charges active. Honor applicable Retry-After delays and use bounded backoff with jitter when retrying under load. The fixture accepts an explicit delay so that boundary is testable.[8]

08-execute-gateway-path.py
@dataclass(frozen=True)
class RouteEvent:
    request_id: str
    policy_id: str
    cost_release_id: str
    action: str
    lane: str | None
    contract_summary: str
    reason: str
    evaluated_cost_usd: Decimal
    attempted_lanes: tuple[str, ...] = ()
    elapsed_ms: int = 0
    fixture_spend_usd: Decimal = Decimal("0")

def lane_by_name(name: str) -> Lane:
    return next(lane for lane in LANES if lane.name == name)

def summarize_contract(contract: RouteContract) -> str:
    return (
        f"data={contract.data_class.value};"
        f"context_tokens={contract.context_tokens};"
        f"schema={str(contract.needs_schema).lower()};"
        f"schema_id={REQUIRED_ANSWER_SCHEMA};"
        f"citations={str(contract.needs_citations).lower()};"
        f"review={str(contract.needs_human_review).lower()};"
        f"budget<={contract.max_answer_cost_usd}"
    )

@dataclass(frozen=True)
class Attempt:
    duration_ms: int
    charge_usd: Decimal
    failure: FailureKind | None = None

    def __post_init__(self) -> None:
        if not integer_count(self.duration_ms) or not money(self.charge_usd):
            raise ValueError("need a nonnegative integer duration and finite Decimal charge")
        if self.failure is not None and type(self.failure) is not FailureKind:
            raise ValueError("need a known failure or explicit completion")

def execute_with_failure(
    request: GatewayRequest,
    failure: FailureKind | None,
    now: float = 200.0,
    scripted: dict[str, Attempt] | None = None,
    retry_delay_ms: int = 0,
    request_cap: Decimal = MAX_REQUEST_SPEND_USD,
) -> RouteEvent:
    if failure is not None and type(failure) is not FailureKind:
        raise ValueError("need a classified failure or explicit completion")
    if not seconds(now):
        raise ValueError("need a finite nonnegative fixture clock")
    if not money(request_cap):
        raise ValueError("request cap must be a finite nonnegative Decimal")
    if scripted is None:
        scripted = {}
    if type(scripted) is not dict or any(
        not owned_text(name) or name not in {lane.name for lane in LANES}
        or type(attempt) is not Attempt or attempt.charge_usd > ATTEMPT_RESERVATION_USD
        for name, attempt in scripted.items()
    ):
        raise ValueError("need known scripted lanes and charges within the reserved bound")
    contract = compile_contract(request)
    contract_summary = summarize_contract(contract)
    candidates = sorted(
        compatible_lanes(contract),
        key=lambda lane: (lane.evaluated_answer_cost_usd, lane.expected_latency_ms, lane.name),
    )
    if type(retry_delay_ms) is not int or retry_delay_ms < 0:
        raise ValueError("retry delay must be nonnegative integer milliseconds")
    # Synthetic known charges, distinct from registry cost estimates.
    charges = dict(zip((lane.name for lane in LANES), map(Decimal, (
        "0.001100", "0.003800", "0.004200", "0.004500", "0.004560", "0.001500",
    ))))
    attempted: list[str] = []
    elapsed = 0
    spend = Decimal("0")
    reason = "no_primary_lane" if not candidates else "no_permitted_compatible_lane"

    def event(action: str, lane: Lane | None = None) -> RouteEvent:
        return RouteEvent(
            request.request_id, GATEWAY_POLICY_ID, COST_RELEASE_ID, action,
            lane.name if lane else None, contract_summary, reason,
            lane.evaluated_answer_cost_usd if lane else Decimal("0"),
            tuple(attempted), elapsed, spend,
        )

    for index, lane in enumerate(candidates):
        if elapsed >= REQUEST_DEADLINE_MS:
            reason = "deadline_exhausted"
            break
        if len(attempted) >= MAX_GENERATION_ATTEMPTS:
            reason = "attempts_exhausted"
            break
        # Don't wait or spend for a fallback that can't finish in the time left.
        delay = retry_delay_ms if attempted else 0
        if attempted and delay + lane.expected_latency_ms >= REQUEST_DEADLINE_MS - elapsed:
            reason = "deadline_exhausted"
            continue
        if spend + ATTEMPT_RESERVATION_USD > request_cap:
            reason = "request_spend_reservation_denied"
            break
        if not breaker.permit(lane.provider, now + elapsed / 1000):
            if index == 0:
                reason = "primary_circuit_open"
            continue
        elapsed += delay
        attempt = scripted.get(lane.name, Attempt(
            lane.expected_latency_ms, charges[lane.name], failure if index == 0 else None,
        ))
        attempted.append(lane.name)
        remaining = REQUEST_DEADLINE_MS - elapsed
        elapsed += min(attempt.duration_ms, remaining)
        spend += attempt.charge_usd
        outcome = attempt.failure
        if attempt.duration_ms >= remaining and outcome is not FailureKind.MID_STREAM_DROP:
            outcome = FailureKind.TIMEOUT_BEFORE_OUTPUT
        if outcome is None:
            breaker.success(lane.provider)
            reason = "primary_contract_match" if index == 0 else reason + ";contract_preserved"
            return event("served" if index == 0 else "served_fallback", lane)
        # Auth, refusal, and schema failures aren't provider-health evidence here.
        if outcome in {FailureKind.TIMEOUT_BEFORE_OUTPUT, FailureKind.MID_STREAM_DROP}:
            breaker.failure(lane.provider, now + elapsed / 1000)
        elif breaker.states[lane.provider].status == CircuitStatus.HALF_OPEN:
            breaker.success(lane.provider)  # reachable, even if response was unusable
        reason = ("primary_" if index == 0 else "fallback_") + outcome.value
        if not may_retry_transparently(outcome):
            return event("escalate")
    return event("escalate")

breaker = CircuitBreaker()
before_output = execute_with_failure(private_access, FailureKind.TIMEOUT_BEFORE_OUTPUT, now=200.0)
mid_stream = execute_with_failure(private_access, FailureKind.MID_STREAM_DROP, now=203.0)
breaker.failure("local-private", 204.0)
breaker.failure("local-private", 205.0)
skip_open_local = execute_with_failure(private_access, None, now=206.0)

# These are scripted charges, not provider-reported usage.
primary_timeout_usage = Decimal("0.004200")
fallback_usage = Decimal("0.004500")
request_attempt_costs = [primary_timeout_usage, fallback_usage]
request_total_spend = sum(request_attempt_costs, Decimal("0"))
assert before_output.evaluated_cost_usd == Decimal("0.004500")
assert request_total_spend == Decimal("0.008700")
assert request_total_spend > MAX_GENERATED_ANSWER_USD
assert before_output.fixture_spend_usd == request_total_spend
assert before_output.elapsed_ms == 2_040

print(f"before_output={before_output.action} lane={before_output.lane} reason={before_output.reason}")
print(f"audit_policy={before_output.policy_id} cost_release={before_output.cost_release_id}")
print(f"audit_contract={before_output.contract_summary}")
print(f"mid_stream={mid_stream.action} lane={mid_stream.lane} reason={mid_stream.reason}")
print(f"circuit_after_failures={breaker.states['hosted-private'].status.value}")
print(f"open_local_skipped={skip_open_local.action} lane={skip_open_local.lane} reason={skip_open_local.reason}")
print(f"attempt_costs={','.join(str(c) for c in request_attempt_costs)}")
print(f"request_total_spend={request_total_spend} admission_ceiling={MAX_GENERATED_ANSWER_USD}")
Output
before_output=served_fallback lane=local-private-cited-review reason=primary_timeout_before_output;contract_preserved
audit_policy=gateway-policy-v1 cost_release=support-release-2026-09-cost-v1
audit_contract=data=tenant_private;context_tokens=24000;schema=true;schema_id=cited-support-answer-v3;citations=true;review=true;budget<=0.005000
mid_stream=escalate lane=None reason=primary_mid_stream_drop
circuit_after_failures=open
open_local_skipped=served_fallback lane=regional-private-cited-review reason=primary_circuit_open;contract_preserved
attempt_costs=0.004200,0.004500
request_total_spend=0.008700 admission_ceiling=0.005000

Replay policy before promoting it

Build a gateway policy from a mechanical replay of low-risk traffic, high-risk cases, private data, outage paths, and unsupported contracts. Use hand-picked examples as a smoke test, then check every simulated success for contract preservation and admission-ceiling compliance. Evaluate answer quality on a reviewed task set.

The replay below adds a request with a private context too large for any approved private lane. It also opens the primary circuit on a simulated timeout, then checks that the next request uses a contract-preserving fallback during cooldown. Unsupported work escalates instead of silently truncating evidence or sending private notes somewhere unapproved.

09-replay-gateway-policy.py
too_large_private_request = GatewayRequest(
    "access-long-context", "prod_access_decision", DataClass.TENANT_PRIVATE, 70_000,
    risk_amount_cents=90_000, requires_citations=True,
)

breaker = CircuitBreaker(threshold=1, cooldown_seconds=10.0)
replay_cases = [
    (status_request, None),
    (private_access, None),
    (private_access, FailureKind.TIMEOUT_BEFORE_OUTPUT),
    (private_access, None),
    (too_large_private_request, None),
]
events = [
    execute_with_failure(request, failure, now=300.0 + 3 * index)
    for index, (request, failure) in enumerate(replay_cases)
]

served = []
expected_actions = ["served", "served", "served_fallback", "served_fallback", "escalate"]
for (request, _), event, expected in zip(replay_cases, events, expected_actions):
    assert event.action == expected
    assert event.elapsed_ms <= REQUEST_DEADLINE_MS
    assert len(event.attempted_lanes) <= MAX_GENERATION_ATTEMPTS
    assert event.fixture_spend_usd <= MAX_REQUEST_SPEND_USD
    if event.action.startswith("served"):
        lane = lane_by_name(event.lane or "")
        assert not contract_violations(lane, compile_contract(request))
        assert event.evaluated_cost_usd <= MAX_GENERATED_ANSWER_USD
        served.append(event)

for event in events:
    print(f"{event.request_id}: {event.action} lane={event.lane}")
route_field_violations = sum(
    len(contract_violations(lane_by_name(event.lane), compile_contract(request)))
    for (request, _), event in zip(replay_cases, events)
    if event.action.startswith("served")
)
assert route_field_violations == 0
print(f"compatible_transport_completions={len(served)}/{len(events)}")
print(f"route_field_violations={route_field_violations}")
Output
docs-Q102: served lane=fast-public-json
access-R900: served lane=primary-private-cited-review
access-R900: served_fallback lane=local-private-cited-review
access-R900: served_fallback lane=local-private-cited-review
access-long-context: escalate lane=None
compatible_transport_completions=4/5
route_field_violations=0

The replay reads no answers. Its cost assertion compares supplied lane estimates; it doesn't verify a future charge bound or a provider bill. The zero violation count covers these five mechanical cases, not population safety.

After each real attempt, normalize its usage and price it with that lane's effective model/tier/region rate card using the preceding lesson's ledger mechanics. Pin the card's contents and retain unknown usage as unresolved. Cross-provider fallback neither transfers a prefix cache nor guarantees the replacement has no independently warm prefix. Before promotion, join route events with answer correctness, citation support, and measured spending.

A can satisfy the mechanical contract and still send too many difficult public questions to a weak but formally compatible lane.

Only after hard requirements pass should you test learned or heuristic ranking for quality and cost. Useful comparison metrics are generated-answer correctness, citation correctness, human-escalation rate, fallback success rate, , and actual spend by lane.

Now exercise the runtime. Each case starts with a fresh breaker. Two failed attempts must stop, a slow primary must leave less time for its fallback, and a request cap must reject a reservation even when each lane passes admission.

10-exercise-runtime-limits.py
primary_name = "primary-private-cited-review"
local_name = "local-private-cited-review"
cases = {
    "fallback_also_fails": ({
        local_name: Attempt(500, Decimal("0.002000"), FailureKind.TIMEOUT_BEFORE_OUTPUT),
    }, 0, MAX_REQUEST_SPEND_USD, "attempts_exhausted", 2),
    "shared_deadline": ({
        primary_name: Attempt(1_800, Decimal("0.004200"), FailureKind.TIMEOUT_BEFORE_OUTPUT),
    }, 0, MAX_REQUEST_SPEND_USD, "deadline_exhausted", 1),
    "retry_delay_uses_deadline": ({}, 2_000, MAX_REQUEST_SPEND_USD, "deadline_exhausted", 1),
    "request_cap": ({}, 0, Decimal("0.008000"), "request_spend_reservation_denied", 1),
    "auth_failure": ({
        primary_name: Attempt(20, Decimal("0"), FailureKind.AUTH_DENIED),
    }, 0, MAX_REQUEST_SPEND_USD, "primary_auth_denied", 1),
    "visible_output_at_deadline": ({
        primary_name: Attempt(3_000, Decimal("0.004200"), FailureKind.MID_STREAM_DROP),
    }, 0, MAX_REQUEST_SPEND_USD, "primary_mid_stream_drop", 1),
}
for name, (scripted, delay, cap, reason, attempts) in cases.items():
    breaker = CircuitBreaker()
    result = execute_with_failure(
        private_access, FailureKind.TIMEOUT_BEFORE_OUTPUT,
        scripted=scripted, retry_delay_ms=delay, request_cap=cap,
    )
    assert result.action == "escalate" and result.reason == reason
    assert len(result.attempted_lanes) == attempts
    assert result.elapsed_ms <= REQUEST_DEADLINE_MS
    assert result.fixture_spend_usd <= cap
    print(f"{name}: {reason}, attempts={attempts}, elapsed_ms={result.elapsed_ms}")
Output
fallback_also_fails: attempts_exhausted, attempts=2, elapsed_ms=1440
shared_deadline: deadline_exhausted, attempts=1, elapsed_ms=1800
retry_delay_uses_deadline: deadline_exhausted, attempts=1, elapsed_ms=940
request_cap: request_spend_reservation_denied, attempts=1, elapsed_ms=940
auth_failure: primary_auth_denied, attempts=1, elapsed_ms=20
visible_output_at_deadline: primary_mid_stream_drop, attempts=1, elapsed_ms=2500

The 0.008000 USD case stops after spending 0.004200 USD: another 0.005000 USD reservation would exceed the cap. Knowing a fallback's typical charge is 0.004500 USD doesn't make it safe to reserve less than the bound. All charges are known in this fixture; unknown real usage must stay reserved until reconciled. The slow-primary and delay cases stop as soon as the fallback can't fit: with 700 ms left it can't finish, and a 2,000 ms delay can't fit in the 1,560 ms left, so waiting would only deliver the same answer later.

The last trace preserves the known visible-output state when the deadline also expires. A deadline doesn't turn an interrupted answer into a pre-output failure. In a real streaming adapter, latch whether any output was published and carry that state through every timeout or cancellation path.

Validate the reply returned by each lane

Lane health and answer validity require separate checks. Adapters must normalize provider-specific text blocks, tool-call identifiers, stop reasons, refusals, and usage into an explicit application result. Reject truncated JSON, made-up citations, and model-authored claims that a human approved the answer.

After the lane is chosen, the reply can still be the wrong object. The check below validates a reduced local reply shape after adapter normalization. The application supplies currently authorized evidence IDs for this request; this function checks ID membership but does not resolve permissions or fetch passages. Compare answer wording with those passages to verify support. Confirm that review="pending" entered the actual review queue and received a human decision.

11-validate-normalized-replies.py
def validate_reply(payload: object, contract: RouteContract, evidence_ids: frozenset[str]) -> dict:
    if type(contract) is not RouteContract or type(evidence_ids) is not frozenset or any(not owned_text(item) for item in evidence_ids):
        raise ValueError("need a validated contract and owned evidence identifiers")
    fields = {"schema", "status", "answer", "citations", "review"}
    if type(payload) is not dict or set(payload) != fields:
        raise ValueError("unexpected reply shape")
    if payload["schema"] != REQUIRED_ANSWER_SCHEMA or payload["status"] != "complete":
        raise ValueError("wrong schema, refusal, or incomplete reply")
    if type(payload["answer"]) is not str or not payload["answer"].strip():
        raise ValueError("missing answer")
    citations = payload["citations"]
    if type(citations) is not list or any(type(item) is not str for item in citations):
        raise ValueError("invalid citations")
    if len(set(citations)) != len(citations) or not set(citations) <= evidence_ids:
        raise ValueError("duplicate or unknown citation")
    if contract.needs_citations and not citations:
        raise ValueError("citations required")
    # Require the marker; a real workflow must separately create/verify handoff state.
    expected_review = "pending" if contract.needs_human_review else "not_required"
    if payload["review"] != expected_review:
        raise ValueError("invalid review state")
    return payload | {"citations": citations.copy()}

good_reply = {
    "schema": REQUIRED_ANSWER_SCHEMA, "status": "complete",
    "answer": "Send this access request for human review.",
    "citations": ["E1"], "review": "pending",
}
assert validate_reply(good_reply, private_contract, frozenset({"E1"})) == good_reply
for name, patch in {
    "refused": {"status": "refused"},
    "truncated": {"status": "length"},
    "wrong_schema": {"schema": "plain-json-v1"},
    "invented_citation": {"citations": ["E99"]},
    "self_approved": {"review": "approved"},
}.items():
    try:
        validate_reply(good_reply | patch, private_contract, frozenset({"E1"}))
    except ValueError:
        print(f"{name}: blocked before publication")
    else:
        raise AssertionError(name)
Output
refused: blocked before publication
truncated: blocked before publication
wrong_schema: blocked before publication
invented_citation: blocked before publication
self_approved: blocked before publication

Before publishing, apply the complete versioned schema, factual/citation checks, and current workflow state. Returning a copied local result prevents later mutation of the caller's citation list from changing the result; validate its claims separately. Keep provider credentials server-side and scoped to approved deployments. A valid provider key does not grant a user access to another tenant's evidence or access tools. Audit records need principal/tenant and policy references, restricted access and retention, and appropriate redaction; never include credentials.

A generation retry must leave a completed write alone

Suppose a separately authorized human approved the access grant and the write succeeded, but the explanation timed out. Retrying generation should consume the recorded receipt. It shouldn't execute the grant again.

This in-memory service uses fake identities and receipts. It demonstrates repeated-key handling and a generation simulation that never invokes the write function. It neither authenticates a reviewer nor attaches a receipt to a real model prompt. A real approval must bind the operation's tenant, principal, target, duration, policy, and lifetime; a Boolean alone isn't that artifact.

The local ledger binds the key to tenant, principal, and exact parameters, rejecting changed intent. Production also needs retained evidence and a durable atomic write-plus-idempotency transaction, or reconciliation when an external commit can't share that transaction. Define retention and late-retry behavior; forgetting a key can allow a later duplicate write.[6]

12-separate-tool-writes-from-generation.py
@dataclass(frozen=True)
class Caller:
    principal: str
    tenant: str
    permissions: frozenset[str]

    def __post_init__(self) -> None:
        if not owned_text(self.principal) or not owned_text(self.tenant) or type(self.permissions) is not frozenset or any(not owned_text(value) for value in self.permissions):
            raise ValueError("need owned caller identity and explicit permission names")

ledger: dict[tuple[str, str, str], tuple[tuple[str, int], str]] = {}
write_count = 0

def grant_fixture(caller: Caller, tenant: str, operation_key: str,
                  target: str, minutes: int, human_approved: bool) -> str:
    global write_count
    if type(caller) is not Caller or not owned_text(tenant):
        raise PermissionError("need a validated caller and named tenant")
    if caller.tenant != tenant or "access:grant" not in caller.permissions:
        raise PermissionError("grant not authorized")
    if human_approved is not True:
        raise PermissionError("trusted human approval required")
    if not owned_text(operation_key) or not owned_text(target) or not integer_count(minutes) or minutes == 0:
        raise ValueError("invalid operation")
    key = (tenant, caller.principal, operation_key)
    parameters = (target, minutes)
    if key in ledger:
        previous, receipt = ledger[key]
        if previous != parameters:
            raise ValueError("idempotency key reused with different parameters")
        return receipt
    write_count += 1
    receipt = f"grant-receipt-{write_count}"
    ledger[key] = (parameters, receipt)
    return receipt

caller = Caller("reviewer-7", "tenant-a", frozenset({"access:grant"}))
receipt = grant_fixture(caller, "tenant-a", "access-R900-grant", "prod-console", 15, True)
# Retrying after a lost write response returns the same receipt.
assert grant_fixture(caller, "tenant-a", "access-R900-grant", "prod-console", 15, True) == receipt
breaker = CircuitBreaker()
generation = execute_with_failure(private_access, FailureKind.TIMEOUT_BEFORE_OUTPUT)
assert generation.action == "served_fallback" and write_count == 1
for tenant, minutes, approved in [("tenant-b", 15, True), ("tenant-a", 30, True), ("tenant-a", 15, False)]:
    try:
        grant_fixture(caller, tenant, "access-R900-grant", "prod-console", minutes, approved)
    except (PermissionError, ValueError):
        pass
    else:
        raise AssertionError("unsafe write accepted")
assert write_count == 1
print(f"receipt={receipt} writes={write_count} generation={generation.action}")
print("cross-tenant, changed-parameters, and unapproved writes blocked")
Output
receipt=grant-receipt-1 writes=1 generation=served_fallback
cross-tenant, changed-parameters, and unapproved writes blocked

Hand the support agent a policy artifact

A support agent still has to answer after the gateway chooses a lane. It should not recreate those routing rules inside orchestration code. Export a versioned artifact that states the intended requirements and escalation rules. Lane selection and retry behavior stay in the gateway; the agent adds retrieval, tool authorization, and human handoff on top. This fixture artifact isn't approval to deploy: the registry, real adapters, durable accounting, output validation, and representative quality evaluation still need integration evidence.

The final cell emits that handoff: inherited cost contract and schema, demonstrated primary/fallback choices, and explicit attempt, deadline, and spend limits.

13-export-gateway-policy.py
policy_artifact = {
    "policy_id": GATEWAY_POLICY_ID,
    "cost_release_id": COST_RELEASE_ID,
    "required_answer_schema": REQUIRED_ANSWER_SCHEMA,
    "max_generated_answer_usd": str(MAX_GENERATED_ANSWER_USD),
    "retry_limits": {
        "max_generation_attempts": MAX_GENERATION_ATTEMPTS,
        "request_deadline_ms": REQUEST_DEADLINE_MS,
        "request_spend_cap_usd": str(MAX_REQUEST_SPEND_USD),
        "max_attempt_reservation_usd": str(ATTEMPT_RESERVATION_USD),
    },
    "fixture_examples": {
        "public_deploy_policy": "fast-public-json",
        "private_high_risk_access": "primary-private-cited-review",
        "private_high_risk_access_fallback": "local-private-cited-review",
    },
    "escalate_when": [
        "no lane preserves all contract fields",
        "failure occurs after visible output begins",
        "approved private context capacity is exceeded",
        "retry attempts or request deadline are exhausted",
        "request spend reservation is denied",
        "failure is not eligible for transparent generation retry",
        "normalized reply fails the versioned application contract",
    ],
}

print(json.dumps(policy_artifact, indent=2))
Output
{
  "policy_id": "gateway-policy-v1",
  "cost_release_id": "support-release-2026-09-cost-v1",
  "required_answer_schema": "cited-support-answer-v3",
  "max_generated_answer_usd": "0.005000",
  "retry_limits": {
    "max_generation_attempts": 2,
    "request_deadline_ms": 2500,
    "request_spend_cap_usd": "0.010000",
    "max_attempt_reservation_usd": "0.005000"
  },
  "fixture_examples": {
    "public_deploy_policy": "fast-public-json",
    "private_high_risk_access": "primary-private-cited-review",
    "private_high_risk_access_fallback": "local-private-cited-review"
  },
  "escalate_when": [
    "no lane preserves all contract fields",
    "failure occurs after visible output begins",
    "approved private context capacity is exceeded",
    "retry attempts or request deadline are exhausted",
    "request spend reservation is denied",
    "failure is not eligible for transparent generation retry",
    "normalized reply fails the versioned application contract"
  ]
}

Before moving on, change one boundary at a time:

  1. Set the primary duration to 1,400 ms. A 1,100 ms fallback reaches exactly 2,500 ms and is rejected under this fixture's strict-before-deadline rule. At 1,399 ms, it completes at 2,499 ms.
  2. Open all three compatible circuits. No lane should be attempted and spend should stay zero. Closing only regional should permit it without relaxing the contract.
  3. Replace E1 with E99 in a well-formed reply. The response must be blocked even though lane selection passed.
  4. Remove access:grant from the caller and replay the same operation key. Authorization is checked again; a cached receipt isn't permission to perform or disclose a write.
Complete the lesson

Mastery Check

Answer every question, then check your score. Score 75% or higher to mark this lesson complete.

1.An unsafe router returns "local-private" as soon as it sees tenant-private data, before checking whether a break-glass access request carries at least 50,000 cents of risk. For a tenant-private request with 1,200 USD of risk that requires citations, what is the failure?
2.Two lanes can be called through the same OpenAI-shaped chat adapter. A request is a private high-risk break-glass access request that requires citations and structured output. What evidence should the gateway use before admitting either lane?
3.A model has already streamed visible text to the user when the connection drops mid-answer. What should the gateway do?
4.A replay lets a learned router rank only lanes that already passed deterministic filtering. The replay shows every served event had no route-contract violations and evaluated lane cost was at most 0.005000 USD. What conclusion is supported before promotion?
5.A private high-risk break-glass access request's primary private cited-review lane times out before output. A healthy cheap fallback is under budget and allows tenant-private data, but it returns plain text and has no citation or human-review support. Should the gateway use it?
6.A private high-risk break-glass access request has compatible lanes ranked by cost: primary-private, local-private, then regional-private. The primary provider circuit is open, and the local-private circuit is also open during cooldown. Regional-private is healthy and within budget. What should the gateway do?
7.A primary attempt times out, and the provider's usage report later shows it cost 0.003800 USD. A compatible fallback is permitted within the deadline and a 0.010000 USD request cap, with its 0.005000 USD charge bound reserved. The fallback then costs 0.004600 USD. What should accounting record?
8.A tenant-private break-glass access request requires 80,000 context tokens, schema, citations, and human review. Every approved private lane supports at most 64,000 tokens, and no other lane satisfies all contract fields. What should the gateway do?
9.An authorized access write has already succeeded. The model then times out before visible output, and a compatible generation fallback exists. What should the gateway do?
10.A request has a shared 3,000 ms deadline. Its primary attempt times out after 1,700 ms, and there is no retry delay. A compatible fallback needs 1,500 ms. What must the gateway do under the fixture's deadline rule?

10 questions remaining.

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.