Model Gateways, Routing, and Fallbacks

The primary model times out on access-R900. A cheap backup is healthy, but it can't return the cited, structured answer that this private access request needs. Returning something quickly would turn a provider outage into an application error.

The cost engineering lesson supplied a fictional release support-release-2026-05-cost-v1, schema cited-support-answer-v3, and a 0.005000 USD single-generation ceiling. A model gateway can apply these requirements when choosing a large language model (LLM) path. The app sends a stable internal request; the gateway filters approved paths, invokes an adapter, checks the result, and records the decision. Privacy and human review also require authorization and workflow controls around that model call.

Compatibility is the constraint that makes every later choice meaningful. The gateway must enforce the full contract so application code can't satisfy one field by dropping another.

What belongs at the gateway boundary

An adapter gives application code a stable internal schema while translating provider wire formats. You can put that adapter inside a service or behind a shared gateway. A separate gateway centralizes credentials, routing, accounting, and telemetry, but also adds a dependency, latency, and an operational failure boundary.

For example, native schema configuration differs across APIs. These are request fields checked September 21, 2026, not interchangeable SDK helper arguments:

Claude also offers strict tool inputs; tools aren't the only route to structured output. Both providers document supported schema subsets and exceptions such as refusals and incomplete output. Keep application validation even when constrained decoding is enabled.^{[1]Reference 1Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs[2]Reference 2Structured outputshttps://platform.claude.com/docs/en/build-with-claude/structured-outputs}

Cost engineering separated generation cost from cache decisions. Now the gateway consumes that budget contract alongside product and safety requirements. A stored semantic-cache hit ends before this decision, but still needs its own authorization and freshness checks. The gateway decides where one generated support request may run.

Keep three terms separate:

An OpenAI-shaped adapter only proves that a request can be translated. It doesn't prove that two lanes share a contract. Anthropic describes its OpenAI SDK compatibility layer as a way to test and compare Claude, not as a long-term production path. In that layer, prompt caching isn't supported, response_format is ignored, and the strict tool-calling flag is ignored too. For guaranteed schema conformance, Anthropic points you to native structured outputs on the Claude API.^{[3]Reference 3OpenAI SDK compatibilityhttps://platform.claude.com/docs/en/cli-sdks-libraries/libraries/openai-sdk}

Record capabilities on each lane. Don't infer them from a familiar URL shape. Here, citation support means preserving application evidence identifiers in JSON for subsequent checking. It doesn't guarantee claim support or enable a provider's native citation blocks: Claude currently rejects native citations combined with output_config.format.^{[2]Reference 2Structured outputshttps://platform.claude.com/docs/en/build-with-claude/structured-outputs} Test capability combinations, not just each Boolean in isolation. Human review belongs to the surrounding workflow. A lane can prepare a reviewable answer; it can't approve its own access grant.

Before routing access-R900, inventory the cost artifact, trusted request facts, a compiled contract, and lane capability records. All lanes, capabilities, costs, and timings below are teaching fixtures, not measurements or provider prices. This CPU lab neither authenticates requests nor runs adapters, reviews answers, creates a human queue, or enforces live spending. A deployed registry needs retained evaluations tied to model, adapter, schema, policy, and registry revisions. Price each attempt using its own effective model/tier/region rate card; the preceding lesson's GPT-5.4 row doesn't price every fallback.

Run the numbered Python cells in order using Python 3.10 or later; they share state and need no network or third-party packages. The first examples select routes. Later cells simulate attempts, advance a clock, and reject invalid replies. No example grants real access or calls a model.

Compile requirements before choosing a lane

The bug fits in two early returns:

For a private break-glass access request with 900 USD of risk, the first return silently drops the second requirement. Give each stage one job:

1. Convert every request fact into one contract.
2. Drop every lane that violates any contract field.
3. Rank only the remaining lanes by measured preferences such as cost or latency.

Predict access-R900 before running the compiler: its risk requires human review, its private notes require a private lane, and its answer still needs cited-support-answer-v3. The next cell compiles those requirements. Neither check erases the other, and needs_schema means that exact schema, not "any JSON."

These records come from a trusted application boundary, not unchecked JSON from a client. Authenticate the caller, authorize the task and tenant, and derive classification and risk from server-side policy before constructing one. A client-supplied data_class="public" can't declassify private notes. The compiler below also forces private handling and citations for the access task, even if those flags are weakened accidentally.

The single context_tokens count is a fixture simplification. Tokenizers and prompt wrappers differ by lane. At dispatch, tokenize each candidate's complete input, including tools and evidence, using that lane's tokenizer; reserve output and other tokens against its actual limits. Recheck region, retention policy, credential scope, and registry revision too. A private-data Boolean can't express all those conditions.

The output preserves every requirement. Now make each lane explain its rejection instead of returning a bare Boolean. If no lane survives, an operator needs to know whether the missing capability is private-data handling, context length, citations, review, or budget.

Before reading the rows, predict the result for access-R900: three private cited-review lanes should survive; cheap text should fail schema, citations, and review.

Three private cited-review lanes survive the supplied fields. Cheap text looks tempting: its declared data classes include tenant-private and its cost estimate is below 0.005000 USD. It still fails schema, citations, and review. An estimated cost isn't a charge bound; the later executor demonstrates reservations separately.

💡 Key insight: Budget and a private-data flag aren't a contract. cheap-text-fallback fails three hard fields at once.

The gateway can rank survivors by a soft preference without weakening any requirement. Here it chooses lower fixture answer cost, then latency as a deterministic tie-breaker.

Expect docs-Q102 to take the fast public lane and access-R900 to take the lowest-cost private cited-review lane. The next cell makes that choice explicit and records its reasons.

Let learned routing rank only legal lanes

The filter above is deliberately rule-based. Data boundaries, schema requirements, review requirements, and approved cost ceilings are policy constraints. A probabilistic classifier that relaxes any of them would make a plausible prediction more important than authorization.

Learned routing still has a job inside the feasible set. Consider three preferences:

• Task quality: Evaluate candidate models on your actual lookup, reconciliation, and reasoning cases. Parameter count or a "frontier" label doesn't establish the best route. RouteLLM learns from preference data and reports more than a twofold cost reduction in certain benchmark cases without reducing its measured quality. That isn't a guarantee for this private access workflow or a new model pair.^{[4]Reference 4RouteLLM: Learning to Route LLMs with Preference Datahttps://arxiv.org/abs/2406.18665}
• Latency: Set time-to-first-token (TTFT) and complete-answer targets for the product, then measure realistic load. There's no universal sub-500 ms requirement or hosted-only solution. Batch work has a different completion window; minutes of acceptable delay alone don't make a workload eligible for a 24-hour Batch path.
• Spend and quotas: Metering helps choose a cheaper approved lane, defer work, or reject it under a declared policy. A 90% budget warning is a policy choice, not a hard cap. Reservations, settlement, and atomic account-level admission are what prevent concurrent calls from overspending an enforceable bound.

Treat learned routers and heuristics as ranking engines over already-legal lanes. They're never a replacement for privacy, context, or schema validation gates.

The control boundary has two paths. A soft router may rank already-compatible candidates. A failure re-enters the same contract filter instead of jumping to a convenient provider.

Make fallback another contract decision

When a primary lane fails before output, the gateway may get a second chance to generate. It doesn't get a second contract. A cited access answer must remain private, structured, reviewable, and admitted under the same cost policy during an outage.

Gateway products expose fallback mechanisms, but a config file doesn't prove semantic compatibility. LiteLLM documents ordered fallbacks and separate chains for ordinary errors, context-window errors, and content-policy errors. Those knobs decide when another model group may be tried; your contract still decides which replacement is acceptable. Its allowed_fails and cooldown_time settings can temporarily remove failing deployments. One documented exception matters: fallback to a specific model ID bypasses cooldown checks for that replacement. Test the exact configured path before relying on its breaker behavior.^{[5]Reference 5Fallbacks (Provider Failover)https://docs.litellm.ai/docs/proxy/reliability}

⚠️ Common mistake: Copying an ordered fallback list from a proxy config and treating every backup as legal. The proxy will happily fail over. The contract still has to admit that lane.

Classify the failure before the gateway looks for a candidate. A transient rate limit or timeout before output is shown may be retried against a compatible lane if attempts, time, and spend remain. A quota-exhausted account isn't the same as a temporary throttle. Authentication failures and policy refusals stop here; switching credentials or providers to bypass them isn't recovery. Once a model has streamed visible text, silently switching models can produce a contradictory continuation. Stop that answer and expose an explicit retry or handoff.

This lab retries answer generation separately from tool writes. Generation isn't generally a pure deterministic function: sampling can vary, and hosted state, storage, or enabled tools can have effects beyond token production. A retry requires a known generation-only path and compatible conversation/input state.

Suppose an access-grant tool commits a write, then the explanation times out. Replaying the whole turn can duplicate the grant. Preserve its verified receipt, recheck authorization, and retry only the explanation when allowed. Write services need scoped operation identities, parameter comparison, retained results, and atomic commit or reconciliation. A pre-output timeout alone says nothing about whether a write committed.^{[6]Reference 6Making retries safe with idempotent APIshttps://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/}

A context rejection also stops here. The filter already checked capacity, so a provider rejection means stale registry data or another mismatch. Investigate it instead of hiding it behind a downgrade.

Some production gateways retry a different schema-capable lane after a validation miss, as long as the caller never saw the bad payload. This lab escalates schema_invalid so a broken supports_schema flag can't hide behind another attempt. Cheap text is illegal either way.

The next cell makes the transparent-retry boundary explicit:

For access-R900, suppose the lower-cost primary lane times out before emitting output. The fallback selector excludes the failed lane, applies the same contract filter, and chooses from what remains.

Predict the result before looking at the output: local private should serve, regional private should remain available, and cheap text should still be rejected.

Both backup estimates fit the single-generation ceiling. That doesn't bound a future call's bill, and a failed primary can still consume work before its timeout reaches the gateway. A maximum for one call also doesn't cap cumulative task spending.

Keep two numbers separate: the admission ceiling used to filter lanes, and the request-total spend that sums usage from every attempt, failed primary plus fallback. Production policy also needs a retry budget: a maximum attempt count, one wall-clock deadline shared across attempts, and post-call pricing for usage reported by every provider attempt.

A critical invariant across sequential attempts is cumulative deadline propagation: $T_{\text{remaining}} = T_{\text{total}} - T_{\text{spent}}$

With a 2,500 ms total deadline and 1,800 ms already spent, only 700 ms remain. Giving fallback a fresh 2,500 ms timeout would permit 4,300 ms of attempt time, exceeding the original deadline. Real elapsed time also includes queueing, policy checks, retries, and adapter overhead. Use one monotonic deadline and carry the remaining budget into cancellation and timeout handling. A socket read timeout alone may reset between chunks; it isn't necessarily a total-call deadline, and local cancellation doesn't prove remote work stopped.

This lab allows at most one fallback (MAX_GENERATION_ATTEMPTS = 2) within a shared 2.5-second deadline. The executor below advances a simulated clock, including retry delay. If the primary consumes 1,800 ms, only 700 ms remain, not another 2,500 ms.

A hard request-spend cap needs more than estimated lane cost: reserve a conservative charge bound before each attempt, reconcile actual usage, and retain reservations for timed-out calls whose usage is unknown. Concurrent requests need atomic reservations at the tenant/account boundary. The fixture uses a separate 0.010000 USD request cap and a 0.005000 USD reservation per attempt. Real bounds must include all billable token categories and enabled tools; an estimate or typical cost isn't such a bound.

There's another outage cost. If every request still tries the sick primary first, you pay for that failed attempt over and over. The next section stops that herd.

Keep an outage from multiplying

The fallback path protects one request. Under load, every request can still poke an unhealthy primary before moving to the backup. When an upstream provider suffers degraded capacity, multiple concurrent requests time out simultaneously. If clients retry naively without backoff or coordination, traffic spikes multiplicatively against an already struggling upstream, turning a temporary slowdown into a total outage.

To break that feedback loop, resilient gateways combine two core patterns:

• Jittered exponential backoff: Fixed retry intervals cause clients to synchronize their retry attempts into destructive pulses. Full jittered backoff distributes delays randomly between zero and an exponential ceiling: $t_{\text{sleep}} = \text{random}(0, \min(T_{\text{max}}, T_{\text{base}} \cdot 2^{\text{attempt}}))$ Jitter reduces synchronized retry clusters; it doesn't guarantee smooth traffic or solve overload by itself. The exponential index starts at zero for the first retry in this convention.^{[7]Reference 7Exponential Backoff And Jitterhttps://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/}
• Circuit breakers: A circuit breaker suppresses normal calls after chosen health failures cross a threshold. A compatible healthy replacement may exist; otherwise escalate. After cooldown, this policy allows one probe. Other implementations allow a bounded probe set. Probe success can close the circuit; failure reopens it. Define health failures separately from authentication errors, refusals, or unsupported contracts.

This compact implementation models the three states: closed, open, and half_open. Its key is provider because each fixture provider names one upstream target.

A production registry may need provider, endpoint, deployment, or lane keys depending on failure scope. This local breaker rejects malformed clocks and expires an unanswered probe. It has no synchronization or result lease: concurrent workers can race, and an old completion can overwrite newer state. Shared breakers need atomic probe leases, generations that reject stale results, bounded backoff, and actual probe cancellation. A cooldown timer alone isn't proof that an upstream recovered.

Execute one outage path and record why

To investigate an incident, retain enough evidence to reconstruct lane selection: policy and registry revisions, approved cost artifact, effective lane/model/adapter, contract fields, attempt/task identities, status, and pricing evidence. A string ID or an audit row alone doesn't prove that the decision or its provenance is truthful.

That record lets a later review distinguish a model-quality problem from a bad policy decision. The next cell follows one private request through timeout, partial output, and circuit cooldown.

The executor consumes scripted outcomes instead of making HTTP calls. Its defaults assume successful completion with fixed durations and charges; failure overrides let you break that assumption. Each attempted lane advances the simulated clock and contributes its supplied charge. An open circuit consumes no provider attempt or charge in this fixture. served means transport completion only; no payload has yet passed reply validation or human review.

In a real adapter, propagate the remaining deadline to connection, read, and total-call timeouts; disable hidden SDK retries or charge them to the same attempt budget. Cancellation doesn't prove the remote provider stopped generating or billing. Honor applicable Retry-After delays and use bounded backoff with jitter when retrying under load. The fixture accepts an explicit delay so that boundary is testable.^{[8]Reference 8Timeouts, retries, and backoff with jitterhttps://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter}

Replay policy before promoting it

Three hand-picked examples can't establish a gateway policy. Start with a mechanical replay of low-risk traffic, high-risk cases, private data, outage paths, and unsupported contracts. For every simulated success, check that the selected lane preserves the contract and stays inside the admission ceiling. That doesn't establish answer quality.

The replay below adds a request with a private context too large for any approved private lane. It also opens the primary circuit on a simulated timeout, then checks that the next request uses a contract-preserving fallback during cooldown. Unsupported work escalates instead of silently truncating evidence or sending private notes somewhere unapproved.

The replay reads no answers. Its cost assertion compares supplied lane estimates; it doesn't verify a future charge bound or a provider bill. The zero violation count covers these five mechanical cases, not population safety.

After each real attempt, normalize its usage and price it with that lane's effective model/tier/region rate card using the preceding lesson's ledger mechanics. Pin the card's contents and retain unknown usage as unresolved. Cross-provider fallback neither transfers a prefix cache nor guarantees the replacement has no independently warm prefix. Before promotion, join route events with answer correctness, citation support, and measured spending.

A routing policy can satisfy the mechanical contract and still send too many difficult public questions to a weak but formally compatible lane.

Only after hard requirements pass should you test learned or heuristic ranking for quality and cost. Useful comparison metrics are generated-answer correctness, citation correctness, human-escalation rate, fallback success rate, p95 latency, and actual spend by lane.

Now break the runtime, not just the route table. Each case starts with a fresh breaker. Two failed attempts must stop, a slow primary must leave less time for its fallback, and a request cap must reject a reservation even when each lane passes admission.

The 0.008000 USD case stops after spending 0.004200 USD: another 0.005000 USD reservation would exceed the cap. Knowing a fallback's typical charge is 0.004500 USD doesn't make it safe to reserve less than the bound. All charges are known in this fixture; unknown real usage must stay reserved until reconciled.

Validate the reply, not just the lane

A green lane row isn't a green answer. Adapters must normalize provider-specific text blocks, tool-call identifiers, stop reasons, refusals, and usage into an explicit application result. A truncated JSON object isn't success. Neither is a valid object containing a made-up citation or a model-authored claim that a human approved it.

The next cell checks a small local reply shape after adapter normalization. It isn't the full inherited cited-support-answer-v3 schema. We assume the application supplies currently authorized evidence IDs for this request; this function doesn't resolve permissions or fetch passages. Membership catches unknown IDs, not unsupported wording. Likewise, review="pending" is an expected marker, not proof of an actual review queue or human approval.

Before publishing, apply the complete versioned schema, factual/citation checks, and current workflow state. Returning a copied local result prevents later mutation of the caller's citation list from changing this validated result; it doesn't certify its claims. Keep provider credentials server-side and scoped to approved deployments. A valid provider key doesn't authorize a user to read another tenant's evidence or invoke an access tool. Audit records need principal/tenant and policy references, restricted access and retention, and appropriate redaction; never include credentials.

A generation retry must leave a completed write alone

Suppose a separately authorized human approved the access grant and the write succeeded, but the explanation timed out. Retrying generation should consume the recorded receipt. It shouldn't execute the grant again.

This in-memory service uses fake identities and receipts. It demonstrates repeated-key handling and a generation simulation that never invokes the write function. It neither authenticates a reviewer nor attaches a receipt to a real model prompt. A real approval must bind the operation's tenant, principal, target, duration, policy, and lifetime; a Boolean alone isn't that artifact.

The local ledger binds the key to tenant, principal, and exact parameters, rejecting changed intent. Production also needs retained evidence and a durable atomic write-plus-idempotency transaction, or reconciliation when an external commit can't share that transaction. Define retention and late-retry behavior; forgetting a key can allow a later duplicate write.^{[6]Reference 6Making retries safe with idempotent APIshttps://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/}

Hand the support agent a policy artifact

The next lesson builds a complete support agent, but it shouldn't recreate routing rules inside orchestration code. Export a versioned artifact that states the intended requirements and escalation rules. Lane selection and retry behavior stay in the gateway; the agent adds retrieval, tool authorization, and human handoff on top. This fixture artifact isn't approval to deploy: the registry, real adapters, durable accounting, output validation, and representative quality evaluation still need integration evidence.

The final cell emits that handoff: inherited cost contract and schema, demonstrated primary/fallback choices, and explicit attempt, deadline, and spend limits.

Before moving on, change one boundary at a time:

1. Set the primary duration to 1,400 ms. A 1,100 ms fallback reaches exactly 2,500 ms and is rejected under this fixture's strict-before-deadline rule. At 1,399 ms, it completes at 2,499 ms.
2. Open all three compatible circuits. No lane should be attempted and spend should stay zero. Closing only regional should permit it without relaxing the contract.
3. Replace E1 with E99 in a well-formed reply. The response must be blocked even though lane selection passed.
4. Remove access:grant from the caller and replay the same operation key. Authorization is checked again; a cached receipt isn't permission to perform or disclose a write.