Model Gateways, Routing, and Fallbacks

Your PolicyOps support assistant can price each generated answer. The budget contract says an evaluated answer may cost at most 0.005000 USD and must still keep its evidence. A production app faces the harder routing question: which large language model (LLM) lane may answer each request, and what happens when that lane is unavailable?

A model gateway makes that decision. The app submits one request and one set of requirements. The gateway selects a compatible lane, invokes a model adapter, records why it chose that lane, and either finds a compatible fallback or escalates safely after failure.

The important word is compatible. A private, high-risk break-glass access request needs privacy, human review, cited structured output, and a budget ceiling together. A small gateway can enforce that full contract so code can't discard one requirement while satisfying another.

What belongs at the gateway boundary

In the previous lesson, the cost ledger measured model generation after cache decisions had already been made. The gateway consumes that budget contract alongside product and safety requirements. It doesn't decide whether a stored semantic-cache answer is trustworthy; it decides where a request that still requires generation may run.

Use three terms precisely:

An OpenAI-shaped adapter isn't proof that two lanes have the same contract. Anthropic's OpenAI SDK compatibility documentation describes that path as a comparison/testing option rather than its usual production path, and documents limitations including unsupported prompt caching and an ignored strict tool-calling parameter. Native structured outputs are required for guaranteed schema conformance there.^{[1]Reference 1OpenAI SDK compatibilityhttps://platform.claude.com/docs/en/cli-sdks-libraries/libraries/openai-sdk} A gateway therefore records capabilities explicitly instead of assuming interface similarity means behavior similarity.

The first lab cell defines the artifact arriving from cost engineering, the request facts the app knows, the contract the gateway must compile, and a registry of candidate lanes. The dollar figures are teaching fixtures measured before promotion, not live provider prices. They are preflight evidence, not the final bill: after generation, the gateway still records provider-reported usage and sends it back through the cost ledger.

Compile requirements before choosing a lane

A tempting router is a stack of early returns:

For a private break-glass access request with 900 USD of risk, that code returns from the first branch and silently drops the second requirement. The safe sequence is different:

1. Convert all request facts into one contract.
2. Filter out every lane that violates any contract field.
3. Rank only the remaining lanes by measured preferences such as cost or latency.

The next cell compiles the requirements. A break-glass access request at or above the lesson's policy threshold needs human review; private incident notes independently require a private data lane. Neither check erases the other.

Now the filter can explain every rejection. This is more useful than returning a Boolean: if no lane survives, an operator needs to know whether the missing capability is private-data handling, context length, citations, review, or budget.

Three private cited review lanes survive. The gateway can now choose between them by a soft preference without weakening a hard requirement. Here it chooses the lower measured answer cost, then latency as a deterministic tie-breaker.

A route is a policy decision, not a provider popularity vote

The filter above is deliberately rule-based. Data boundaries, schema requirements, review requirements, and approved cost ceilings are policy constraints. Letting a probabilistic classifier relax any of them would make a plausible prediction more important than authorization.

Learned routing becomes useful inside the feasible set. RouteLLM trains routers from preference data to choose between a stronger and a weaker LLM; its paper reports cost reductions greater than two times in some evaluated settings without a reported response-quality loss on those settings.^{[2]Reference 2RouteLLM: Learning to Route LLMs with Preference Datahttps://arxiv.org/abs/2406.18665} That result supports experimenting with quality-cost routing. It doesn't turn a learned router into a privacy or authorization check.

This flow diagram shows the control boundary. A soft router may rank already-compatible candidates; a failure re-enters the same contract filter rather than jumping directly to a convenient provider.

A low-cost learned router recommends a public lane for a request containing private incident notes. What should happen?

Answer

The deterministic contract filter rejects the lane before any ranking decision can use it. Learned ranking can choose only among lanes already approved for the data boundary and required output contract.

Fallback is another contract decision

A fallback is a second route attempt after the first attempt fails. It isn't permission to drop requirements. If a cited access answer must be private, structured, reviewable, and within budget before an outage, it must remain so during an outage.

Gateway frameworks expose fallback mechanisms, but configuration doesn't prove semantic compatibility. For example, LiteLLM documents ordered fallbacks and separate regular, context-window, and content-policy fallback settings.^{[3]Reference 3Fallbacks (Proxy Reliability)https://docs.litellm.ai/docs/proxy/reliability} Those mechanisms decide when another lane may be attempted. Your contract still decides which replacement lane is acceptable.

Failure type matters before the gateway even looks for a candidate. A rate limit or timeout before any output is shown can be retried against a compatible lane. Once a model has streamed visible text, silently switching models can produce a contradictory continuation. The safer product behavior is to stop that answer and ask the user to retry or hand it to an operator.

This fixture routes answer generation only. It doesn't replay side-effecting tool calls. A write action such as granting break-glass access needs its own authorization boundary and idempotency key; a gateway must not silently execute it again because generation failed. A context rejection also stops here: the filter already checked context capacity, so a provider rejection signals stale registry data or another mismatch that needs investigation rather than a hidden downgrade.

The next cell classifies failures by whether a transparent fallback is still allowed.

For the private break-glass access request, suppose the lower-cost primary lane times out before emitting output. The fallback selector excludes the failed lane, applies the same contract filter, and then chooses from what remains.

Both candidate lanes fit the per-answer admission ceiling, but a failed primary attempt can still consume billable provider work before the timeout reaches the gateway. That means max_answer_cost_usd isn't a guarantee on total request spend during an incident. Keep two numbers separate: the admission ceiling used to filter lanes, and the request-total spend that sums usage from every attempt (failed primary plus fallback). Production policy also needs a retry budget: a maximum attempt count, one wall-clock deadline shared across attempts, and post-call pricing for usage reported by every provider attempt. This lab allows at most one fallback (MAX_GENERATION_ATTEMPTS = 2) and carries a 2.5-second request deadline into the exported policy. The simulator doesn't advance a clock, so the downstream runtime must enforce that deadline.

The primary private lane times out. A cheap lane is healthy and under budget, but it returns plain text without citations or human-review support. Is it an acceptable fallback?

Answer

No. Budget is only one contract field. The gateway must reject the cheap lane because it drops required schema, citations, and review handling.

Stop incidents from multiplying

Fallback can make an outage worse if every request keeps trying an unhealthy primary before moving to the backup. A circuit breaker tracks recent failures for an upstream target. Once that target reaches a failure threshold, its circuit opens and new requests skip it during a cooldown period. After cooldown, one probe is allowed through; success closes the circuit, while failure opens it again.

This compact implementation models those three states: closed, open, and half_open. Its key is provider because each fixture provider names one upstream target. A production registry may need a provider, endpoint, deployment, or lane key depending on failure scope, plus bounded retry attempts, backoff, and a deadline for the half-open probe. Sequential calls already fail closed while a probe is in flight (HALF_OPEN rejects further permits). Concurrent workers racing permit() after cooldown can both enter half-open unless the shared breaker holds a single-flight probe lease.

What should an open circuit breaker do after a provider crosses its failure threshold?

Answer

Stop sending normal traffic to that lane for a bounded period, then allow controlled probes before closing the circuit. Continuing ordinary retries would amplify latency and provider load during the outage.

Execute one outage path and record why

Routing decisions are operational evidence. An audit row should carry the gateway-policy identifier, inherited cost-release identifier, selected lane, fallback event, evaluated cost, and every hard requirement that caused selection. That record lets a later incident review distinguish a model-quality problem from a bad policy decision.

The next cell simulates several attempts for the private break-glass access request. It consults circuit state before attempting the primary lane and records failures when an attempt breaks. A timeout before output uses the compatible local fallback. A mid-stream drop escalates because changing generators after output begins would hide an inconsistent user experience. The final attempt opens both earlier circuits and proves that the gateway scans ranked contract-compatible fallbacks until it finds a permitted regional lane.

Test policy before promoting it

A gateway policy shouldn't be promoted because three hand-picked examples look sensible. Replay an evaluated set that represents low-risk traffic, high-risk cases, private data, outage paths, and unsupported contracts. For every generated response, check that selected lane preserves the contract and stays inside the inherited budget.

This small replay adds a request with a private context too large for either approved private lane. It also opens the primary circuit on a simulated timeout and confirms that the next request uses a contract-preserving fallback during cooldown. An honest gateway escalates unsupported work rather than truncating evidence or routing private incident notes somewhere unapproved.

The replay isn't an answer-quality evaluation by itself. Its cost assertion checks preflight lane evidence, not the final bill. After generation, record provider-reported usage and price it with the inherited rate card in the ledger from the previous lesson. Before promotion, also join these route events with the answer-correctness and citation checks from the evaluation lessons. A routing policy can satisfy the mechanical contract and still send too many difficult public questions to a weak but formally compatible lane.

Only after hard requirements pass should you test learned or heuristic ranking for quality and cost. Useful comparison metrics are generated-answer correctness, citation correctness, human-escalation rate, fallback success rate, p95 latency, and actual spend by lane.

Hand the support agent a policy artifact

The next lesson assembles a complete support agent. It shouldn't re-create routing rules inside orchestration code. The gateway should export a small, versioned policy artifact that states what has been approved and what must escalate.

The final cell emits the artifact from the lesson: an inherited cost contract, two demonstrated routes, and failure behavior that refuses unsafe degradation.