Model Fallback Router

Choose the best healthy model candidate for a request using a routing policy.

Requirements

• Define choose_model(request, candidates).
• request has capabilities, tokens, and an optional max_latency_ms.
• Each candidate has id, healthy, capabilities, remaining_tokens, latency_ms, and cost.
• A candidate is eligible only if:
  • it's healthy,
  • it has every required capability,
  • it has enough remaining tokens,
  • it satisfies max_latency_ms when present.
• Choose the eligible candidate with lowest cost.
• Ties use lower latency_ms, then id alphabetically.
• Return the chosen model ID, or None.

Example

The unhealthy primary is ineligible. Of the two healthy models, the lower-cost candidate wins even though another candidate is faster.

request = {"capabilities": ["tool_use"], "tokens": 100}
models = [
    {"id": "primary", "healthy": False, "capabilities": ["tool_use"], "remaining_tokens": 500, "latency_ms": 10, "cost": 0},
    {"id": "fast", "healthy": True, "capabilities": ["tool_use"], "remaining_tokens": 500, "latency_ms": 20, "cost": 3},
    {"id": "cheap", "healthy": True, "capabilities": ["tool_use"], "remaining_tokens": 200, "latency_ms": 80, "cost": 1},
]
assert choose_model(request, models) == "cheap"

Constraints

• Don't mutate candidates.
• Use deterministic tie-breaking.
• Missing max_latency_ms means no latency ceiling.
• Inputs contain all required fields. Capability names and model IDs are case-sensitive strings; IDs are unique. Token counts, latency, and cost are finite nonnegative numbers (token counts are integers).
• max_latency_ms may also be None, meaning no ceiling. A ceiling of 0 is an actual limit. Equality at the token and latency boundaries is eligible.
• tokens is the complete token reservation supplied by the caller. Selection only reads quota; it does not reserve tokens or call a provider. cost is a comparable cost estimate for this request.