LeetLLM
MediumServingPython 3

Model Fallback Router

Route requests to the cheapest eligible healthy model with capability and quota checks.

30m3 sample tests6 hidden tests

Choose the best healthy model candidate for a request using a .

Requirements

  • Define choose_model(request, candidates).
  • request has capabilities, tokens, and an optional .
  • Each candidate has id, healthy, capabilities, remaining_tokens, latency_ms, and cost.
  • A candidate is eligible only if:
    • it's healthy,
    • it has every required capability,
    • it has enough remaining tokens,
    • it satisfies max_latency_ms when present.
  • Choose the eligible candidate with lowest cost.
  • Ties use lower latency_ms, then id alphabetically.
  • Return the chosen model ID, or None.

Example

The unhealthy primary is ineligible. Of the two healthy models, the lower-cost candidate wins even though another candidate is faster.

python
1request = {"capabilities": ["tool_use"], "tokens": 100} 2models = [ 3 {"id": "primary", "healthy": False, "capabilities": ["tool_use"], "remaining_tokens": 500, "latency_ms": 10, "cost": 0}, 4 {"id": "fast", "healthy": True, "capabilities": ["tool_use"], "remaining_tokens": 500, "latency_ms": 20, "cost": 3}, 5 {"id": "cheap", "healthy": True, "capabilities": ["tool_use"], "remaining_tokens": 200, "latency_ms": 80, "cost": 1}, 6] 7assert choose_model(request, models) == "cheap"

Constraints

  • Don't mutate candidates.
  • Use deterministic tie-breaking.
  • Missing max_latency_ms means no latency ceiling.

Editor