LeetLLM
MediumEvaluationPython 3

Canary Metric Judge

Judge canary promotion with traffic, error-rate, latency, and quality thresholds.

35m2 sample tests3 hidden tests

Decide whether a canary release should promote, hold, or roll back.

Requirements

  • Define judge_canary(baseline, candidate, thresholds).
  • Return {"decision": ..., "reasons": [...]}.
  • Decisions are "promote", "hold", or "rollback".
  • If candidate requests are below thresholds["min_requests"], return hold with ["not_enough_traffic"].
  • Roll back if candidate error_rate exceeds max_error_rate.
  • Roll back if candidate latency divided by baseline latency exceeds max_latency_ratio.
  • Roll back if candidate quality delta is below min_quality_delta.
  • Promote only when no rollback reason exists.
  • Reason order: traffic, error, latency, quality.

Example

python
1baseline = {"p95_latency_ms": 100, "quality": 0.8} 2candidate = {"requests": 1000, "error_rate": 0.01, "p95_latency_ms": 110, "quality": 0.81} 3thresholds = {"min_requests": 100, "max_error_rate": 0.05, "max_latency_ratio": 1.5, "min_quality_delta": 0} 4assert judge_canary(baseline, candidate, thresholds)["decision"] == "promote"

Constraints

  • Do not divide by zero.
  • Missing metrics should fail closed with rollback reasons.
  • Keep reason ordering deterministic.

Editor
Results
Run sample tests or submit all tests.