MediumEvaluationPython 3

Canary Metric Judge

Judge canary promotion with traffic, error-rate, latency, and quality thresholds.

35m3 sample tests5 hidden tests

Decide whether a canary release should promote, hold, or roll back.

Requirements

  • Define judge_canary(baseline, candidate, thresholds).
  • Return {"decision": ..., "reasons": [...]}.
  • Decisions are "promote", "hold", or "rollback".
  • If candidate requests are below thresholds["min_requests"], return hold with exactly ["not_enough_traffic"]; rollback reasons are excluded.
  • Roll back if candidate error_rate is missing or exceeds max_error_rate → reason token "error_rate".
  • Roll back if baseline/candidate latency is missing, baseline latency is zero, or candidate latency divided by baseline latency exceeds max_latency_ratio → reason token "latency".
  • Quality is higher-is-better. Define quality_delta = candidate["quality"] - baseline["quality"]. Roll back if either quality is missing or quality_delta < min_quality_delta → reason token "quality".
  • Comparisons are strict: traffic holds when requests < min_requests; error and latency roll back on >; quality rolls back on <. Equality passes each gate.
  • Rollback reason tokens must be exactly "error_rate", "latency", and "quality" (not "error", "latency_ratio", or "quality_delta").
  • Promote only when no rollback reason exists.
  • Reason order: not_enough_traffic (hold only), then rollback reasons in order error_rate, latency, quality.

Example

python
1baseline = {"p95_latency_ms": 100, "quality": 0.8} 2candidate = {"requests": 1000, "error_rate": 0.01, "p95_latency_ms": 110, "quality": 0.81} 3thresholds = {"min_requests": 100, "max_error_rate": 0.05, "max_latency_ratio": 1.5, "min_quality_delta": 0} 4assert judge_canary(baseline, candidate, thresholds)["decision"] == "promote"

Constraints

  • Don't divide by zero.
  • Missing metrics should fail closed with rollback reasons.
  • Keep reason ordering deterministic.

Editor