MediumEvaluationPython 3
Canary Metric Judge
Judge canary promotion with traffic, error-rate, latency, and quality thresholds.
35m3 sample tests5 hidden tests
Decide whether a canary release should promote, hold, or roll back.
Requirements
- Define
judge_canary(baseline, candidate, thresholds). - Return
{"decision": ..., "reasons": [...]}. - Decisions are
"promote","hold", or"rollback". - If candidate requests are below
thresholds["min_requests"], return hold with exactly["not_enough_traffic"]; rollback reasons are excluded. - Roll back if candidate
error_rateis missing or exceedsmax_error_rate→ reason token"error_rate". - Roll back if baseline/candidate latency is missing, baseline latency is zero, or candidate latency divided by baseline latency exceeds
max_latency_ratio→ reason token"latency". - Quality is higher-is-better. Define
quality_delta = candidate["quality"] - baseline["quality"]. Roll back if either quality is missing orquality_delta < min_quality_delta→ reason token"quality". - Comparisons are strict: traffic holds when
requests < min_requests; error and latency roll back on>; quality rolls back on<. Equality passes each gate. - Rollback reason tokens must be exactly
"error_rate","latency", and"quality"(not"error","latency_ratio", or"quality_delta"). - Promote only when no rollback reason exists.
- Reason order:
not_enough_traffic(hold only), then rollback reasons in ordererror_rate,latency,quality.
Example
python
1baseline = {"p95_latency_ms": 100, "quality": 0.8}
2candidate = {"requests": 1000, "error_rate": 0.01, "p95_latency_ms": 110, "quality": 0.81}
3thresholds = {"min_requests": 100, "max_error_rate": 0.05, "max_latency_ratio": 1.5, "min_quality_delta": 0}
4assert judge_canary(baseline, candidate, thresholds)["decision"] == "promote"Constraints
- Don't divide by zero.
- Missing metrics should fail closed with rollback reasons.
- Keep reason ordering deterministic.
Editor