EasyEvaluationPython 3
Eval Failure Aggregator
Aggregate eval run rows into stable pass-rate and error diagnostics.
25m3 sample tests6 hidden tests
Summarize model evaluation runs into pass rate, failing cases, and top errors.
Requirements
- Define
summarize_runs(runs). - Each run is a dict with
case_id,passed, and optionalerror. An optionalcategoryfield may appear on runs; ignore it in this problem (used only in follow-ups). - Return a dict with:
totalpassedpass_ratefailing_casestop_errors
failing_casescontains unique failed case IDs sorted alphabetically.top_errorscontains(error, count)pairs for failed rows with a non-empty error.- Ignore error strings on passing rows; only failed attempts contribute to error counts.
- Sort top errors by count descending, then error alphabetically.
- For no runs,
pass_rateis0.
Example
python
1runs = [
2 {"case_id": "a", "passed": True},
3 {"case_id": "b", "passed": False, "error": "timeout"},
4]
5assert summarize_runs(runs) == {
6 "total": 2,
7 "passed": 1,
8 "pass_rate": 0.5,
9 "failing_cases": ["b"],
10 "top_errors": [("timeout", 1)],
11}Constraints
- Don't mutate input runs.
- Count each run toward
total, even repeated case IDs. - Count unique case IDs only in
failing_cases.
Editor