Skip to content

fix(judge): give a judge outage its own outcome, not JUDGE MISMATCH - #333

Open
AmirF194 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
AmirF194:fix/299-judge-outage-scored-as-failure
Open

fix(judge): give a judge outage its own outcome, not JUDGE MISMATCH#333
AmirF194 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
AmirF194:fix/299-judge-outage-scored-as-failure

Conversation

@AmirF194

Copy link
Copy Markdown

What does this PR do?

Fixes #299, all three asks.

judge.py already returns match=None when the judge never renders a verdict (provider outage, retries exhausted, or an unparseable reply), distinct from match=False (a genuine mismatch). But downstream, run.py's final-status block treats both the same way, sharing exit 1, and batch.py's exit-code map turns exit 1 into job.status = "failed". So a judge-provider outage was indistinguishable from an agent failure in batch-summary.json.

Ask 1: give match=None a distinct outcome

run.py now exits 3 instead of 1 when judge_result.get("match") is None, keeping it separate from a genuine JUDGE MISMATCH (which still exits 1).

Ask 2: count it separately in batch-summary.json

batch.py maps exit 3 to job.status = "judge_inconclusive", its own bucket alongside passed/failed/error/skipped, carried into batch-summary.json's totals and the printed batch summary.

Ask 3: retarget those runs for re-judging

clawbench-rescore now retries a cached verdict whose match is null even without --force, since that cached file never held a scored result. --resume's own skip logic in batch.py is untouched here; it currently keys off batch-log existence rather than the recorded outcome, and reworking that is a larger, separate change.

Corpus

  • v2
  • v1
  • both
  • not applicable

Host-side runner change; no task data involved.

Test plan

  • tests/test_run_judge_stage.py::test_judge_failure_after_the_run_still_writes_run_meta, updated: a judge_setup_failed outcome now exits 3, not 1. New test_judge_mismatch_keeps_exit_code_1_distinct_from_inconclusive locks in that a real match=False still exits 1.
  • tests/test_batch_judge_inconclusive.py, new: run_job() against a mocked subprocess maps exit 0/1/3/other to passed/failed/judge_inconclusive/error; write_summary_json() counts judge_inconclusive separately from failed in batch-summary.json.
  • tests/test_rescore_retarget.py, new: rescore_one() retries a cached match: null verdict without --force, and still trusts a cached scored verdict (match: true/false) without retrying it.
  • Full suite in a clean python:3.11-slim container via uv run --frozen pytest: 226 passed (218 on main, +8 from this PR). All new/changed tests fail on main and pass on this branch, confirmed both ways in the same container.
  • ruff check ., ruff format --check ., pyright src/clawbench tests, uv build + twine check dist/*, and the three --help console-script checks from static-check.yml: all clean. validate-docs.yml's link checker (triggered by the CHANGELOG.md edit): clean.

Not verified: no live judge-provider outage was reproduced end-to-end; the fix is proven through judge.py's existing judge_call_failed/unsupported_api_type fallback plus the tests above, not a real API timeout.

Related issues

Fixes #299.

judge.py already returns match=None when the judge never rendered a
verdict (provider outage, retries exhausted, or an unparseable reply).
run.py folded that into the same exit 1 as a genuine JUDGE MISMATCH, and
batch.py's exit-code map then counted it as job.status = "failed", so a
judge-provider outage was indistinguishable from an agent failure in
batch-summary.json.

run.py now exits 3 for match=None, kept separate from exit 1 (a real
mismatch). batch.py gives exit 3 its own job.status = "judge_inconclusive"
bucket, counted separately in batch-summary.json's totals instead of
folded into "failed". clawbench-rescore now retries a cached match: null
verdict even without --force, since that cached file never held a scored
result.

Fixes TIGER-AI-Lab#299
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge outages are scored as agent failures (match=None → exit 1 → 'failed')

1 participant