fix(judge): give a judge outage its own outcome, not JUDGE MISMATCH - #333
Open
AmirF194 wants to merge 1 commit into
Open
fix(judge): give a judge outage its own outcome, not JUDGE MISMATCH#333AmirF194 wants to merge 1 commit into
AmirF194 wants to merge 1 commit into
Conversation
judge.py already returns match=None when the judge never rendered a verdict (provider outage, retries exhausted, or an unparseable reply). run.py folded that into the same exit 1 as a genuine JUDGE MISMATCH, and batch.py's exit-code map then counted it as job.status = "failed", so a judge-provider outage was indistinguishable from an agent failure in batch-summary.json. run.py now exits 3 for match=None, kept separate from exit 1 (a real mismatch). batch.py gives exit 3 its own job.status = "judge_inconclusive" bucket, counted separately in batch-summary.json's totals instead of folded into "failed". clawbench-rescore now retries a cached match: null verdict even without --force, since that cached file never held a scored result. Fixes TIGER-AI-Lab#299
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #299, all three asks.
judge.pyalready returnsmatch=Nonewhen the judge never renders a verdict (provider outage, retries exhausted, or an unparseable reply), distinct frommatch=False(a genuine mismatch). But downstream,run.py's final-status block treats both the same way, sharing exit 1, andbatch.py's exit-code map turns exit 1 intojob.status = "failed". So a judge-provider outage was indistinguishable from an agent failure inbatch-summary.json.Ask 1: give match=None a distinct outcome
run.pynow exits 3 instead of 1 whenjudge_result.get("match") is None, keeping it separate from a genuineJUDGE MISMATCH(which still exits 1).Ask 2: count it separately in batch-summary.json
batch.pymaps exit 3 tojob.status = "judge_inconclusive", its own bucket alongsidepassed/failed/error/skipped, carried intobatch-summary.json'stotalsand the printed batch summary.Ask 3: retarget those runs for re-judging
clawbench-rescorenow retries a cached verdict whosematchisnulleven without--force, since that cached file never held a scored result.--resume's own skip logic inbatch.pyis untouched here; it currently keys off batch-log existence rather than the recorded outcome, and reworking that is a larger, separate change.Corpus
Host-side runner change; no task data involved.
Test plan
tests/test_run_judge_stage.py::test_judge_failure_after_the_run_still_writes_run_meta, updated: ajudge_setup_failedoutcome now exits 3, not 1. Newtest_judge_mismatch_keeps_exit_code_1_distinct_from_inconclusivelocks in that a realmatch=Falsestill exits 1.tests/test_batch_judge_inconclusive.py, new:run_job()against a mocked subprocess maps exit 0/1/3/other topassed/failed/judge_inconclusive/error;write_summary_json()countsjudge_inconclusiveseparately fromfailedinbatch-summary.json.tests/test_rescore_retarget.py, new:rescore_one()retries a cachedmatch: nullverdict without--force, and still trusts a cached scored verdict (match: true/false) without retrying it.python:3.11-slimcontainer viauv run --frozen pytest: 226 passed (218 onmain, +8 from this PR). All new/changed tests fail onmainand pass on this branch, confirmed both ways in the same container.ruff check .,ruff format --check .,pyright src/clawbench tests,uv build+twine check dist/*, and the three--helpconsole-script checks fromstatic-check.yml: all clean.validate-docs.yml's link checker (triggered by theCHANGELOG.mdedit): clean.Not verified: no live judge-provider outage was reproduced end-to-end; the fix is proven through
judge.py's existingjudge_call_failed/unsupported_api_typefallback plus the tests above, not a real API timeout.Related issues
Fixes #299.