fix(batch): resume from run-meta.json, not from log-file existence

[vaibhavdabas16]

What does this PR do?

Fixes #297 — all three asks.

--resume decided a job was finished if batch-logs/<case>-<model>.log existed. That log is opened when a job starts (batch.py:320), so resume skipped every job that was ever attempted — including the ones killed mid-run. That inverts the flag: after a network blip or an OOM takes out 20 of 130 tasks, --resume skips exactly those 20 and reports the batch complete.

The second effect destroyed data rather than just omitting it. write_summary_json rewrites batch-summary.json from the in-memory job list, and every carried-over job was written as status: "skipped", duration_seconds: 0. A resumed batch reported none of the results it already had, in the artifact the per-run stats and the HuggingFace upload consume.

Ask 1 — decide completion from run-meta.json

run.py writes run-meta.json for any run that got far enough to have an outcome, failures included — the invariant #303, #302 and #314 each protect. load_recorded_runs() indexes those by (test_case, model), read from the metadata rather than the path, so:

• a model name containing / matches its job even though the run directory sanitises it to --;
• the newest run wins when repeated resumes have left several directories for one case × model pair;
• a truncated run-meta.json counts as an unknown outcome and the job runs again, rather than having a result invented for it. fix(batch): one truncated run-meta.json no longer discards the batch summary #312/Fix/303 write batch summary json before printing stats #325 established that this file can be truncated.

A job with no recorded outcome stays pending and runs again. That is the whole fix for ask 1: "has a log" becomes "has a result".

Ask 2 — --retry-failed

Re-runs jobs whose recorded outcome was an infra-class failure, using the scorer's own NON_MODEL_FAILURE_CATEGORIES (infra_failure, api_or_credit, task_data, build_instruction). Those are already the categories excluded from adjusted scoring, so such a run carries no signal about the model and a re-run is the only way to get one.

One judgment call worth your eye: genuine model failures (model_not_intercepted) are kept, not retried — the model was asked and it failed, so a re-run buys the same answer at full agent cost. The issue says "failure/infra error", which could be read either way. Say the word and I'll widen it to retry those too, or split it into two flags.

Ask 3 — merge prior statuses into the summary

A carried-over job keeps its recorded result — passed / failed / error — with its real duration, so the rewritten batch-summary.json preserves the original tallies instead of zeroing them.

Jobs are marked resumed, which does two things: the scheduler still skips executing them (it previously keyed off status == "skipped", which is no longer what a carried-over job looks like), and a reader of batch-summary.json can tell a carried-over row from one this invocation produced. That adds one key per job to the summary — flagging it since that file is consumed downstream, though it is additive and existing keys are unchanged.

Corpus

• v2
• v1
• both
• not applicable

_{Host-side batch driver; no task data involved.}

Test plan

• New tests/test_batch_resume.py, 17 tests. This path had no coverage — every existing test passes resume=None, which batch --resume skips failed jobs (log-file heuristic) and overwrites batch-summary.json with 'skipped' rows #297 notes and test(resilience): verify resume works — auto-restart on abort + alerting #160 asks to fix.
• Control, because tests written after a fix prove nothing. I swapped the log-existence heuristic back into apply_resume, held the tests constant, and re-ran: 8 of the 17 fail, including the two that matter most — a started-but-unfinished job being skipped, and the summary losing its tallies.
• Covered: a job with a start log but no metadata runs again; a finished job does not; carried-over status and duration survive into batch-summary.json (totals stay passed: 1, failed: 1 rather than collapsing to skipped: 2); the outcome→status mapping; missing/unparsable duration_seconds; --retry-failed on and off; newest-run-wins; truncated metadata; a / in the model name; an empty or absent output directory.
• Full suite: 227 passed, 4 skipped. The 4 deselected are test_host_tasks.py::test_checked_task_json_files_parse_and_validate, which fails only on this Windows checkout — the task files are git symlinks (mode 120000) checked out as text — and is green on CI.
• ruff check reports the same 8 pre-existing findings on batch.py as main does — no new ones. ruff format --check clean. pyright --pythonplatform Linux (matching the ubuntu runner) 0 errors.
• batch.py must stay importable without a container engine (test_batch_stays_importable_without_a_container_engine). The new import is run_support.results, which does not pull in run_support.config; verified, and that test still passes.
• Merge-clean against main and against all five of my other open PRs — including fix(runner): bound container wait so one wedged run can't stall a batch #316, which touches the same file.

Not verified: no live batch run — I don't have Docker on this machine. The resume decision logic is exercised against real-shaped run-meta.json fixtures on disk, not against an interrupted 130-task sweep.

Related issues

Fixes #297. Related: #160 (resume/resilience testing) asks for exactly this path to be verified end-to-end; this adds the unit-level half.

_{docs/cli.md's --resume row described skipping "finished runs", which was the intent but not the behaviour — updated to match what the code now does, plus a row for --retry-failed.}