fix(runner): bound container wait so one wedged run can't stall a batch

[vaibhavdabas16]

What does this PR do?

Fixes #298 — both asks.

The only time limit lived inside the container (entrypoint.sh:251, MAX_WAIT=${TIME_LIMIT_S:-1800}). On the host side there was no deadline anywhere:

• docker_wait() (docker.py:522) blocked on <engine> wait with while proc.poll() is None: and no bound
• batch.py:313 awaited proc.communicate() with no per-job bound

If the in-container watchdog never fired — entrypoint crash, wedged Chromium, engine hiccup, zombie container — the host waited forever. clawbench-run blocked; in batch mode the job held a concurrency slot indefinitely. With batches routinely running 8–20 hours, one wedged container could stall a 130-task run overnight with no error and no summary.

Ask 1 — deadline in docker_wait

It now takes timeout_s and returns whether the deadline expired. On expiry it kills the container and returns True. run.py passes time_limit_s + HOST_TIMEOUT_GRACE_S (5 min), records host_timeout: ... as the failure reason, and classifies the run as infra_failure so it stays out of adjusted scoring — then carries on to copy results and write run-meta.json as usual. Partial data from a killed container is still worth keeping, and this preserves the same invariant #303 and #302 are about: a run that got far enough to produce anything always records it. --human runs stay unbounded by design.

Ask 2 — per-job bound in batch.py

proc.communicate() is wrapped in asyncio.wait_for, sized from the case's own time_limit plus BATCH_JOB_GRACE_S (15 min) so it sits above the run's own deadline — clawbench-run reports its own timeout first, and this only fires when the child process itself is wedged. On expiry it kills the process group, marks the job error, notes host_timeout in the job log, and the batch proceeds. --job-timeout overrides the derived value; 0 disables it.

job_timeout_s() reads time_limit from both corpus layouts — <case>/task.json and claw-eval's flat <suite>/<id>.json — and falls back to 1800s when the task file is unreadable.

On the new utils/timeouts.py

Three constants shared by run.py, docker.py and batch.py. batch.py cannot import them from run_support.docker, because that pulls in run_support.config, which resolves a container engine at import time and sys.exits when neither Docker nor Podman is installed — and batch.py must stay importable without one (tests/test_batch_and_tui_helpers.py depends on it).

A shared module seemed better than duplicating two integers across modules that must agree. That import-time probe is the underlying problem, and I've filed it separately as #315 with a suggested lazy-engine() fix; if that lands, this module could fold back into docker.py. Happy to take a different shape here if you'd prefer.

Corpus

• v2
• v1
• both
• not applicable

_{Host-side runner/batch change; no task data involved.}

Test plan

• New tests/test_host_timeout.py, 12 tests: job_timeout_s against both corpus layouts, four unreadable-task fallbacks, override and disable; docker_wait killing a never-exiting container and returning promptly; timeout_s=None preserving the old unbounded behaviour; BATCH_JOB_GRACE_S > HOST_TIMEOUT_GRACE_S so the layering cannot silently invert; and batch.py still importing with no container engine present.
• Demonstrated the hang directly rather than only through tests — driving the unpatched docker_wait against a stub container that never exits, on a background thread:
  UNPATCHED docker_wait still running after 6s? True <- hangs forever PATCHED returned True after 2.0s; issued: ['kill', 'wedged']
• Full suite: 204 passed, 3 skipped. The one failure, test_host_tasks.py::test_checked_task_json_files_parse_and_validate[v1-lite], reproduces identically on a clean main on this machine — those task files are git symlinks (mode 120000) that Windows checks out as text. Unrelated.
• ruff check and ruff format --check clean.

Not verified: no live containerized run — I don't have Docker on this machine. The wedged-container path is exercised by injection, not by a genuinely stuck Chromium.

Related issues

Fixes #298. Root cause of the utils/timeouts.py workaround filed as #315.

_{One deliberate choice worth a maintainer's eye: a host timeout is classified as the existing infra_failure category with host_timeout: ... in failure_reason, matching the convention at run.py:415. The issue's wording ("infra_failure / host_timeout") could instead mean a distinct host_timeout category — that would need adding it to NON_MODEL_FAILURE_CATEGORIES and widening the infra_failure predicate in classify_run, which changes semantics shared with every other failure path. I took the conservative route; say the word and I'll switch it.}

[Perry2004]

I think this kills the per-run Python process and left the container process.
Also it is using SIGKILL so it would bypass the per-run cleanups in the finally block.

[vaibhavdabas16]

You're right on both counts, and the second one made the first worse. Fixed in c8ebd8a.

The container isn't in that process group. run.py shells out to the engine (docker_run(container, ...), run.py:508) and the daemon owns the container from there, so killpg reached the Python child and the CLI client and nothing else. And because SIGKILL can't be caught, run.py never reached its finally: docker_rm(container) (run.py:743-745, which is an rm -f). So every timed-out job leaked exactly the container whose CPU and memory the bound exists to reclaim. The marker I wrote into the job log — "container and child process killed" — was describing something the code didn't do.

The fix

stop_wedged_job() sends SIGTERM first. run.py already installs a SIGTERM handler that raises KeyboardInterrupt (run.py:423-429) and unwinds through that same finally, so the container actually goes away. Worth noting that handler is installed for every run, not just --human — I'd initially misremembered the --human Ctrl+C path at run.py:478 as the only graceful hook.

SIGKILL follows only if the child is still alive after JOB_KILL_GRACE_S (60s). Graceful teardown shouldn't become a second way to hang the batch, and a run wedged badly enough to ignore SIGTERM must not keep its concurrency slot. The grace covers docker rm -f, the disposable-mailbox delete and the browser-runtime teardown — short subprocess and network calls, not agent work.

Two things I changed beyond the literal comment

• The job log now reports which of the two actually happened. On escalation it says the container may still be running rather than claiming a clean teardown, since that's precisely the case where run.py's cleanup was skipped.
• An already-gone process group is reported as not escalated: it means clawbench-run reached the end of its own teardown, and labelling it SIGKILLed would send someone hunting for a container that isn't there.

Tests

Four new tests in tests/test_host_timeout.py drive a stub run that honours only the signals it's told to, so the SIGTERM-then-SIGKILL sequence is observable. As a control I reverted stop_wedged_job to the old SIGKILL-first behaviour and re-ran — 3 of the 4 fail, so they're pinning the fix rather than passing vacuously.

One limitation to flag: those 4 skip on Windows, where signal.SIGKILL and os.killpg don't exist — which is also why this path doesn't run there. I verified them locally by re-running the file with signal.SIGKILL shimmed in (16 passed), and that's how the control above was run too, but genuine POSIX coverage comes from the ubuntu and macos jobs here.

Also rebased onto current main (the branch was 32 commits behind). Full suite locally: 222 passed, 8 skipped. ruff format clean; ruff check adds no new finding classes over the previous head.

Still not verified: no live containerized run — I don't have Docker on this machine, so the orphaned-container behaviour is reasoned from the code and exercised by injection, not observed against a genuinely stuck Chromium. If you have a box where you can wedge one, that's the check I can't do.