Docker.exec turns a hardening timeout into a bare ProcessLookupError, so the rollout is recorded as "verifier crashed: " with no message

[Jiankai-Sun]

Required prerequisites

• I have searched the Issue Tracker and this has not been reported before.
• I have verified the problem against the installed release and read the code path on main.

What version of BenchFlow are you using?

0.7.5 (PyPI).

System information

Host: macOS 15 (arm64, 14 cores), Python 3.12. Sandbox: Docker. Agent: claude-agent-acp with azure/anthropic/claude-opus-5, reasoning effort max. One task per bench eval run, --concurrency 4, agent budget 4 h.

Problem description

This is the same code path as #1058 but a different failure, and #1062 does not cover it.

Docker.exec handles a timeout like this (benchflow/sandbox/docker.py, both exec helpers, around lines 360-380 and 412-430):

except TimeoutError:
    process.terminate()          # <-- unguarded
    try:
        stdout_bytes, stderr_bytes = await asyncio.wait_for(
            process.communicate(), timeout=5
        )
    except TimeoutError:
        process.kill()
        stdout_bytes, stderr_bytes = await process.communicate()
    raise RuntimeError(f"Command timed out after {timeout_sec} seconds") from None

If the docker exec child has already exited and been reaped by the time the asyncio.wait_for timeout fires, process.terminate() raises ProcessLookupError. That escapes the except TimeoutError block, so the RuntimeError("Command timed out after Ns") is never raised.

Since the hardening execs run inside the try in _verify_rollout, the exception lands in the generic except Exception branch:

except Exception as e:
    verifier_error = f"verifier crashed: {e}"

str(ProcessLookupError()) is empty, so the rollout is recorded as literally verifier crashed: with nothing after the colon, rewards: null, and an empty verifier/ directory because the verifier was never constructed. The signal that this was a timeout — the one thing #1058 and #1062 are about — is destroyed on the way out.

Reproducible example code

bench eval run --tasks-dir <tasks> --include <task> \
  --agent claude-agent-acp --model azure/anthropic/claude-opus-5 \
  --reasoning-effort max

Probabilistic: it needs the hardening exec to time out and its child to exit in the same window. Five long rollouts of one task hit it; short rollouts of the same task at the same head never did.

Traceback

Not visible in a stock build, which is the point of this report. After patching _verify_rollout locally to print the type and the traceback:

verifier crashed: ProcessLookupError:

Stock builds print only:

verifier crashed:
[ERR] <task> (tools=124) (verifier crashed: )

Expected behavior

Two things:

1. Docker.exec should guard the teardown, e.g. with contextlib.suppress(ProcessLookupError): process.terminate() and the same around process.kill(), so a timeout is always reported as the timeout it is. With that, this failure becomes the case fix(sandbox): use the verifier-setup budget for hardening execs #1062 fixes.
2. _verify_rollout's generic branch should include the exception type, since f"verifier crashed: {e}" is empty for any exception with no args. f"verifier crashed: {type(e).__name__}: {e}" would have made this a five-minute diagnosis rather than a five-occurrence mystery.

Additional context

Worth ruling out for anyone who finds this by symptom: it is not resource exhaustion. I logged docker stats every 30 s across nineteen containers of these runs. The peak was 620 MiB against a 7.75 GiB limit, and the container that failed was at 69 MiB (0.87 %) at the moment of the crash. Nothing was OOM-killed.

The practical cost is that a finished agent rollout — hours of work and, in one of these, 26 M tokens — is scored as nothing, and none of its deliverables are preserved, because verifier/ is cleared before the step that dies.

[tulerfeng]

Thank you for catching this important issue. I opened PR #1081 for it.

Two things worth flagging that go past the report.

The window is narrower than "already exited", and pinning that down is what made it reproducible. terminate() only fails once asyncio has cleared its handle on the process, which happens on the next loop tick — so the timeout has to fire inside a one-tick gap. Chasing the condition as written, I got 840 attempts and zero hits on an idle loop. Holding the event loop busy the way a saturated run does, it went to 288 hits out of 288. Under concurrency this isn't an occasional race, it's the expected outcome.

Three more sites had the same defect, so the PR fixes all five: the pre-compose hook, both Apple Container paths, and SubprocessLiveProcess.close() whose returncode guard can't cover its escalation branch. The one I'd single out is acp/transport.py's close() — no liveness check at all, while the equivalent close() next door has one and documents itself as "safe to call after process death". It runs during teardown with a real exception usually already in flight, so raising there hides it.

On your second suggestion I used the existing describe_exception rather than the literal f"{type(e).__name__}: {e}" — the agent-side funnel already uses it for your exact reason, the verifier path just never picked it up. The message becomes verifier crashed: ProcessLookupError (no message).

Verified on a real container: the same live docker compose exec process gives ProcessLookupError (args=()) → 'verifier crashed: ' on main, and RuntimeError: Command timed out after 10 seconds after the fix.