Hardcoded 10s timeout on `mkdir -p /logs/agent` discards completed rollouts under I/O load

[EYH0602]

Summary

_publish_trajectory_for_verifier gates a mkdir -p /logs/agent at a hardcoded
timeout_sec=10. When that exec exceeds 10s on a loaded host, the exception
propagates and the whole rollout is lost — after the agent ran to completion.

https://github.com/benchflow-ai/benchflow/blob/main/benchflow/rollout/_setup.py#L440 (v0.6.6; same call at _setup.py:418 in v0.6.5)

await env.exec("mkdir -p /logs/agent", user="root", timeout_sec=10)

The command is trivial, but the exec is not: on a busy host, Docker exec setup
alone can exceed 10s. The asymmetry is what makes it expensive — a ~40 minute
rollout is discarded by a 10 second budget on a bookkeeping step, after all the
costly work has already succeeded.

Impact

Seen in a 16-launch replication series where the task under test is itself
I/O-heavy (an AFL++ campaign with ~20 parallel workers saturating disk). One
launch died this way:

RuntimeError: Command timed out after 10 seconds

on mkdir -p /logs/agent during trajectory publish, on the third internal
attempt (the first two had failed for unrelated transient reasons). It had to be
excluded from the experiment's denominators entirely — it is the only launch in
the series we could not classify as either a success or a genuine agent-side
failure, because the failure is purely in publishing.

Environment: benchflow 0.6.5 and 0.6.6, Docker sandbox, Linux.

Why this looks like a bug rather than a tight-but-intentional budget

On the Docker backend specifically, the exec appears to be doing no work.
_publish_trajectory_for_verifier calls agent_dir.mkdir(parents=True, exist_ok=True)
two lines above, and per the function's own docstring Docker bind-mounts
/logs/agent to that same host dir — so the directory already exists by the time
the container-side mkdir -p runs. The call is defensive for backends that
don't mirror (Daytona, Modal), but on Docker it can only fail, never help.

The await env.upload_file(...) immediately after would fail loudly on its own
if the directory genuinely were missing, so the mkdir failing fatally is not
load-bearing for correctness.

Suggested fix

In rough preference order:

1. Make the mkdir non-fatal — wrap in contextlib.suppress(Exception) (or
   retry once) and let the subsequent upload_file be the real error surface.
2. Retry the publish-path execs before failing the rollout.
3. Make the timeout configurable, or scale it with the configured agent timeout,
   rather than fixing it at 10s.

Happy to send a PR for (1) if that's the direction you'd prefer.

[MrHappyEnding]

Hi, I have a small fix with a regression test. The fix preserves the completed host-side trajectory when the sandbox-side mkdir -p /logs/agent hits its client timeout, logs a warning, and still attempts publication for the verifier. I’d be happy to open a PR if this direction matches the intended behavior.

[tulerfeng]

Opened #1043 for this, taking approach (1) from the suggested fix.

Both publish-path execs are now non-fatal. Besides the mkdir -p /logs/agent in
_publish_trajectory_for_verifier, the same 10s exec exists in
_scrape_agent_trajectory, which verify() awaits immediately before the
publish call — so under the load you describe it would hit the timeout first and
abort the rollout before the publish-side guard could apply. Both are fixed.

upload_file stays fatal, so a genuinely missing /logs/agent still surfaces
rather than letting a rollout claim a trajectory the verifier never received.
Both guards log a warning instead of suppressing silently, since the visible
RuntimeError is what made this diagnosable in your report.