feat(sandbox): add Apptainer backend for HPC environments

[zhiheng-yang]

feat(sandbox): add Apptainer backend for HPC environments

What this solves

BenchFlow tasks are usually packaged as Dockerfile build contexts, but Docker is often unavailable on HPC and other shared Linux systems. Those systems commonly provide Apptainer instead.

This PR adds --sandbox apptainer to the normal evaluation path:

bench eval run \
  --tasks-dir path/to/tasks \
  --agent oracle \
  --sandbox apptainer \
  --skill-mode no-skill

The provider accepts existing Dockerfile-based task environments and local SIF images. Dockerfile contexts are built into cached SIF images, while each rollout gets a separate writable overlay.

Scheduler allocations, environment modules, and site-specific setup stay outside BenchFlow, so the backend is not tied to Slurm or a particular cluster.

Design

Integration

apptainer uses the existing provider registry, capability checks, and sandbox factory. It is imported lazily, so other providers do not require an Apptainer installation.

Image inputs and build

The image layer supports two inputs:

1. A task build context containing a Dockerfile.
2. An existing local .sif image.

For Dockerfile-based tasks, the backend uses Apptainer's native BuildKit bootstrap:

Dockerfile + build context
          |
          v
Apptainer buildkit bootstrap
          |
          v
SIF image

Why not translate to a .def file: doing so reliably would require BenchFlow to reproduce multi-stage builds, build arguments, environment expansion, COPY ownership and permissions, shell forms, and .dockerignore behavior.

Instead, BuildKit evaluates the Dockerfile and build context, Apptainer creates the SIF, and BenchFlow owns caching and rollout lifecycle. Existing tasks do not need a second .def file, and no Docker daemon is involved.

Built images are stored in a content-addressed cache. Construction happens through a temporary output path; the completed image is inspected before an atomic rename makes it visible to other rollouts. An in-process lock prevents duplicate concurrent builds of the same context.

Rollout isolation

Each sandbox receives its own runtime directory, instance name, and writable overlay:

cached read-only SIF
        +
per-rollout writable overlay
        +
scoped bind mounts
        |
        v
Apptainer instance

The cached SIF is never modified by a rollout. Agent changes, generated files, package installations, and verifier state are written to the rollout overlay and disappear when that sandbox is deleted.

Instances are started with:

• --fakeroot for container-root semantics without host root privileges;
• --containall and --no-home to avoid inheriting the user's host environment;
• --cleanenv to prevent unintended environment leakage;
• an isolated writable overlay for rollout-local state;
• explicit bind mounts for staging and BenchFlow-owned logs;
• an isolated network namespace when the runtime policy disables networking.

--fakeroot does not grant host root privileges. It maps the calling user to root inside the container namespace, which preserves the task behavior expected from container images while keeping execution unprivileged on the host. The overlay is what makes this useful for agent workloads: the SIF remains immutable, while package installs and changes under paths such as /root, /app, and /usr are written to rollout-local storage.

A readiness probe completes before the sandbox is returned to the rollout. Partial startup failures use the same cleanup path as normal shutdown, preventing abandoned instances and temporary runtime directories.

Execution and agent transport

Commands run through apptainer exec against the named instance. A rollout-scoped environment file avoids command-line quoting problems and keeps the host environment separate. Working directories, timeouts, output capture, exit status, sandbox users, and background processes continue to use BenchFlow's existing interfaces.

Uploads and downloads use a bound staging directory, while a dedicated log mount keeps artifacts available after teardown. ApptainerProcess handles the bidirectional stdin, stdout, and stderr required by long-lived ACP agents; keeping it separate makes process and lifecycle behavior independently testable.

Failed builds never replace cached images, and instance cleanup is scoped and idempotent.

Runtime requirements

Dockerfile builds require Apptainer with BuildKit bootstrap support and a compatible rootless BuildKit service:

export BUILDKIT_HOST="unix:///path/to/buildkitd.sock"

An existing local SIF does not need BuildKit. Image caches may live on shared storage, while temporary BuildKit and Apptainer state can use node-local storage.

Implementation details by file

File	Change
sandbox/apptainer_image.py	Resolves Dockerfile contexts and local SIFs; implements deterministic caching, build locking, BuildKit construction, validation, and atomic publication.
sandbox/apptainer.py	Implements overlays, instance lifecycle, isolation, command execution, transfer, attachment, and cleanup.
sandbox/process/apptainer.py	Provides asynchronous bidirectional ACP process transport and termination.
sandbox/process/__init__.py	Exposes the Apptainer process implementation.
sandbox/providers.py	Registers the provider and its runtime capabilities.
sandbox/setup.py	Adds lazy construction and runtime preflight.
rollout/__init__.py	Skips process-pattern cleanup for synchronous Oracle runs.

Testing

The backend was tested end to end with a subset of SkillsBench tasks on a real HPC system. Oracle and ACP agent runs completed successfully, covering image construction, writable rollout state, verification, artifact collection, and cleanup.

[devin-ai-integration]

Devin Review found 6 potential issues.

4 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[zhiheng-yang]

Addressed the posted Devin findings in 60f53eb:

• included symlink targets in the image cache key;
• handled a leading relative Dockerfile WORKDIR;
• stopped deriving a prebuilt SIF workdir from an unrelated Dockerfile;
• cleaned partial runtime storage after setup timeout or cancellation;
• handled graceful-stop timeouts and verified instance liveness after forced-stop failures.