Repository navigation
Declare checkpoint sandbox paths on samples, tell agent about checkpointed paths - #77
Merged
Merged
Conversation
Samples capture all of /workspace in the agent sandbox and nothing in the comparator. This only configures checkpointing; enabling it is left to the eval layer (e.g. checkpoint.enabled in a Hawk eval-set config).
Adds a WORKSPACE constant to apn.layout, used by PROJECT, the prompt line and the sample checkpoint sandbox_paths.
Collaborator
Author
|
Checkpointing was confirmed to be working by manually killing the runner in erdos3-sol61-9600usd-14d-g5ji9eqc7bgre38b. Checkpointing was every 6 hours. After ~68 hours of running, Giles manually killed the runner. It took ~8 hours to restore the state and begin making model calls again. 8 hours is pathologically long but still better than restarting from scratch. |
tadamcz
marked this pull request as ready for review
October 20, 2026 10:30
tadamcz
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every sample built by
build_datasetnow carries aCheckpointSampleConfigdeclaring what Inspect checkpointing should capture: all of/workspacein the agent sandbox, and nothing in the comparator (it is reset per check).This does not enable checkpointing. The sample layer is customize-only in Inspect; enabling is left to the eval layer, e.g. in a Hawk eval-set config:
Without explicit paths, Inspect 0.3.273 would capture each sandbox's home directory (
/root, including the Lean toolchain under/root/.elan) in both sandboxes, contrary to Hawk's docs, which say only host state is captured.