feat(agents): checkpoint after turns and recover lost sessions from them

[tee8z]

A Spot node lost without warning sends no SIGTERM, so the harness never runs its save on stop. The session then ends with checkpoint=missing, and the thread's work is gone. With this PR, the harness also saves a checkpoint in the background after completed turns. When a lost Pod left no termination receipt, the reaper uses the newest checkpoint in the session's own S3 (Amazon Simple Storage Service) prefix.

sequenceDiagram
  participant H as buzz-acp harness
  participant C as buzz-agent-checkpoint save
  participant S3 as S3 checkpoint bucket
  participant R as buzz-sandbox-reaper
  participant SB as Sandbox
  H->>C: start in the background after a completed turn
  C->>S3: upload under developer prefix + generation/
  C-->>H: receipt with the checkpoint key
  H->>H: write the key to /dev/termination-log
  Note over H: Node lost without warning, no save on stop, no receipt in the Pod status
  R->>R: classify the Sandbox as Lost (bound, no Pod for 60s or more)
  R->>S3: ListObjectsV2 of developer prefix + generation/
  S3-->>R: newest key directly under the prefix, by LastModified
  R->>SB: tombstone with ended-reason=lost and checkpoint=key
  Note over SB: Checkpoint-mode recovery restores this key

Loading

• buzz-acp/src/checkpoint.rs: TurnCheckpoints runs one save at a time and starts at most one per BUZZ_ACP_CHECKPOINT_AFTER_TURN_SECS (default 600, 0 disables). It runs only with BUZZ_ACP_CHECKPOINT_ON_STOP.
• Before the save on stop, the harness waits up to 60s for a running background save, then cancels it. A failed, timed-out, or cancelled save kills the helper's whole process group.
• turn_completed observer events and activity records now carry duration_ms.
• buzz-sandbox-reaper: the new Evidence::Listed(key) applies only to a Sandbox that got past binding and has no receipt. A Completed session with Listed evidence still waits out the 24h hold.
• All sessions of one developer share one IAM role. A compromised session can write into that developer's prefixes, but not into another developer's prefix (see docs/remote-agents.md).

Stack

Position 19 of 20 in the DEV-2884 Buzz stack. Base: #4. Next: #7.

Validation

Run in a detached worktree at this commit (CARGO_TARGET_DIR shared, CARGO_BUILD_JOBS=4):

• cargo test -p buzz-acp -p buzz-sandbox-reaper: 1,131 passed, 0 failed, 3 ignored. This includes the new TurnCheckpoints, evidence::latest, and a_lost_session_records_its_newest_turn_checkpoint tests.
• cargo clippy -p buzz-acp -p buzz-sandbox-reaper --all-targets -- -D warnings: pass.

Limits

• No cluster, Spot interruption, or S3 bucket was used. The reaper tests use a fake store, and the harness tests use a fake save.
• A background save does not stop the adapters, so another agent slot can be mid-turn while the save reads the workspace.
• Buzz AGENTS.md requires a human to test the change before it leaves draft.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}