feat(agents): verify sandbox checkpoint evidence against S3

[tee8z]

A tombstone must not record a checkpoint key that the Pod alone claims. This PR adds src/evidence.rs, which confirms a key against an S3 (Amazon Simple Storage Service) ListObjectsV2 of the session's own prefix. The prefix comes from the reaper config and the Sandbox annotations, never from the Pod. An S3 failure gives Unavailable, so the reaper postpones the decision instead of recording missing.

flowchart TD
    receipt["from_receipt: agent container termination message"] --> own{"Receipt present, version 1, key directly under the session prefix?"}
    restore["inherited: restore-checkpoint annotation"] --> prev{"Key directly under the recovered-from prefix?"}
    own -- no --> missing["Missing"]
    prev -- no --> missing
    own -- yes --> list["CheckpointStore::contains: ListObjectsV2 of the prefix, max 10 pages"]
    prev -- yes --> list
    list -- "key found" --> found["Verified (receipt) or Inherited (restore key)"]
    list -- "key not found" --> missing
    list -- "S3 error or more than 10 pages" --> unavailable["Unavailable: decide on a later pass"]

Loading

• S3Store uses only ListObjectsV2 (IAM permission s3:ListBucket). Connect, attempt, and operation timeouts are 5 s, 10 s, and 25 s.
• Each S3 failure increments buzz_sandbox_reaper_s3_errors_total.
• docs/remote-agents.md gets a "Checkpoint evidence" section that states these rules.
• This layer adds the AWS SDK crates to Cargo.lock. Most of the lockfile growth is here.

Stack

Position 16 of 21. Base: #18. Next: #20. Each PR contains one signed commit. #24 and #25 repair the fork’s existing CI prerequisites; the feature layers remain scoped to remote agents.

Validation

• Integrated provider and reaper suite: 237 tests passed; the live-controller test also passed against the isolated local kind cluster.
• Signed stack and per-layer parentage verified. Workspace Rust Clippy, desktop lint, Rust formatting, file-size checks, and the desktop build passed. Native Clippy passed with default features and with mesh-llm. Full local just ci and the updated GitHub checks are in progress.

Limits

Evidence tests use an in-memory store. Real S3 and workload credentials require deployment validation.

Human workflow confirmation is still required by AGENTS.md before readiness. Human acceptance has not been confirmed.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[tee8z]

@coderabbitai full review

Please review the current revision against this PR’s immediate stack base. The descriptions document the remote-agent scope, regression coverage, and outstanding runtime validation.

[coderabbitai]

⚠️ Action not completed

Review rate limited.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. Your current included review allowance is based on your included PR review attempts over the past 7 days. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 58 minutes.