feat(agents): recover ended Sandbox sessions explicitly

[tee8z]

An ended Agent Sandbox session must resume only on explicit user action, and recovery must never discard a live workspace.
This PR adds provider_config.sandbox.recovery, which the desktop sends once per click.
{"mode":"checkpoint"} restores the tombstone's verified checkpoint. {"mode":"fresh"} starts with an empty workspace.

sequenceDiagram
    participant P as Provider deploy
    participant S as Sandbox
    participant J as Journal Secret
    participant O as Old Pod
    P->>S: GET Sandbox, or load the journal if no Sandbox exists
    Note over P: verify identity, require ended and Suspended, read generation and checkpoint key
    P->>J: CREATE immutable Secret with tombstone.json and no ownerReference (or reuse a matching journal)
    P->>S: DELETE with uid and resourceVersion preconditions
    loop every 1 s, until 300 s
        P->>S: GET until the old uid is gone
        P->>O: GET until no Pod owned by the old Sandbox exists
    end
    P->>S: CREATE Sandbox with a new generation and recovered-from
    P->>S: Bind the first Pod and write the launch Secret
    P->>J: DELETE with uid and resourceVersion preconditions

Loading

The 300 s wait is the 240 s Pod termination grace plus 60 s for the controller and garbage collection.
In checkpoint mode, the new Sandbox has restore-checkpoint, and the launch Secret sets BUZZ_CHECKPOINT_RESTORE_KEY.
The journal Secret is named <sandbox>-recovery, uses type buzz.block.xyz/sandbox-recovery-v1, and holds no private key.
The provider deletes the journal only after the replacement binds, and only if the journal records the replaced generation.

Condition	Deploy error
recovery on a live or ending Sandbox	recovery requires an ended session; this Sandbox is still active, so end it first
checkpoint mode, tombstone has no checkpoint key	this session has no verified checkpoint; start fresh instead
checkpoint mode, no Sandbox and no journal	no ended session exists for this thread to recover; start fresh instead
No Sandbox, journal exists, request has no recovery	Sandbox recovery was interrupted; explicit recovery is required
Journal is not an immutable record of an Ended, Suspended tombstone	invalid Sandbox recovery journal; retain it for operator recovery
A foreign Pod holds the Sandbox name	a foreign Pod occupies the recovered Sandbox name; retain the recovery journal for operator cleanup

stop now refuses a request with recovery. An unknown mode is refused before any cluster contact (deploy-sandbox-bad-recovery fixture).

Sandbox Pods use the same 300-second shutdown budget as ordinary agent Pods, and stop waits for that budget before treating termination as overdue.

Stack

Position 13 of 22. Base: #16. Next: #17. Each PR contains one signed commit. #27 fixes the existing message-edit CI failure. #24 and #25 repair fork CI and security prerequisites; the feature layers remain scoped to remote agents.

Validation

• Integrated provider and reaper suite: 237 tests passed; the live-controller test also passed against the isolated local kind cluster.
• Signed stack and per-layer parentage verified. Workspace Rust Clippy, desktop lint, Rust formatting, file-size checks, and the desktop build passed. Native Clippy passed with default features and with mesh-llm. The final integrated revision includes the message-edit fix and Secret retry regressions. Full local just ci and updated GitHub checks are in progress; the previous local run stopped at a missing host Opus library, now supplied by the validation environment.

Limits

The live controller test simulates the reaper tombstone. It does not exercise an actual checkpoint upload or restore.

Human workflow confirmation is still required by AGENTS.md before readiness. Human acceptance has not been confirmed.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[tee8z]

@coderabbitai full review

Please review the current revision against this PR’s immediate stack base. The descriptions document the remote-agent scope, regression coverage, and outstanding runtime validation.

[coderabbitai]

⚠️ Action not completed

Review rate limited.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. Your current included review allowance is based on your included PR review attempts over the past 7 days. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 59 minutes.