feat(agents): add the sandbox reaper reconcile engine and health server

[tee8z]

This PR adds the engine that applies the layer 1 plan to a live namespace. Reconciler::pass classifies each managed Sandbox, gathers checkpoint evidence, and makes at most one write per Sandbox. Every write is guarded by CAS (compare-and-swap) on the observed resourceVersion. The apiserver Date header is the only clock, so local clock skew cannot age a session into a tombstone.

flowchart TD
    list["List managed Sandboxes; read apiserver Date"] -- "no usable Date" --> fail["Pass fails; nothing written"]
    list --> pods["List managed Pods; capture termination receipts"]
    pods --> classify["Per Sandbox: classify State against owner_pubkey"]
    classify --> evidence["Gather Evidence (only for states that can tombstone)"]
    evidence --> plan["plan()"]
    plan -- "None" --> done["Next Sandbox"]
    plan -- "Ignore" --> ignored["Warning Event SandboxIgnored"]
    plan -- "Hold" --> held["Warning Event CheckpointMissing"]
    plan -- "Tombstone" --> patch["Merge patch: Suspended + lifecycle annotations, with observed resourceVersion"]
    plan -- "Delete" --> delete["Delete with uid and resourceVersion preconditions"]
    patch -- "applied" --> ended["Normal Event SessionEnded"]
    patch -- "Conflict" --> reread["Re-read Sandbox and Pod (max 3 attempts)"]
    delete -- "Conflict" --> reread
    reread --> classify

Loading

• run_namespace runs one pass per interval. Between passes, a Pod watch keeps termination receipts after a stop or replacement deletes the Pod.
• Each Event is created once per Sandbox and reason. NotFound on a write ends that Sandbox's work without error.
• dry_run logs each decision and writes no patch, delete, or Event.
• src/server.rs serves /metrics, /healthz, and /readyz. The health window is 3 × interval, minimum 120 s.
• After the first window, /healthz fails when a namespace loop has not attempted a pass within the window. /readyz fails when a namespace has no successful pass within the window.
• Metrics include buzz_sandbox_reaper_passes_total, buzz_sandbox_reaper_sessions, buzz_sandbox_reaper_tombstones_total, and buzz_sandbox_reaper_cas_conflicts_total.

A failed object reconciliation or unavailable S3 evidence makes the pass fail after other objects have been processed. Readiness and --once therefore reflect failed work. Recovery inheritance retains the original verified ancestor checkpoint through repeated failed recovery attempts, constrained to the configured developer prefix. Failed-exit Pod messages are excluded from the watch cache.

Stack

Position 18 of 22. Base: #19. Next: #21. Each PR contains one signed commit. #27 fixes the existing message-edit CI failure. #24 and #25 repair fork CI and security prerequisites; the feature layers remain scoped to remote agents.

Validation

• Integrated provider and reaper suite: 237 tests passed; the live-controller test also passed against the isolated local kind cluster.
• Signed stack and per-layer parentage verified. Workspace Rust Clippy, desktop lint, Rust formatting, file-size checks, and the desktop build passed. Native Clippy passed with default features and with mesh-llm. The final integrated revision includes the message-edit fix and Secret retry regressions. Full local just ci and updated GitHub checks are in progress; the previous local run stopped at a missing host Opus library, now supplied by the validation environment.

Limits

The next layer supplies the fake-apiserver scenarios. This PR is 1,028 changed lines: splitting failure propagation and checkpoint inheritance out of the engine would leave its readiness and data-preservation contract incomplete. The process entry point is added in #4.

Human workflow confirmation is still required by AGENTS.md before readiness. Human acceptance has not been confirmed.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[tee8z]

@coderabbitai full review

Please review the current revision against this PR’s immediate stack base. The descriptions document the remote-agent scope, regression coverage, and outstanding runtime validation.

[coderabbitai]

⚠️ Action not completed

Review rate limited.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. Your current included review allowance is based on your included PR review attempts over the past 7 days. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 58 minutes.