feat(deploy): roll production back when the post-deploy checks fail

[hseshadr]

Replaces #149, which GitHub closed automatically when its base branch (fix/deploy-concurrency-cancel) was deleted after #147 merged. It cannot be reopened (base gone, head force-pushed). Same branch, same commits, rebased onto main.

Rebased onto main (#148 and #147 are merged). Both shared modules and CENTRAL_MODULE_SHA are pinned to hseshadr/ci main 363be0b, the merge commit of ci#51 (verified Pages rollback). Pin test: red at 468c14e, green at 363be0b.

TL;DR

When the post-deploy live smoke (#148) goes red, production now rolls itself back to the deployment that was live before the upload, then gets checked again in a real browser. The job still fails, and the error says which of three outcomes happened.

Claim touched: "a release that fails in a real browser does not stay live."

Flow

1. Before upload: previous_production_deployment(token, account, "aml-filter"). It's read-only and records the target. If it can't be read, there's no upload, because a deploy we can't roll back is refused.
2. Upload, live identity verify, fresh + returning smoke (unchanged from feat(deploy): live browser smoke after every deploy and list publish #148).
3. Red smoke, or a failed live identity check (production serving the wrong commit or bundle; the smoke is then skipped): rollback(token, account, "aml-filter", deployment_id=target) through the shared module, which confirms Cloudflare's canonical_deployment is now the target. Then the fresh smoke re-runs against https://aml-filter.com, without the SHA pin because the restored release is older.
4. LiveSmokeFailedError either way:
   • rolled production back from X to Y …; recovery smoke PASSED (production recovered, the release is still bad)
   • recovery smoke FAILED … production is STILL BROKEN after rollback
   • automatic rollback FAILED: <reason> … roll back … by hand now (no recheck)

Rollback happens only through the shared module (dag.cloudflare_pages()), so the contract that confines provider mutation still holds.

Red → green

Guard	Red	Green
New verdict tests (test_smoke.py: recovered / still broken / rollback refused)	import error, then red	10/10
New orchestration contracts: target recorded before construct:deploy; no rollback on green; no upload if target unreadable; red smoke → rollback(recorded target) → recovery smoke; recovery also red → "STILL BROKEN"; rollback refused → manual, no recheck; recovery container = fresh, no SHA pin	8 red before implementation	green
Mutation M1: skip the rollback call	3 failed	restored, green
Mutation M2: recovery verdict ignores the recheck's exit code	2 failed	restored, green
Identity failure (new): stubbed _live_verify error → rollback exactly once → recovery smoke, post-deploy smoke skipped; red smoke + red recovery → still exactly one rollback	1 red before the change	green
Mutation M3: re-raise the identity error instead of recovering	1 failed	restored, green
Pin test (test_should_pin_both_shared_modules_to_exact_central_main)	red at 468c14e (previous PR-head pin)	green at 363be0b (ci#51 merge)

Gates: .dagger 180 passed / 1 skipped (includes the real dagger functions/--help schema tests against the 363be0b modules); ruff, mypy, xenon A clean.

Unverified

• The real rollback and the target read have never run against Cloudflare. I have no production token and did no production writes. The first red smoke in a real deploy after merge is the actual test.

🤖 Generated with Claude Code

https://claude.ai/code/session_015oBArfm762nN1r4F4Fst5a