[FEATURE][RELIABILITY] Add restore quarantine and external-state reconciliation for backup recovery

[Joncallim]

Parent programme: #333
Execution mode: implementation
Depends on: #342
Blocks: #343, #344
Spec references: SPEC-0002 (Mission/Execution terminal/recovery semantics), SPEC-0003 (current security/revocation), SPEC-0004 (side-effect uncertainty/reconciliation), SPEC-0008 (restart/failure conformance), SPEC-0012 (authoritative audit/evidence), SPEC-0014 (migration/evidence preservation)

Problem Statement

Forge VNext is designed to recover from ordinary process/worker/Redis restart using PostgreSQL as durable truth. That is not sufficient for backup restore / database point-in-time rollback.

If PostgreSQL is restored to an older snapshot, Forge can forget real events that happened after the backup while external Resources retain them: completed GitHub/service/email/notification mutations, Trigger occurrences, credential revocations/rotations, autonomy demotions, budget usage, Mission/Execution progress and Operation evidence. Starting normal workers against the old snapshot could therefore replay schedules, recreate already-completed external effects, resurrect stale Grants/authority, double-spend budget, or treat an already-handled incident as new.

A backup is locally consistent historical truth, not automatically current world truth.

Desired Outcome

Forge has an explicit backup/export + restore protocol that restores durable data without automatically resuming consequential autonomy. Any restored instance enters a restore quarantine before Triggers, model calls or external mutations can resume. The operator/recovery system establishes a new restore epoch, revalidates current security/credentials/resources, reconciles externally persistent side effects and establishes fresh Trigger/Resource watermarks before normal Mission execution resumes.

Ordinary process restart remains automatic under #340/#347; backup restore is a distinct disaster-recovery authority transition.

User Story

As the Forge operator,
I want to restore Forge from a backup without Forge repeating actions or resurrecting stale authority,
So that backup recovery is safer than losing the database and does not create a second incident in external systems.

Core Invariants

• A restored database snapshot MUST NOT immediately resume autonomous external effects, model invocations or Trigger-created Executions.
• A restored snapshot MUST NOT be assumed current merely because its internal tables are transactionally consistent.
• External effects that occurred after the backup MUST NOT be replayed blindly.
• Historical evidence in the backup remains immutable under its original schema/policy; restore metadata does not rewrite it as newly observed truth.
• Current security/credential/autonomy state is revalidated before new consequential admission; old stored allow state never overrides a newer real revoke/rotation merely because the database rolled back.
• Trigger catch-up after restore is explicit and bounded; elapsed time during the lost interval is not silently burst-replayed.
• Restore quarantine is durable and fail-closed across process reboot until explicitly completed/aborted by the trusted recovery path.

Requirements

A. Backup/export contract

Provide a supported Forge backup/export procedure/manifest that records at minimum:

• Forge instance identity;
• backup/export identity and schema version;
• database schema/migration version;
• backup creation DB timestamp;
• current runtime/migration/authority epoch/version;
• evidence/artifact storage coverage and retention status;
• protected key/reference generations needed to interpret encrypted/digested records without embedding raw secrets;
• expected external Resource/adapters/Trigger classes requiring restore reconciliation;
• content digest/checksum of the backup/manifest where appropriate.

Backup must be transactionally consistent across the PostgreSQL authoritative state it claims to cover. If Artifact/evidence bytes live outside PostgreSQL, the backup protocol must define a consistent snapshot/manifest relationship and explicitly report missing/unbacked content.

Raw credentials/private keys are handled through the existing secure store/backup policy and are not copied into ordinary backup metadata, logs, GitHub artifacts or issue comments.

B. Restore detection and quarantine

A supported restore command/procedure creates a new restore epoch and durable quarantine state before ordinary application/worker execution is allowed.

Quarantine blocks at least:

• new Trigger-created/manual autonomous Executions;
• new model invocations;
• new external/consequential Operations;
• elevated autonomy use;
• compatibility Task mutations that would resume work automatically;
• catch-up scheduling/outbox publication whose world-state assumptions are not yet reconciled.

Read-only operator inspection and explicit recovery/reconciliation Operations may remain available under a narrow recovery Principal/Grant.

Do not rely on an in-memory flag or “remember to stop the worker.” Startup reads durable quarantine state before starting processing loops.

C. Restore epoch and stale-work fencing

Restore creates a monotonic current runtime/restore generation that invalidates pre-restore ephemeral ownership/admission state.

• old worker/lease/admission tokens are invalid;
• stale Redis/outbox deliveries from the pre-restore runtime cannot execute under the new epoch;
• current continuation/Trigger/Execution admission includes the restore/runtime epoch in its fencing context;
• Redis is treated as disposable/reconstructable and should normally be cleared/reinitialized or fully fenced against the restored database state according to the protocol;
• old signed/current projections whose applicability depends on post-backup state are re-evaluated rather than assumed current.

A monotonic epoch stored only inside the restored snapshot cannot by itself detect an arbitrary unsignalled storage rollback. Therefore normal production support must require restoration through the explicit recovery procedure or an external/current installation marker/epoch where practical. Document that copying an old raw database over a running installation is unsupported and unsafe.

D. External uncertainty horizon

The interval from backup_created_at to restore_started_at is a lost-current-state horizon. Forge may not know which external effects/events occurred in that interval.

For every configured consequential adapter/Resource domain with active/recent Missions/Operations, classify recovery using current external truth where supported:

• exact stable external Operation/idempotency/audit identity proves an effect already occurred;
• current Resource state can prove the intended effect did not/does exist according to the Operation's reconciliation contract;
• external state is insufficient/ambiguous -> operator/human-required block;
• no relevant consequential work was possible in the horizon -> explicit no-effect evidence.

Do not invent Forge Operation provenance for an external effect that exists but cannot be tied to a backed-up Forge Operation identity. It is current external state and may be recorded as restore reconciliation evidence, not rewritten history.

E. Operation recovery

For Operations present in the backup:

• terminal confirmed historical rows remain historical, subject to current Resource/security revalidation for future actions;
• submission_uncertain/reconciling rows continue through [FEATURE] VNext Phase 2 — secure generic execution envelope and side-effect recovery #336/[FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342 reconciliation under quarantine;
• non-terminal rows whose exact external status cannot be established remain blocked/human-required;
• no Operation is blindly retried because the local snapshot predates its possible external acknowledgement;
• idempotency keys/external refs remain exact; restore does not mint a fresh key to “try again.”

For possible Operations after the backup that are absent from restored DB, Resource/domain reconciliation establishes a safe new baseline before that Resource can receive autonomous writes again.

F. Trigger/schedule/event recovery

Consume #341 semantics rather than creating a backup-specific scheduler.

For each Trigger binding establish a new restore activation/watermark:

• schedule catch-up policy is explicitly skip | latest | bounded and defaults conservatively; never replay the entire outage by default;
• historical schedule instants are not claimed as verified/processed unless exact preserved occurrence/Resource evidence exists;
• webhook/event source resumes from a supported current external/update watermark where the protocol provides one, otherwise uses a documented restore floor and replay policy;
• occurrences before the restore floor are not automatically reissued because the restored DB forgot them;
• Trigger definition/enabled/current security state is revalidated before reactivation.

G. Persistent Mission/Execution recovery

Restore does not blanket resume every backed-up non-terminal Execution.

For each active Mission/Execution:

• pin/verify exact Mission spec/Workflow/Resource revisions available after restore;
• inspect checkpoints only as hints/references to canonical restored evidence;
• inspect externally consequential Operation state and lost horizon;
• decide resume | wait/reconcile | supersede/restart with new Execution | human_required | cancel through a closed trusted recovery policy;
• no terminal identity is reopened;
• a new Execution created because the previous current world state is unknowable has new identity and preserves the old restored Execution as historical/indeterminate evidence according to policy.

H. Current security, autonomy and credentials are revalidated

Before leaving quarantine:

• current Forge/system security policy/configuration is reloaded from trusted current installation sources;
• credential bindings are revalidated for existence/generation/audience/expiry where possible;
• provider/adapter readiness becomes unknown/stale until current evidence exists;
• explicit operator caps/revocations and current adverse-authority blocks are re-established/confirmed;
• elevated [FEATURE] Add evidence-based earned autonomy policy engine #189 earned-autonomy decisions from the restored snapshot are not automatically effective. Default to a safe baseline/requalification until policy proves the exact decision/evidence is still applicable under the new restore epoch/current security state;
• unknown/missing current authority state fails closed.

A restore cannot resurrect a revoked credential/Grant merely because the snapshot predates revocation.

I. Budget/cost reconciliation

The restored budget ledger may omit provider calls/Operations that incurred cost after the backup.

• do not assume restored remaining = ceiling - consumed is current spend truth;
• provider/account reconciliation uses available external billing/quota evidence where supported;
• unknown lost-horizon model spend is conservatively held/accounted according to [FEATURE] VNext Phase 1 — deterministic budget, routing, and context economics #335 hard-budget policy;
• if current spend cannot be bounded under a hard monetary ceiling, cost-incurring autonomy remains blocked until explicit operator reconciliation/policy reset;
• resetting/rebasing a budget after disaster recovery is an explicit audited operator action and does not rewrite historical usage.

J. Evidence/audit integrity after restore

Append a restore record that states:

• backup identity/time/schema;
• restore epoch/start/completion;
• operator/recovery Principal;
• reconciliation manifest digest;
• Resources/Missions/Triggers/credentials/budgets reviewed;
• unresolved uncertainty;
• activation floors/watermarks;
• resulting safe state.

Do not fabricate transition/audit events for the lost interval. The audit should honestly show a gap/horizon after the backup.

If immutable evidence/artifact bytes required to interpret backed-up records are missing/corrupt, affected decisions remain unverifiable/blocked; absence is not success.

K. Restore completion gate

Quarantine may be lifted only when a trusted deterministic restore gate proves:

• database schema/current runtime migration valid;
• stale workers/Redis deliveries fenced;
• required Resource domains reconciled or explicitly blocked from writes;
• active Missions have recovery dispositions;
• Triggers have explicit activation/catch-up floors;
• current credentials/security/autonomy/budget state is safe;
• no unresolved side-effect uncertainty can be blindly replayed;
• post-restore conformance canaries pass.

Quarantine may be lifted partially by Resource/Capability domain only if the architecture implements explicit scoped recovery fences; otherwise use one global gate initially. Do not allow a model to decide restore completion.

L. Backup retention/security

Backup files contain sensitive orchestration/evidence state.

• storage access is least privilege;
• encryption/integrity according to deployment policy;
• retention/deletion is explicit;
• restore tooling never logs raw secrets/protected payloads;
• test fixtures use synthetic credentials/data only;
• backup metadata exposed to UI/logs is sanitized/classification-aware.

Implementation Sequence

1. Backup/restore state contract + threat model — backup manifest, restore epoch/quarantine, unsupported raw-rollback statement.
2. Startup/write fencing — worker/Trigger/model/Operation admission blocked under quarantine; stale Redis/lease/admission tokens invalid.
3. Restore reconciliation manifest — Resource/Mission/Trigger/credential/budget categories and operator workflow.
4. Operation/external-resource reconciliation — [FEATURE] VNext Phase 2 — secure generic execution envelope and side-effect recovery #336/[FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342 adapters, uncertain/lost-horizon handling.
5. Trigger schedule/event floors — [FEATURE] VNext Phase 7 — Trigger/Event runtime with dedupe, causality, and zero-token idle #341 catch-up/watermark integration.
6. Mission recovery dispositions — [FEATURE] VNext Phase 6 — persistent Missions, checkpoints, leases, and bounded autonomy #340 checkpoint/Execution semantics without reopening terminal rows.
7. Security/autonomy/credential/budget revalidation — [FEATURE] Add evidence-based earned autonomy policy engine #189/[FEATURE] VNext Phase 1 — deterministic budget, routing, and context economics #335/[FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342 current state.
8. Completion gate + scoped/current activation — deterministic proof, post-restore canaries.
9. Backup artifact/security/retention — supported operator procedure and sanitized evidence.
10. Disaster-recovery release gate — populated realistic backup, external side-effect fixtures and repeated restore/reboot tests.

Orthogonal Checkpoints

1. State rollback: backup is internally valid but older than real external world; no replay/fake current truth.
2. Authority: old Grant/autonomy/credential state tries to regain access after current revoke/rotation.
3. Side effects: operation occurred after backup but local DB forgot it; no duplicate.
4. Triggers: missed schedules/webhooks across backup horizon; no burst/replay without explicit policy.
5. Budgets: missing post-backup spend; unknown never becomes zero/free.
6. Evidence: lost audit interval remains explicit; no synthetic history.
7. Startup: quarantine survives reboot and blocks all ordinary autonomous processing.
8. Storage/security: backup secrets/evidence protected and test fixtures synthetic.
9. Reconciliation failure: inability to establish external truth leaves bounded human-required state rather than “restore successful.”
10. Repeatability: restore procedure is deterministic/idempotent enough to retry after failure without progressively widening authority.

Acceptance Criteria

• [FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342 and its transitive Mission/Trigger/Operation foundations are closed before implementation starts.
• A supported backup has a versioned integrity manifest and explicit evidence-storage coverage.
• Restoring a backup enters durable quarantine before ordinary workers/Triggers/model/external Operations can act.
• Stale Redis/lease/admission state cannot execute after restore epoch change.
• A confirmed external mutation performed after the backup but before restore is not repeated when current Resource truth can prove it exists.
• Ambiguous lost-horizon external effects block/human-reconcile rather than blind retry.
• Triggers resume from explicit restore floors/catch-up policy and do not replay the whole lost interval by default.
• Non-terminal Missions/Executions are individually classified for safe recovery; terminal identities never reopen.
• Restored elevated autonomy/credentials/Grants do not become current until revalidated under current security/restore epoch.
• Unknown post-backup model spend remains conservatively budget-blocking rather than zero.
• Audit/evidence clearly records the backup horizon and restore/reconciliation instead of inventing missing history.
• Quarantine completion is deterministic, evidence-backed and fail-closed on unresolved required domains.
• Reboot during an incomplete restore remains quarantined and restart-safe.
• Hostile fixtures cover stale credentials, forgotten external effects, missed schedules, unknown spend, missing evidence and repeated restore attempts.

Out of Scope

• Transparent active-active disaster recovery across Forge instances.
• Automatic cross-region failover.
• Importing arbitrary unsupported raw database snapshots and pretending rollback detection is infallible.
• Reconstructing external events/effects that neither Forge nor the external system retained enough evidence to determine.
• Replacing external service backup/disaster-recovery policy.

Implementation Scope

Large / trust-critical disaster-recovery slice. Expected as 4-6 small PRs plus operator recovery fixtures after generic adapters land.

Technical Notes

This issue exists because process restart and storage rollback are different failure classes. PostgreSQL is Forge's durable authority during normal operation, but restoring an older PostgreSQL snapshot moves local truth backward while the external world does not. Safe recovery therefore requires a new explicit authority epoch/quarantine and external-state reconciliation before autonomy resumes.

[github-actions]

FORGE issue validation

This issue is not semantically dispatchable.

Reason	Detail
queue.issue_dependency_open	A declared dependency is still open. Dependency: #342.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #341.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #340.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #339.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #338.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #337.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #188.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #336.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #334.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #335.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #346.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #353.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #355.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #189.
queue.issue_dependency_open	A declared dependency is still open. Dependency: #347.

Readiness labels are projections, not authority. Command, dispatch, and handoff always re-resolve current semantic truth.

[Joncallim]

Hostile restore addendum — same-instance recovery vs clone restore, split-brain fencing, and extensible recovery domains

The first review after creating #366 found an additional disaster-recovery failure mode that must be part of the contract: a valid backup can be restored onto a different host while the source Forge instance is still alive, or restored deliberately for testing. If both copies retain the same instance identity, credentials, Trigger state and external idempotency namespace, they can become two writers to the same external world.

1. Restore mode is explicit: same_instance_recovery vs clone/test_restore

Never infer from hostname/path.

Same-instance disaster recovery

Intention: replace a failed/lost Forge installation and eventually resume its responsibilities.

Before quarantine can lift, prove/record an operator-controlled old-instance fencing decision appropriate to the deployment:

• old service/host is confirmed stopped/unreachable/decommissioned or its Forge credentials/network authority are revoked;
• no second worker/Trigger/adapter process with the old instance identity can still act;
• current external webhook/consumer ownership is transferred exactly once;
• current secrets/credentials are rebound/rotated as policy requires;
• restore epoch/current installation marker now identifies the replacement runtime.

If old-instance liveness cannot be established, do not start autonomous writes. Remain quarantine/human-required. Forge VNext does not claim active-active/split-brain consensus.

Clone/test restore

Intention: inspect/debug/test historical data without replacing the source installation.

Mandatory defaults:

• allocate a new Forge instance identity;
• permanent restore quarantine/read-only posture unless an explicit isolated test policy enables synthetic Resources only;
• external Triggers/webhooks/schedules disabled;
• production credential bindings/Grants not activated/copied into usable form merely because references exist in DB;
• external Resource write bindings marked unavailable/rebind-required;
• provider/cloud model access disabled unless separately approved for the clone with its own current credentials/budget/egress policy;
• no production notification/remediation/PR/email/etc. Operation can submit;
• Redis/outbox reconstructed only for explicit isolated fixture/test domains, never production catch-up;
• source instance id remains historical provenance on imported records, not the clone's acting Principal identity.

A test clone must never send a production webhook/notification because someone opened the dashboard.

2. Instance identity participates in external-writer/recovery fencing where semantically required

Do not globally put instance id into business idempotency keys if that would allow two restored copies to perform the same logical external effect under different keys. External-effect idempotency must remain scoped to the logical Operation/Resource domain as #336/#342 specify.

Instead use current installation/restore instance identity to fence who may act, while preserving business Operation identity so a replacement same-instance recovery can reconcile old effects rather than evade them with a fresh namespace.

For a clone, production write authority is simply absent. If synthetic/test Resources are rebound, they receive their own Resource/credential/Operation domains.

3. Supported restore tooling must create quarantine before exposing normal startup

Avoid a dangerous sequence like:

restore pg_dump -> start normal forge -> later set quarantine flag.

The supported procedure should restore into an isolated/stopped database/runtime, create/advance restore metadata/quarantine under controlled migration privileges, reconcile schema/version, then allow the application to start in quarantine-aware mode.

If the application sees a backup/restore state requiring initialization but no valid current restore epoch/mode, fail startup for write-capable services rather than guessing normal operation.

Document raw pg_restore into a live Forge DB followed by normal service start as unsupported.

4. Old-schema backup restore is migration-under-quarantine, never normal live upgrade

A backup can predate the current application schema.

Protocol:

1. inspect backup manifest/schema version before normal app startup;
2. reject a backup newer than supported code/migrations;
3. restore old compatible snapshot into quarantined/stopped state;
4. run the normal trusted migration chain under restore quarantine, preserving historical evidence versions;
5. verify protected roles/routines/privileges and migration registry;
6. only then run Resource/external reconciliation and post-restore canaries.

Do not launch current worker code against a partially migrated historical schema and call resulting errors recovery evidence.

Application binary downgrade against a newer restored DB is not a rollback mechanism unless separately supported/tested.

5. Recovery domains are extensible trusted Core registrations, not a hardcoded list frozen before later phases

#366 can become implementable after #342 and before #190. It therefore must not require every future canonical subsystem to already exist, nor should later phases fork restore logic.

Define a small trusted restore/recovery-domain contract owned by Core. A recovery domain can contribute, through reviewed code only:

• domain identity/schema version;
• backup coverage manifest contribution;
• quarantine inspection query;
• current revalidation/reconciliation requirements;
• blocker/ready result with stable reasons/evidence refs;
• activation/floor action performed through the domain's own canonical writer;
• post-restore canary.

Core owns global restore epoch/quarantine and combines required-domain results fail-closed. A domain cannot declare another domain ready or widen authority.

Initial domains cover core runtime/Missions/Executions, #335 budget/provider, #341 Triggers, #342 adapters/credentials/Resources, Operations/evidence. Later #190 can register Sentinel/current adverse-authority state using this same interface without #366 importing Sentinel. Reporting (#191) reads restore state; it does not become a recovery authority.

Untrusted Workforce/package/model content cannot register recovery handlers.

6. When #190 lands, restore must re-establish current critical authority blocks before action resumes

Through the generic recovery-domain/#189 adverse-block contracts:

• active backed-up critical findings are revalidated from their current source evidence where required;
• their applicable [FEATURE] Add evidence-based earned autonomy policy engine #189 generic current authority block is restored/rebuilt before privileged admission;
• missing/stale source evidence does not optimistically clear the block;
• findings/events believed to have arisen after the backup but absent locally are part of the lost-horizon/source reconciliation problem;
• clone restores keep these as historical/reporting evidence but do not gain production action authority regardless.

#343 depends on both #190 and #366, so its integration fixture must explicitly prove a restored active critical condition remains action-blocking.

7. Backup copy/restore tests must include source instance still running

Hostile additions:

• restore backup to second host while original Forge remains healthy -> replacement cannot leave quarantine as same instance without old-instance fencing;
• clone/test mode while source runs -> new instance id, no production Trigger/model/external write despite copied DB rows;
• clone gets source production credentials accidentally present in environment -> credential binding/current policy still refuses production action until explicit clone rebind (prefer environment isolation so credentials are absent entirely);
• same-instance replacement reconciles prior Operation using original logical idempotency/external identity, not a new instance-scoped key that duplicates it;
• raw old-schema restore then attempted normal startup -> fail closed before worker processing;
• migration fails midway during restore -> next reboot remains quarantined and restartable;
• future recovery domain missing/unknown for a required canonical subsystem -> global completion gate blocks rather than ignores it;
• active critical Sentinel/adverse block survives same-instance recovery before [FEATURE] VNext Phase 9 — Infrastructure Ops Workforce and persistent side-effect proof #343 action admission;
• test clone dashboard/reporting works read-only without sending provider health generations, notifications, webhooks or external mutations.

The strengthened restore invariant is:

A backup may reproduce historical data, but it never reproduces current execution authority automatically. Same-instance recovery proves the old writer is fenced before resuming; clone restore gets a new identity and no production authority.

[Joncallim]

Implementation handoff supplement — primary code/control seams and PR slicing

To make #366 meet the programme's implementation-handoff standard, inspect these concrete seams first once its dependencies are closed. Current paths may evolve before the issue becomes ready, so re-discover them on main rather than treating names below as immutable.

Primary code/control seams to inspect

• web/db/schema.ts and the current migration registry/migrator — restore epoch/quarantine/recovery-domain persistence, protected writer ownership and historical schema compatibility.
• installer/upgrade/repair/service-startup scripts (scripts/install.sh, scripts/repair.sh, setup/service helpers, web/db/migrate.ts, worker/bootstrap entrypoints) — ensure restored databases are quarantined before normal web/worker processing and old-schema restore migrations occur offline/fenced.
• [FEATURE][ARCH] VNext Phase 0 — generic runtime contracts and compatibility seam #334 canonical runtime/migration authority services — Mission/Execution state revisions, Task compatibility writer, authoritative audit and runtime authority epoch.
• Redis/worker/outbox startup from [BUG][P0] Make Postgres-to-Redis wakeups durable, idempotent, and recoverable #347 — stale transport clearing/fencing and restore-epoch checks before claim/dispatch.
• [FEATURE] VNext Phase 1 — deterministic budget, routing, and context economics #335 budget/provider/credential governance — unknown lost-horizon spend, provider readiness reset and current security/egress admission.
• [FEATURE] VNext Phase 6 — persistent Missions, checkpoints, leases, and bounded autonomy #340 Mission checkpoint/lease recovery — per-Mission recovery dispositions without reopening terminal identities.
• [FEATURE] VNext Phase 7 — Trigger/Event runtime with dedupe, causality, and zero-token idle #341 Trigger definitions/occurrences/schedules — restore activation floors and bounded catch-up.
• [FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342 generic Resource/Capability adapter + credential broker — current external-state reconciliation and re-binding.
• [FEATURE] VNext Phase 2 — secure generic execution envelope and side-effect recovery #336 Operation journal/reconciliation — backed-up uncertain Operations and external lost-horizon state.
• [FEATURE] Add evidence-based earned autonomy policy engine #189 generic adverse-evidence/current authority-block contract — restored elevated autonomy defaults safe/current blocks re-established.
• future [FEATURE] Add Project Sentinel detection and escalation flow #190 Sentinel restore participant — integrate through the trusted recovery-domain/[FEATURE] Add evidence-based earned autonomy policy engine #189 contracts after [FEATURE] Add Project Sentinel detection and escalation flow #190 exists; [FEATURE][RELIABILITY] Add restore quarantine and external-state reconciliation for backup recovery #366 must not reverse-depend on it.
• evidence/Artifact storage layer and backup location/configuration — prove cross-store coverage, integrity and retention without exposing secrets.
• deployment/service-manager/container configuration for the supported Forge install — same-instance old-writer fencing and clone/test-mode isolation.

Suggested reversible PR stack

1. DR contract + durable quarantine foundation — schema, restore modes, restore/runtime epoch, startup fail-closed behavior, protected writer/audit, synthetic tests only.
2. Backup manifest + offline restore/migration tooling — integrity/evidence coverage, same-instance vs clone initialization, old-schema migration under quarantine.
3. Core recovery-domain registry + Mission/Trigger/outbox integration — generic domain interface, stale work fencing, recovery manifests/floors; no external writes yet.
4. Adapter/Operation/credential/budget reconciliation — current-world comparison using [FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342/[FEATURE] VNext Phase 2 — secure generic execution envelope and side-effect recovery #336/[FEATURE] VNext Phase 1 — deterministic budget, routing, and context economics #335, unknown-horizon semantics and human-required paths.
5. Completion gate + same-instance/clone conformance — deterministic lift criteria, repeated reboot/failed-restore tests, source-instance-still-running split-brain fixtures.
6. Realistic DR release proof/docs — populated backup and external fake/service fixtures, sanitized operator procedure, evidence/artifact retention proof.

Keep each slice independently reviewable. Do not let an early schema PR silently enable normal autonomy on a restored database before the completion-gate slices exist; feature/authority mode must remain fail-closed until the full DR contract is proven.

Scope guard

Do not solve this with:

• active-active Forge consensus;
• a second scheduler/outbox;
• raw Hermes/other orchestrator state import;
• a model deciding whether restore is safe;
• generic external Resource resets that destroy current-world evidence;
• instance-id changes that alter logical external Operation idempotency and thereby make duplicate effects easier;
• a clone that keeps production credentials/Triggers active for convenience.

The implementation handoff is complete only when the restore path is demonstrably more conservative than ordinary restart, because it is recovering from locally rolled-back truth rather than a merely interrupted process.

[Joncallim]

Architecture reconciliation — canonical DR owner and positive writer fencing

Reviewed 1 October 2026 against #365/#366 and the restore-mode/recovery-domain addenda. #366 is the sole disaster-recovery implementation owner. #365's body is now tracking-only with its original requirements retained; do not create parallel recovery systems.

Required corrections / retained coverage

1. Unreachable is not fenced. The earlier same-instance addendum includes “unreachable” among old-host conditions. A network-partitioned old instance may still reach external providers and perform writes. Quarantine may lift only after positive external fencing appropriate to the deployment: verified stop/decommission, revoked effective credentials/endpoint authority, or another reviewed mechanism that actually prevents the old writer. Mere inability to contact it is insufficient.
2. Recovery generation cannot live only in the rolled-back snapshot. Retain the explicit supported restore procedure/external-current marker contract. New recovery must not reuse old admission/lease tokens, and normal startup must fail closed if restore initialization is incomplete. Do not claim arbitrary raw database replacement can always be detected.
3. Unknown lost-horizon spend is a budget obligation. Reconcile provider/account usage where attributable, or retain a conservative policy hold/charge with unknown actual usage explicitly labelled. Restored low local totals must not create “free” budget. Do not double-count known backed-up settled usage while accounting for the missing interval.
4. Carry [BUG][P2][RELIABILITY] Prove disaster-recovery restore safety for Forge authority and evidence state #365's key and storage proof into closeout. Verify historical encryption/signing generations, bounded backup/verification, PostgreSQL↔Artifact coverage and actual restore drills. Loss of a necessary historical key is explicit unrecoverable/partial evidence, not permission to drop protected records and continue.
5. Scoped recovery must not release unrelated blocked domains. Each required domain's readiness binds the current restore generation and relevant dependency revisions. If selective resume is supported, enforce a precise allowlist of ready Resource/Mission domains; a partially green recovery summary cannot turn off a global guard for still-unreconciled work.
6. Late old-epoch receipts remain historical. Delayed provider callbacks, outbox messages or signed results from before restore cannot advance a current execution simply because their IDs exist in the restored database. Reconcile exact business identity while enforcing the new actor/authority epoch.
7. Restore verification is not an automatic destructive cleanup. Preserve unreferenced post-backup artifacts and unresolved external evidence in quarantine until classified under retention policy. Deleting them to make DB/store inventories agree can destroy the only clue to a real side effect.

Additional hostile cases

• Old host is unreachable to operator but can still call an external sink: replacement remains quarantined until positively fenced.
• Restored pre-revoke credential state plus active old process: no second writer.
• Partial recovery enables one approved test Resource while another remains uncertain: no cross-domain escape.
• Missing usage interval cannot reset spend allowance; known settled amounts are not charged twice.
• Old-epoch callback arrives after selective resume: historical/reconciliation handling only.
• Missing historical key or orphan artifact prevents the affected evidence claim rather than being silently discarded.
• Repeated restore/quarantine reboot preserves blockers and cannot incrementally widen authority.

Implementation next step

Keep the existing six-slice plan and synthetic resources. Add the positive-fencing test before any same-instance activation proof, and include #365's measured RPO/RTO/key/cross-store checklist in the one release report. #190 remains a later recovery-domain producer; no reverse dependency is added. No backup, restore, host, credential, service or permission action occurred in this review.