chore(port): forward-port v5-next prover + node/world-state backlog to next#24933
Draft
PhilWindle wants to merge 4 commits into
Draft
chore(port): forward-port v5-next prover + node/world-state backlog to next#24933PhilWindle wants to merge 4 commits into
PhilWindle wants to merge 4 commits into
Conversation
…state (#24578) v5 version of #24558 (same change, based on `merge-train/spartan-v5`). ## Problem When the prover broker cancelled a proving job it settled the job as a cached `{ status: 'rejected', reason: 'Aborted' }` result — in memory and in the database. Because a job's id is a deterministic hash of its inputs, any later enqueue of the same job returned that cached abort (`Cached proving job … Not enqueuing again`) and the facade failed the job with `Aborted`. So once a proof was aborted it stayed aborted, and a retry — a fresh epoch attempt after a reorg/failure, or a re-orchestration after a restart — got the cached abort instead of re-proving. A planned mid-epoch redeploy hit this and turned into a testnet epoch prune. ## Fix Cancellation is now its own first-class, **revivable** state: - A new `aborted` proving-job status (`ProvingJobStatus` / `ProvingJobSettledResult`). `cancelProvingJob` records it, notifies the current waiter, and **persists** it (`ProvingBrokerDatabase.setProvingJobAborted`). - `enqueueProvingJob` **revives** an aborted job: when the producer re-requests it, the broker clears the aborted state — in memory and, via the new `deleteProvingJobResult`, in the database — and re-enqueues it as if new. So a restart mid-revival keeps the job pending rather than resurrecting the abort. Genuine agent errors and timeouts still settle as terminal `rejected` results. Net effect: cancelling frees the job cleanly and records why, but re-requesting the same proof always brings it back to life — in the same process or after a restart — so an abort can never permanently block an epoch from proving. The stacked PR (prover-node clean-shutdown, v5) stops a clean prover-node shutdown from cancelling its in-flight jobs in the first place. ## Tests `proving_broker.test.ts` (both DB variants): cancel leaves a job `aborted`; re-request revives it and it completes; the aborted state persists across a restart; and a revived job survives a restart mid-revival as pending (not aborted). Note: not run locally in this session — the prover-client suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit a4e88ca)
…poch (#24436) Resolves A-1290. ## Problem A `CheckpointProver`'s sub-tree work forks world-state per block. When an L1 reorg prunes a base block, those fork reads fault and the prover permanently rejects its one-shot `blockProofs` promise. The store keeps the prover alive across a prune (`markPruned`) so a re-add of identical content can reuse its in-flight work. But a prune that removes the checkpoint also breaks the fork reads, so that reuse only ever hands back a poisoned prover: a re-add or a full `EpochSession` recreate re-references the same rejected `blockProofs` and fails immediately. The affected node then silently abandons the epoch until a process restart (a healthy prover elsewhere still proves it, so it's not a network stall). ## Fix **Drop the reuse machinery — a pruned prover cannot survive, so rebuild instead of reuse.** - `CheckpointStore.cancelAndRemoveAboveBlock` cancels and **removes** orphaned provers (replacing `markPrunedAboveBlock`); a re-add builds a fresh prover with a fresh `blockProofs`. - Removed the `pruned` flag / `markPruned` / `markCanonical` from `CheckpointProver`, and the `SlotWatcher` that only reaped lingering pruned provers. - `listCanonical` / `addOrUpdate` simplified — every prover in the store is now canonical. **Recover the epoch by rebuilding on re-add, rather than trying to prevent the terminal state.** A prune unwinds world-state as soon as it's detected — before the prover-node cancels the affected session — so a checkpoint prover mid-fork faults while still live and its session goes terminal `failed`. That race is inherent (the world-state rejection and the prover being cancelled are unordered), so instead of trying to reclassify it as a cancellation, we make the terminal state harmless: - `SessionManager.openFullSessionIfReady` no longer treats a terminal session as "already open" — it drops it and rebuilds. So re-adding an epoch's checkpoints reopens a fresh live session over fresh provers, via the checkpoint/prune triggers (ungated by the tick high-water mark). This reopen depends on the store no longer holding the poisoned prover: with the reuse machinery gone the rebuilt session gets a fresh prover with a healthy `blockProofs`, so it can actually prove rather than re-inheriting the rejected one. - `runSession` skips the failure-upload when the session's checkpoints no longer match canonical content — a prune-invalidated failure isn't a genuine proving failure, so it no longer emits a spurious post-mortem upload / alert. The only non-recovered case is "content pruned and never re-added" (chain stops publishing that epoch's checkpoints) — there's nothing to prove on this node then, so abandoning the epoch is correct. **Cancel the pruned prover's in-flight work promptly.** `CheckpointProver` now threads its abort signal into `PublicProcessor.process`, so a prune-driven cancel stops the current block's public execution immediately instead of running it to completion before the next `signal.aborted` check. This is independent of the recovery logic above — it just avoids wasting CPU re-executing a block whose checkpoint is being discarded. The in-flight block has not yet enqueued a broker proof, so aborting it loses no reusable work. ### Note: broker-level proof reuse is unaffected Removing the `CheckpointProver`-level reuse does not waste proving work. The expensive part — the SNARK proofs — is cached in the proving broker, content-addressed by job id, independent of world-state. `cancelJobsOnStop` defaults to `false`, so cancelling a prover does not abort its broker jobs; on an identical re-add the fresh prover regenerates identical inputs, and the broker dedups against the cached jobs/results (bounded by the broker's per-epoch cleanup). The reuse we deleted was witness-generation reuse, whose world-state forks cannot survive a prune anyway. ## Testing - `session-manager.test.ts`: pins the retry/recovery invariant — the periodic tick does not re-attempt a failed epoch (gated by `lastTickEpoch`), but a checkpoint re-add recovers it (ungated); a `failed` session's epoch is reopened once its checkpoints are re-added; a genuine proving failure still uploads a post-mortem; a failure coinciding with a canonical content change (prune) skips the upload. (Red/green verified for both the gate and the upload suppression.) - `checkpoint-store.test.ts` / `prover-node.test.ts`: pruned provers are cancelled and removed (not flagged); a re-add builds a fresh prover. - `checkpoint-prover.test.ts`: cancelling a prover mid-block aborts the signal its public execution is running under (red/green verified). - Full prover-node suite passes (166 tests); package typechecks clean; lint OK. - Updated `optimistic.parallel.test.ts` (the e2e that spawned this issue) to the cancelled+removed semantics. ## Notes - No operator/contract-dev-facing config or API changed, so no changelog entry. - The terminal `failed` state is not prevented, only recovered from: the world-state unwind and the prover cancellation are unordered, so the failure can't be reliably reclassified from the session side. Recovering on re-add is deterministic and needs no such race to be won. (cherry picked from commit c1e6229)
Upstreaming the utilities build for `aztec-kit` so external projects can benefit from our e2e scaffolding. Essentially the same thing we already use, but pure ts rather than the unpublishable .sh script. Verified with `ci-full-no-test-cache` (cherry picked from commit 7e09008)
…l validation (#24694) ## Problem On a live mainnet node, checkpoint-proposal validation could throw an uncaught error out of the gossipsub message handler whenever it ran while the world-state synchronizer trailed the archiver (block source) by a block: ``` ERROR world-state:database Call CREATE_FORK failed: Error: Unable to initialize from future block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree ERROR p2p:libp2p_service Error handling gossipsub message: Error: Unable to initialize from future block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree ``` The archiver already had block N (so the block lookups inside checkpoint validation succeeded), but world state's `unfinalizedBlockHeight` was still N-1. When `validateCheckpointProposal` forked world state at the parent block before it had been applied, the raw tree error escaped as an uncaught ERROR-level gossipsub message. The node lost that checkpoint validation/attestation for the round and spammed error logs; it self-healed the next tick, but it should not happen. ## Root cause The two world-state fork sites were asymmetric. The block re-execution path (`reexecuteTransactions`) syncs world state *before* forking. The checkpoint validation path (`validateCheckpointProposal`) forked via `checkpointsBuilder.getFork` *without* a preceding sync. The checkpoint path does sync the block source (archiver) earlier, but that does not advance the world state, so the gap remained. ## Fix - Move the sync-before-fork invariant into `FullNodeCheckpointsBuilder.getFork`: it now calls `worldState.syncImmediate(blockNumber, blockHash)` before `worldState.fork(blockNumber)`, so the single fork helper owns the invariant rather than each caller. `syncImmediate` blocks until world state reaches the block, or throws a typed error if it genuinely cannot, instead of leaking the raw tree error. - In `validateCheckpointProposal`, look up the parent block's hash from the block source and pass it through so the sync re-syncs on a world-state reorg. Wrap the fork in a try/catch that maps any failure to a clean, non-slashable `world_state_not_synced` result, so a sync/fork failure never surfaces as an uncaught ERROR-level gossipsub message. - As a second, independent guard, check that the forked world state's archive root matches the proposal's `lastArchiveRoot` before rebuilding the checkpoint. A mismatch means world state forked from a different chain than the proposal was built on, so we fail fast with a clean, non-slashable `initial_archive_mismatch` result — mirroring the block-proposal re-execution check — instead of a confusing downstream header/archive mismatch. Both new reasons record an `unvalidated` outcome and are non-slashable, matching the other "we couldn't validate right now" / local-state-divergence reasons. Other checkpoint-builder fork sites were reviewed and need no change: the proposer's `checkpoint_proposal_job` forks at its own already-synced, coherence-checked world-state tip, and the block-proposal re-execution path already syncs before forking and performs the equivalent archive-root check. ## Tests - `checkpoint_builder.test.ts`: `getFork` syncs world state to the block before forking, and propagates a sync failure without forking. - `proposal_handler.test.ts`: forks at the parent block number passing its block hash; a fork failure returns `world_state_not_synced`; and a fork whose archive root diverges from the proposal returns `initial_archive_mismatch`. ## Notes The same bug exists on the v4 line — the v4 site is the same function, forking via `this.worldState.fork(parentBlockNumber)` directly without a preceding `syncImmediate`. This fix should be ported there as well. Fixes A-1421 (cherry picked from commit b7a7d1b)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-ports the prover and node / world-state / barretenberg slices of the v5-next → next backlog (work merged to
v5-nextafter the ~2026-07-08 cut that reshapednext).Applied (clean cherry-picks, chronological)
next; not included here)Cherry-pick onto this branch and resolve:
git cherry-pick -x a4e3a445ee— feat(world-state): support prefilled nullifiers in genesis state (feat(world-state): support prefilled nullifiers in genesis state #24567) [RESHAPE — v6 subsystem rewritten; re-implement, do not merge]git cherry-pick -x c44e74e5ae— fix(prover-node): do not abort in-flight proving jobs on a clean shutdown (fix(prover-node): do not abort in-flight proving jobs on a clean shutdown #24579)git cherry-pick -x 1039d38148— feat: allow custom proof submission target address (feat: allow custom proof submission target address #24270)Part of the manual v5-next → next backlog sweep. Draft until conflicts are resolved and CI is green.