replay: prepare status publication and defer checkpoint encoding

docs/status-checkpoint-capture.md

@@ -0,0 +1,98 @@
+# Transaction-status checkpoint capture and encoding
+
+Replay captures immutable lineage and coverage metadata before submitting a
+checkpoint to the promotion worker. Sorting, encoding and writing happen on
+that worker. Capture does not retain parent links outside the selected window.
+Publication and durable-root ordering are unchanged.
+
+Each node memoizes its canonical encoded body on first serialization. Capture
+and pruning share the same cache object when copying a node header; they never
+copy a used synchronization primitive. Encoding depends on the immutable slot,
+block-ID presence/value and status delta, not its parent link. Concurrent
+encoders synchronize through `sync.Once` without taking the live cache lock.
+Each snapshot still constructs its own coverage header and returns an owned
+output buffer. The MTS2 format and restore validation are unchanged.
+
+The cache retains roughly one extra encoded window (30 MB for 1.5 million
+keys), plus any nodes pinned by older views. There is no global encoding map:
+caches become collectible with their last node/view. A completely new window
+still pays for all sorting. Output copying and checkpoint I/O remain necessary.
+
+## Encoding benchmark
+
+`BenchmarkTransactionStatusCheckpointEncoding` uses a 300-root window with
+5,000 keys per root (1.5 million keys, roughly 30 MB encoded). Each iteration
+replaces the specified number of roots. Fixture creation and initial warming
+are excluded; new node headers, sorting and output allocations are included.
+The baseline is the original uncached wire encoder retained in tests.
+
+Apple M4 Pro, Go 1.26.4, one caller, GOMAXPROCS=12; medians of three runs:
+
+| New roots per checkpoint | Original encoding | Cached encoding |
+| --- | ---: | ---: |
+| 1 | 158.05 ms | 1.35 ms |
+| 8 | 157.14 ms | 5.09 ms |
+| 32 | 155.62 ms | 17.51 ms |
+| 128 (default fold cadence) | 153.74 ms | 67.11 ms |
+| 300 (entirely new) | 155.90 ms | 156.29 ms |
+
+At the default cadence, allocated bytes per encoding fell from 99.12 MB to
+57.33 MB; this excludes retained heap. These are encoding measurements, not
+end-to-end fold/replay timings or live FAST improvements. Data distribution
+matters: newly rooted large blocks can account for most keys in the window.
+
+Run `go test ./pkg/replay -run '^$' -bench '^BenchmarkTransactionStatusCheckpointEncoding$' -benchmem -benchtime=1s -count=3`.
+
+Tests compare exact bytes with the original encoder across coverage flags,
+block IDs and sorted groups; check concurrent encoding during pruning/unwind;
+verify cache sharing before and after warming; and restore checkpoints after
+callers mutate their own output buffers. The replay race suite and vet pass.
+
+Related behavior: [status expiry](transaction-status-expiry.md) and
+[status publication](transaction-status-publication.md).
+
+## Native Zen 5 validation
+
+AMD Ryzen 7 9700X, Go 1.26.4, GOMAXPROCS=2, Nice 15 and a two-core CPU quota,
+while the validator continued its normal workload. Same moving-window fixture;
+three samples per case, medians below. This compares the original uncached
+encoder with memoization, not the whole status-publication change against dev.
+
+| New roots per checkpoint | Original encoding | Cached encoding |
+| --- | ---: | ---: |
+| 1 | 195.34 ms | 3.73 ms |
+| 8 | 195.15 ms | 8.56 ms |
+| 32 | 194.77 ms | 23.54 ms |
+| 128 (default fold cadence) | 194.52 ms | 85.02 ms |
+| 300 (entirely new) | 201.88 ms | 198.55 ms |
+
+The default-cadence result is approximately 2.3x, with the same 99.12 → 57.33 MB
+allocation reduction. Cold/all-new windows remain roughly unchanged. Native
+combined race suites, vet and the validator build passed. These are historical staging measurements. They do not establish an isolated
+live reduction in durable-root lag or missed FAST votes.
+
+## Fold admission before collecting account writes
+
+Replay checks for a checkpoint batch on every iteration, including skipped
+slots. `WorkingSet.PromotionChunk` first counts eligible held slots under its
+read lock. If fewer than the configured batch size are available, ordinary
+admission returns nil without allocating account-pointer lists. When ready,
+it collects only the oldest batch, not the entire eligible suffix. Forced
+partial folds still collect the available prefix.
+
+This preflight is not a finality shortcut or a new recovery policy. Replay's
+existing finality/verification gates supply the upper bound. Selection and
+collection hold the same lock; account pointers retain their existing ownership
+contract. Preparation does not prune the suffix or advance the durable root.
+The worker's write/commit order, required resume context, checkpoint reference
+validation, completion bookkeeping, and forced shutdown/epoch-boundary paths
+are unchanged.
+
+`BenchmarkBuildFoldJobWaitingForBatch` holds 127 slots with 512 account writes
+each while waiting for the default 128-slot batch. On Ryzen 9700X,
+GOMAXPROCS=8, three 300 ms runs, median admission-check time fell from 426 µs
+to 31.8 ns; 627,008 bytes and 134 allocations per rejected preparation became
+zero. This measures an ineligible batch check, not encoding, disk I/O, or a
+ready checkpoint. Boundary tests cover gaps, the finality upper bound, a full
+batch, forced partial batches, and selection after promotion; existing replay
+checkpoint/recovery tests cover the unchanged durable path.

docs/status-checkpoint-expiry-evidence.md

@@ -0,0 +1,19 @@
+# Status Checkpoint Expiry: benchmark evidence
+
+The maintained subsystem documentation and reusable Go benchmarks describe the
+implementation and reproduction method. Historical raw results and session
+notes are retained at [the tested source snapshot](https://github.com/Overclock-Validator/mithril/tree/a511ad3b0bc77cf8b5ae4ee16359ac6b453fc7bf)
+(tag `review-evidence-20260916-status-checkpoint-expiry`). They are omitted from this proposed merge.
+
+[Historical result files](https://github.com/Overclock-Validator/mithril/tree/a511ad3b0bc77cf8b5ae4ee16359ac6b453fc7bf/docs/results)
+
+Measurements retain their original baselines. Rebasing onto PR #278 does not
+turn an intermediate-version benchmark into a comparison with the new base.
+Component timings and short live observations do not establish sustained FAST
+inclusion gains. The final review description records validation of the rebased
+source separately from historical benchmark results.
+
+The tagged snapshot also preserves the later 64-partition visible-status-map
+experiment. That experiment is deliberately excluded from this review: it added
+preparation work and did not demonstrate an overall large-block p99 benefit.
+Earlier publication preparation and immutable-node encoding reuse remain.

docs/transaction-status-expiry.md

@@ -0,0 +1,61 @@
+# Batched transaction-status expiry
+
+Applying an asynchronous checkpoint still calls `TransactionStatusCache.Root`
+on replay. Live Zen 5 instruction probes measured 43–101 ms inside that function.
+The previous expiry path visited every key in every retired bank, even when an
+entire recent-blockhash group could be discarded.
+
+Expiry now examines the expired and retained bank deltas by blockhash. It drops
+fully expired groups directly. For a group spanning the cutoff, it either
+subtracts the expired keys or rebuilds the visible reference counts from the
+retained deltas, whichever requires fewer key visits. Retained unrooted banks
+are included. Physical map reclamation is still Go GC work; this is not a claim
+that memory reclamation costs disappear.
+
+The 300-root retention rule, immediate logical expiry, duplicate-key reference
+counts, selected-parent validation, checkpoint format and immutable producer
+views are unchanged. All index changes remain under the existing cache lock.
+This does not move unsafe mutable state to another goroutine or delay expiry.
+A long-lived blockhash with many transactions on both sides of the cutoff can
+still require substantial per-key work. This patch reduces that work to the
+smaller side; it does not give a constant-time worst-case bound.
+
+## Validation
+
+The replay race suite, replay vet and validator production build pass. New tests
+compare exact visible indexes against the original per-key removal for 100
+random lineages with shared hashes, collisions and empty groups, then unwind
+surviving banks. A Root integration test checks pinned producer views,
+checkpoint bytes, restored duplicate detection and rooted-unwind rejection.
+
+M4 Pro, Go benchmark, single caller, two iterations per case. Each iteration
+expires 128 banks of 33,760 unique keys (4,321,280 entries) and retains another
+33,760 entries. Setup is outside the timer. The baseline invokes the original
+per-key removal; the new path invokes batched expiry. These are **expiry-path**
+measurements, not end-to-end Root/replay or a prediction of live FAST scores.
+
+| Recent-blockhash grouping | Old expiry | Batched expiry |
+|---|---:|---:|
+| Groups shared by four expired banks | 185–189 ms | 0.037–0.080 ms |
+| One fully expired group | 604–614 ms | 0.025–0.026 ms |
+| One group shared by expired and retained banks | 590 ms | 1.63–2.36 ms |
+
+Run `go test ./pkg/replay -run '^$' -bench '^BenchmarkTransactionStatusBatchExpiry$' -benchtime=1x -count=2`.
+
+## Native benchmark
+
+Ryzen 7 9700X, Go 1.26.4, original per-key expiry versus batched expiry.
+Benchmarks ran with GOMAXPROCS=2, nice=15, one caller and three iterations per
+case, while the validator and loader remained active. Setup and later GC are
+excluded from the expiry timer. Each case expires 4,321,280 entries (128 banks
+of 33,760) and retains 33,760 entries. These synthetic batches exceed the earlier
+live stall samples and are not an end-to-end replay or FAST-score comparison.
+
+| Shape | Original expiry | New expiry |
+|---|---:|---:|
+| Four-bank blockhash groups | 306–311 ms | 0.049–0.057 ms |
+| One fully expired blockhash group | 700–718 ms | 0.024–0.031 ms |
+| Group crossing the retention boundary | 717–735 ms | 1.85–2.05 ms |
+
+[Historical evidence](status-checkpoint-expiry-evidence.md) preserves the original
+source revisions, raw measurements and validation.

docs/transaction-status-publication.md

@@ -0,0 +1,73 @@
+# Preparing transaction-status publication during execution
+
+Replay previously built the immutable per-bank transaction-status delta and grew the visible duplicate index only after execution and bank-state publication. In a prior live sample of 25 large blocks, TransactionStatusCommit took 7.704 ms median and 10.206 ms maximum. Those live timings motivate this change; they are not the controlled benchmark baseline below.
+
+Count identities by recent blockhash and allocate each delta map at its final capacity. Pre-size newly created visible maps too. For banks with more than 32 transactions and GOMAXPROCS greater than one, prepare the immutable delta during account loading and execution. Smaller banks and single-thread configurations keep the work inline. There is at most one preparation task per ProcessBlock call, and every return joins it, including rejected banks. No status becomes visible during preparation.
+
+The worker reads immutable prepared message identities and briefly snapshots only blockhash slice offsets under the cache read lock. It builds its private maps outside the lock. Commit checks exact block/identity binding, complete coverage and parent lineage under the publication lock. It rechecks ancestor duplicates unless the successful pre-execution validation belongs to the same cache instance, immutable identity set and unchanged cache version (see below). A changed slice offset, disappearance of a previously nonzero-offset group, or mismatched preparation triggers a rebuild from the actual block's identities. Publication still happens only after successful bank-state commit. Failed instructions within an accepted bank remain processed; rejected banks publish nothing. Pinned views, snapshots, reference counts and unwind keep their existing semantics.
+
+TransactionStatusPreparation measures worker wall time, which overlaps execution; it is not additive with replay wall time. TransactionStatusPreparationWait measures the residual join and is nested inside TransactionStatusCommit. The latter still includes waiting, final checks, visible-index updates and node publication. Preparation time excludes initial goroutine scheduling delay; any residual scheduling delay remains in the join/commit timer.
+
+## Native benchmark
+
+AMD Ryzen 7 9700X (Zen 5), Go 1.26.4, GOMAXPROCS=2. Tests ran in a separate process on the validator host with Nice=15 and a 200% CPU quota; the validator and loader continued running. This is a shared-host microbenchmark, with observable timing variation. Five samples per case, ten iterations per sample; values below are medians of sample means, not per-block percentiles.
+
+Each block has 33,760 unique prepared message identities spread across one or four recent blockhashes. Existing-group cases seed 33,760 different ancestor transactions. Fixture creation, hashing, seeding and unwind are untimed. Existing maps retain capacity after unwind: the first timed commit's growth is amortized across the ten iterations. This does not model an index growing indefinitely across live blocks.
+
+The frozen baseline functions exactly match alpenglow-dev commit `33dde4050d9250557583395810799aaac2f54017`. Both versions use the same prepared identities, parent/duplicate checks and fixtures.
+
+| Recent blockhash groups | Parent has keys in these groups | Baseline commit | Sized maps, inline | Preparation + commit, no overlap | Commit after preparation |
+|---|---|---:|---:|---:|---:|
+| 1 | No | 4.990 ms | 3.332 ms | 3.393 ms | 1.587 ms |
+| 1 | Yes | 4.991 ms | 4.657 ms | 4.434 ms | 2.757 ms |
+| 4 | No | 4.625 ms | 3.744 ms | 5.916 ms | 2.205 ms |
+| 4 | Yes | 5.224 ms | 4.644 ms | 4.949 ms | 2.858 ms |
+
+The last column deliberately excludes delta preparation: it measures the work remaining if execution hides preparation completely. It is not total replay or CPU work. Total publication allocations with new groups fell from approximately 6.30 MB to 3.15 MB per block. Existing-group allocation figures include the amortized first growth described above.
+
+The four-new-group total-work sample was slower. Preserve that result rather than claiming improvement in every sample. A subsequent baseline/candidate/candidate/baseline comparison of that same case, with 50 iterations per sample, measured baseline **4.400 and 4.565 ms**, candidate **2.985 and 3.131 ms**. This supports a reduction in work but does not isolate the cause of the earlier timing variation.
+
+## Execution contention and small blocks
+
+A separate controlled benchmark performs 4,096 load-and-execute calls using the existing transfer fixture while preparing 33,760 independent status keys. It does not commit transfer accounts, and its status fixture differs from the repeated transfer fixture. It tests scheduling/allocation contention, not whole-block replay or a valid block workload.
+
+With two Go execution threads, the final implementation measured **20.678 ms baseline**, **18.574 ms with sizing alone**, and **17.091 ms with overlap**. Execution itself measured 15.070, 14.369 and 14.967 ms respectively. Thus preparation competed with execution relative to sizing alone, but the shorter final stage outweighed that cost in this controlled workload. These are separate medians and need not add exactly.
+
+The initial unrestricted version showed no additional total-time benefit from overlap with GOMAXPROCS=1. Tiny-block measurements also showed roughly a microsecond of avoidable scheduling overhead. The final implementation therefore does no background preparation with one Go execution thread or at most 32 transactions. Empty and one-transaction cases retain the baseline allocation counts. The 32-transaction case benefits from sizing without launching a worker. Threshold and single-thread behavior have regression coverage.
+
+## Validation and limits
+
+Full replay and block race suites passed on both Zen 5 and M4 Pro. Metrics has no tests. Native vet for replay/metrics and the validator build passed. Tests cover fork replacement introducing a duplicate after preparation, concurrent sibling publication, stale identity binding, changed snapshot slice offsets, rejected/incomplete banks, mismatched preparation, pinned views, snapshot restore, unwind, empty banks and scheduling boundaries.
+
+Raw logs, source hashes, summaries and the alternating recheck are in [results/status-publication/2026-09-15](https://github.com/Overclock-Validator/mithril/blob/a511ad3b0bc77cf8b5ae4ee16359ac6b453fc7bf/docs/results/status-publication/2026-09-15). The baseline comparison covers only status publication. No live replay or FAST improvement is claimed. The staging binary was not deployed; the existing validator remained active and voting throughout the tests.
+
+Reproduce from this branch:
+
+```sh
+GOMAXPROCS=2 go test -race -p 2 ./pkg/replay ./pkg/block ./pkg/metrics -count=1
+GOMAXPROCS=2 go vet -p 2 ./pkg/replay ./pkg/metrics
+GOMAXPROCS=2 go build -p 2 ./cmd/mithril
+GOMAXPROCS=2 go test ./pkg/replay -run '^$' -bench '^BenchmarkTransactionStatusPublication$' -benchtime=10x -count=5
+go test ./pkg/replay -run '^$' -bench '^BenchmarkTransactionStatus(ExecutionOverlap|SmallPublication)$' -benchtime=100ms -count=5 -cpu=1,2
+```
+
+## Reusing pre-execution ancestor validation
+
+`ProcessBlock` now carries a private validation receipt from its successful ancestor scan to status publication. Under the commit lock, an unchanged receipt avoids scanning all transaction messages again. Publication still checks block binding, complete coverage and parent lineage every time; a missing, foreign or stale receipt performs the full ancestor scan. Direct `CommitBlock` callers retain the full scan.
+
+The receipt is bound to the cache instance and exact immutable prepared-identity pointer. Visible-index insertion/removal, tip binding, root/prune and restore invalidate the version, including empty commits. Committing and then unwinding back to an identical parent cannot revive a receipt. Version saturation disables reuse permanently rather than wrapping. Snapshot/Agave recovery creates a new cache instance. Receipts are never persisted, and no checkpoint format, durability, voting-resume or crash-recovery guarantee changes.
+
+The publication benchmark adds `validated_commit` and `invalidated_commit` alongside `prepared_commit`. All three exclude delta preparation and the pre-execution scan. The first reuses that scan; the second calls `Root` between validation and publication, forcing revalidation. Each iteration unwinds and obtains a fresh receipt outside the timer. These are incremental publication comparisons, not the full PR against alpenglow-dev or per-block tail latency. Tests exercise fork replacement introducing duplicates, concurrent sibling commits, cross-cache and cross-identity misuse, snapshot replacement, pruning/root invalidation, binding changes, transaction replacement and version saturation.
+
+Zen 5 incremental measurement (Ryzen 9700X, Go 1.26.4, GOMAXPROCS=2, five samples × 20 iterations, Nice=19 / 200% CPU quota on the running validator host):
+
+| Recent blockhash groups | Existing ancestor groups | Full recheck | Reused validation | Invalidated validation |
+|---|---|---|---|---|
+| 1 | yes | 2.510 ms | 1.364 ms | 2.536 ms |
+| 4 | yes | 2.412 ms | 1.283 ms | 2.440 ms |
+| 1 | no | 1.461 ms | 1.472 ms | 1.769 ms |
+| 4 | no | 1.323 ms | 1.395 ms | 1.303 ms |
+
+Values are medians of sample means. Existing-group cases remove approximately 1.1 ms of repeated lookup work; new-group cases show no clear gain and shared-host variation. Full native replay/block race suites, targeted node recovery race tests, vet and the combined build passed. Local replay race tests and vet also passed.
+
+Original run artifacts are retained in the [evidence archive](status-checkpoint-expiry-evidence.md).