Release readiness: chain_id, doc accuracy, value-prop benchmarks, tests#13
Closed
KaiCode2 wants to merge 5 commits into
Closed
Release readiness: chain_id, doc accuracy, value-prop benchmarks, tests#13KaiCode2 wants to merge 5 commits into
KaiCode2 wants to merge 5 commits into
Conversation
Final pre-public-release pass closing the gaps surfaced by the readiness review. No behavioral regressions; the full CI gate (fmt, clippy -D warnings, tests, doctests, cargo doc -D warnings, cargo package) stays green. Engine / API - chain_id is now configurable instead of hard-defaulting to Arbitrum (42161). Add `EvmCacheBuilder::chain_id` (recommended) and `EvmCache::set_chain_id`; when unset, infer from the provider via `eth_chainId` and fall back to 1 (Ethereum mainnet). The inference also upgrades the bare `new`/`at_block` constructors. Resolution order: explicit builder value > disk CacheConfig > inferred > 1. (Breaking: the in-memory default changed from 42161 to inferred/1.) - events: `ingest_logs` now tracks `skipped_accounts` so a third-party decoder emitting cold `StateUpdate::Account` patches stays visible to reorg purge and freshness, matching the "every category" contract. Documentation accuracy - Narrow the freshness `Validation::Confirmed`/`Corrected` docs: reconcile covers volatile storage slots only, not account balance/nonce/code. Record the scope in KNOWN_ISSUES. - create3: relabel the factory as the ZeframLou/Solmate `CREATE3Factory` (LiFi-deployed) and disclaim that it is NOT pcaversaccio's CreateX. - access_list: drop the AMM (V2/V3 slot-selection) framing from the module and struct docs; the builder applies no per-entry selection. - Fix the StateDiff skip-accessor docs (all four skip categories), the cold definition (NotExisting overlay accounts), the cancellation/`# Panics`/empty `token_deltas` notes on the freshness optimistic loop, the `load_binary_state` read-error logging, and the overlay `block_hash`/ZERO behavior. Add KNOWN_ISSUES entries for BLOCKHASH-zero, unbounded `derived_slots`, deep-reorg under-purge, and >= 2^255 transfer magnitude. - Remove the never-shipped `SimulationErrorKind` deprecated alias. Benchmarks / packaging / community - Add a README "## Performance" section with measured COW-vs-deep-clone `create_snapshot` and overlay-reuse numbers, methodology, and reproduction. - Cargo.toml: exclude the internal phase specs and CI workflow from the package. - Add SECURITY.md and a dated CHANGELOG 0.1.0 section. Tests - Offline Multicall3 `aggregate3` coverage (input order + allowFailure) by etching the real Multicall3 runtime (new fixture), and a `TxConfig.access_list` EIP-2929 gas-accounting test. New `tests/builder_chain_id.rs` pins the chain_id order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The README "Performance" section led with a copy-on-write `create_snapshot` vs `create_snapshot_deep_clone` table — an internal implementation A/B a first-time reader never chooses between. Replace it with the outcomes a searcher actually cares about, each measured against the fork-per-candidate loop they'd otherwise write. Four pillars, two of them exact CI-pinned integer counts: - **Data-fetch minimization (headline).** New `examples/fetch_minimization_counted.rs` and `tests/fetch_minimization.rs`: 500 candidates over a shared 8-slot hot set fetch 8 slots once (zero during fan-out) vs 4,000 fork-per-candidate — ~500x fewer RPC reads, a deterministic machine-independent count. - **Candidate-fan-out throughput.** New `benches/fanout.rs`: snapshot-once + overlay-per-candidate vs a full independent fork per candidate; per-candidate cost falls as N grows (~545x at N=128 over a 32k-slot fork). - **Reactive sync.** `tests/event_pipeline.rs` pins that `ingest_logs` keeps hot slots fresh with zero RPC fetches/block; `examples/reactive_cache.rs` prints it (0 during ingest, sampled reconcile as the honesty backstop). - **Optimistic latency-hiding.** Promote the existing `benches/freshness.rs` latency arms (time-to-result vs fetch-then-sim) in the README, labeled with the modeled 50ms delay. Honesty guardrails: both shared-hot-set (~Nx) and disjoint (1x, no win) cases are stated; wall-clock ratios are labeled machine-dependent and separated from the exact integer counts; the injected 50ms is called out as modeled, not measured. The internal COW-vs-deep-clone cost model moves to `docs/INTERNALS.md` (the `simulation` bench keeps it as a regression guard). `benches/fanout.rs` is registered in Cargo.toml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
KaiCode2
changed the title
An adversarial review of the benchmark reframe flagged a framing gap: the candidate-throughput baseline (`fork_per_candidate`) is `create_snapshot_deep_clone`, an in-memory deep copy that does NO fetching — so the ~545× throughput speedup is snapshot-construction/isolation cost, not RPC avoidance. The "fair baseline" box implied the baseline cold-fetches, conflating the throughput win (②) with the separate fetch-count win (①). Disambiguate in the README Performance section: ① measures the fetch cost (a cold cache per candidate re-fetches), ② measures the isolation/CPU cost (a full independent clone per candidate, no RPC) — the two are separate and not double-counted. All measured numbers are unchanged and verified accurate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Visuals for the value-prop performance story: - assets/fetch_amplification.svg — the headline: fetches stay flat at 8 while fork-per-candidate grows linearly to 4,000 over 500 candidates (~500x). - assets/fanout_throughput.svg — per-candidate cost (log scale) falls 41→4 µs as N grows while fork-per-candidate stays flat near 2.1 ms (~545x at N=128). Both are self-contained dark-card SVGs (no external CSS) so they render on either GitHub theme; embedded via absolute raw-main URLs so they also resolve on crates.io after merge. - Mermaid state-stack/control-plane flowchart replacing the ASCII diagram in "Core concepts" (lib.rs keeps the ASCII for docs.rs), plus a sequence diagram of the optimistic verify-and-rerun loop showing time-to-result decoupled from RPC validation. Numbers match the measured benchmarks already cited in the Performance section. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… plainly A red-team of the benchmark baselines (steelmanning what a competent revm / foundry-fork-db user actually writes) found the headline multipliers were beating a strawman, not a real peer: - ~500x fewer fetches: a shared foundry-fork-db SharedBackend caches + dedups, so it also fetches each hot slot once within a block (its cache hit path is backend.rs:204-242). Within-block ratio is ~1x, not 500x. - ~545x throughput: the baseline (create_snapshot_deep_clone per candidate) is an O(total-state) copy nobody writes; checkpoint/revert isolation (the crate's own primitive) is O(touched), so single-threaded the gap is ~1x. - ~3,800x latency: ~99.97% of it is the injected 50ms sleep the baseline elects to pay; a competent loop doesn't block on a fetch before acting. I also re-measured the parallelism I expected to be the real win: it is only ~1.2x on these micro-sims (revm is allocation-bound per call, so it contends rather than scaling with cores) — so I do not headline a core-count multiplier either. Rewrites the README "Performance" section to say all this plainly (a candor table showing ~1x where it is ~1x) and lead with what genuinely holds: cross-block freshness (0 RPC fetches/block — foundry-fork-db's cache is not block-keyed), point-in-time consistency, modest-but-real parallel fan-out, and the act-then-validate control plane. benches/fanout.rs now measures sequential vs parallel (the honest comparison), not deep-clone-per-candidate. The two charts that visualized the inflated numbers are removed; a new cross_block_freshness.svg visualizes the one defensible quantitative win. The fetch example prints the shared-backend caveat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 26, 2026
Merged
Owner
Author
|
Superseded by #15. #13 hardened the older |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final pre-public-release pass. Two commits; the full CI gate (fmt, clippy
-D warnings, tests, doctests,cargo doc -D warnings,cargo package) is green and the changes were adversarially reviewed.1. Release hardening (commit 1)
chain_idconfigurable —EvmCacheBuilder::chain_id/EvmCache::set_chain_id; inferred from the provider (eth_chainId) when unset, falling back to1(mainnet) instead of the old hardcoded Arbitrum42161. Breaking (recorded under the dated0.1.0CHANGELOG).Confirmed/Correctedreconcile storage slots only, not account-level state (+ KNOWN_ISSUES entry).CREATE3Factoryrelabel (not CreateX); access_list AMM framing removed; StateDiff skip-accessor + cold-definition docs;load_binary_statelogging; overlayblock_hash; eventsskipped_accountstracking; 4 KNOWN_ISSUES entries.excludeinternal phase specs +.github;SECURITY.md; dated0.1.0CHANGELOG section.aggregate3+TxConfig.access_listEIP-2929 gas + chain_id resolution order.SimulationErrorKindalias.2. Benchmark reframe — searcher value-prop, not COW internals (commit 2)
The README "Performance" section previously led with a copy-on-write
create_snapshotvscreate_snapshot_deep_clonetable — an internal implementation A/B a first-time reader never chooses between. Replaced with the outcomes a searcher cares about, each vs the fork-per-candidate loop they'd otherwise write:examples/fetch_minimization_counted.rs+tests/fetch_minimization.rs(CI-pinned, machine-independent).benches/fanout.rs— per-candidate cost falls as N grows (~545× at N=128 over a 32k-slot fork; ~250k vs ~460 candidates/sec).tests/event_pipeline.rspinsingest_logskeeps hot slots fresh with 0 fetches/block;examples/reactive_cache.rsprints it (sampled reconcile = honesty backstop).benches/freshness.rslatency arms (~14 µs time-to-result vs ~54 ms fetch-then-sim), labeled with the modeled 50 ms delay.Honesty guardrails: shared-hot-set (~N×) and disjoint (1×, no win) cases both stated; wall-clock ratios labeled machine-dependent and separated from exact integer counts; the 50 ms is called out as modeled. The internal COW-vs-deep-clone cost model moved to
docs/INTERNALS.md(kept as a regression guard).🤖 Generated with Claude Code