feat(pxe): constrained tag sync optimization and recipient logs sync benchmarks (port #24275 to next)#24948
Draft
AztecBot wants to merge 1 commit into
Draft
feat(pxe): constrained tag sync optimization and recipient logs sync benchmarks (port #24275 to next)#24948AztecBot wants to merge 1 commit into
AztecBot wants to merge 1 commit into
Conversation
…benchmarks (#24275) Fixes [F-704](https://linear.app/aztec-labs/issue/F-704/optimize-constrained-tag-sync) Constrained delivery emits a gapless tagging-index stream, so PXE can stop scanning at the first missing constrained tag instead of probing the full unfinalized window on every sync. This PR: - Starts constrained scans with a small probe and **doubles** the probe size while every probed index has a log, capped at the existing window. - Keeps steady-state polling cheap: one tag per constrained secret. - Reduces catch-up round-trips geometrically until the probe reaches the window cap. - Decouples probe advancement from finalized-cursor persistence so unfinalized logs are still fetched without persisting unsafe cursors. - Leaves unconstrained sync and `findHighestIndexes` unchanged. Benchmarks compare doubling against fixed-size probes. The takeaway is that doubling preserves the steady-state tag-query floor of `P=1` while avoiding `P=1`'s one-round-trip-per-log catch-up behavior. A sync runs in rounds: each round computes the next batch of tags, sends them to the node, and blocks on the result before deciding the next round. All counts are per sync. - **tag-queries** — total tags looked up on the node. - **round-trips** — sequential client waits. A round's tags are chunked into parallel calls internally, so a wide round is still one round-trip. - **blocking-ms** — measured wall-clock the caller spends blocked on the node with modeled node latency. This is reported only and varies run to run. Scenarios seed a recipient that already synced prior messages, then measure the next single sync. `secrets = N` means N independent sender streams synced together in one batched pass. - **steady-state** — no new logs since the last sync. - **catch-up-K** — K new contiguous logs per secret since the last sync. - **mixed** — 999 idle secrets plus 1 deep straggler (`K=100`) at 1,000 secrets. **doubling is the shipped policy.** The fixed-P columns are the selection comparison that motivated it: `P=84` is the current full-window behavior, and `P=1..5` sweep the constant-step alternative. Unconstrained is not shown as its own row: its windowed scan cannot first-miss, so it is invariant to P and reproduces the `P=84` column. **Tag-queries (thousands)** (exact count = value x 1,000; bold = fewest among the fixed-P columns; `doubling (init=1)` is the shipped policy, `doubling (init=2)`/`doubling (init=4)` start the probe at 2/4 instead of 1): | scenario | doubling (init=1) | doubling (init=2) | doubling (init=4) | P=1 | P=2 | P=3 | P=4 | P=5 | P=84 | |---|---|---|---|---|---|---|---|---|---| | steady, 100 | 0.1 | 0.2 | 0.4 | **0.1** | 0.2 | 0.3 | 0.4 | 0.5 | 8.4 | | steady, 1,000 | 1 | 2 | 4 | **1** | 2 | 3 | 4 | 5 | 84 | | catch-up-1, 100 | 0.3 | 0.2 | 0.4 | **0.2** | 0.2 | 0.3 | 0.4 | 0.5 | 8.4 | | catch-up-1, 1,000 | 3 | 2 | 4 | **2** | 2 | 3 | 4 | 5 | 84 | | catch-up-3, 100 | 0.7 | 0.6 | 0.4 | **0.4** | 0.4 | 0.6 | 0.4 | 0.5 | 8.4 | | catch-up-3, 1,000 | 7 | 6 | 4 | **4** | 4 | 6 | 4 | 5 | 84 | | catch-up-84, 100 | 12.7 | 12.6 | 12.4 | **8.5** | 8.6 | 8.7 | 8.8 | 8.5 | 16.8 | | catch-up-84, 1,000 | 127 | 126 | 124 | **85** | 86 | 87 | 88 | 85 | 168 | | catch-up-50, 100 | 6.3 | 6.2 | 6 | **5.1** | 5.2 | 5.1 | 5.2 | 5.5 | 8.4 | | catch-up-50, 1,000 | 63 | 62 | 60 | **51** | 52 | 51 | 52 | 55 | 84 | | catch-up-100, 100 | 12.7 | 12.6 | 12.4 | **10.1** | 10.2 | 10.2 | 10.4 | 10.5 | 16.8 | | catch-up-100, 1,000 | 127 | 126 | 124 | **101** | 102 | 102 | 104 | 105 | 168 | | mixed, 1,000 (999 idle + 1 deep K=100) | 1.126 | 2.124 | 4.12 | **1.1** | 2.1 | 3.099 | 4.1 | 5.1 | 84.084 | **Round-trips** (bold = fewest among the fixed-P columns; `doubling (init=1)` is the shipped policy, `doubling (init=2)`/`doubling (init=4)` start the probe at 2/4 instead of 1): | scenario | doubling (init=1) | doubling (init=2) | doubling (init=4) | P=1 | P=2 | P=3 | P=4 | P=5 | P=84 | |---|---|---|---|---|---|---|---|---|---| | steady, 100 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | | steady, 1,000 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | | catch-up-1, 100 | 2 | 1 | 1 | 2 | **1** | 1 | 1 | 1 | 1 | | catch-up-1, 1,000 | 2 | 1 | 1 | 2 | **1** | 1 | 1 | 1 | 1 | | catch-up-3, 100 | 3 | 2 | 1 | 4 | 2 | 2 | **1** | 1 | 1 | | catch-up-3, 1,000 | 3 | 2 | 1 | 4 | 2 | 2 | **1** | 1 | 1 | | catch-up-84, 100 | 7 | 6 | 5 | 85 | 43 | 29 | 22 | 17 | **2** | | catch-up-84, 1,000 | 7 | 6 | 5 | 85 | 43 | 29 | 22 | 17 | **2** | | catch-up-50, 100 | 6 | 5 | 4 | 51 | 26 | 17 | 13 | 11 | **1** | | catch-up-50, 1,000 | 6 | 5 | 4 | 51 | 26 | 17 | 13 | 11 | **1** | | catch-up-100, 100 | 7 | 6 | 5 | 101 | 51 | 34 | 26 | 21 | **2** | | catch-up-100, 1,000 | 7 | 6 | 5 | 101 | 51 | 34 | 26 | 21 | **2** | | mixed, 1,000 (999 idle + 1 deep K=100) | 7 | 6 | 5 | 101 | 51 | 34 | 26 | 21 | **2** | **Blocking wall-clock (ms)** (reported only, noisy; the three `doubling` columns are from one paired re-run, the fixed-P columns from the earlier comparison sweep, so not every column is from a single run): | scenario | doubling (init=1) | doubling (init=2) | doubling (init=4) | P=1 | P=2 | P=3 | P=4 | P=5 | P=84 | |---|---|---|---|---|---|---|---|---|---| | steady, 100 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 8 | | steady, 1,000 | 6 | 7 | 7 | 7 | 5 | 7 | 7 | 8 | 21 | | catch-up-1, 100 | 14 | 8 | 8 | 13 | 8 | 8 | 8 | 8 | 12 | | catch-up-1, 1,000 | 32 | 28 | 31 | 32 | 27 | 29 | 31 | 28 | 54 | | catch-up-3, 100 | 23 | 18 | 13 | 29 | 18 | 18 | 12 | 12 | 15 | | catch-up-3, 1,000 | 77 | 77 | 71 | 83 | 73 | 79 | 64 | 67 | 95 | | catch-up-84, 100 | 207 | 215 | 211 | 663 | 407 | 335 | 321 | 265 | 205 | | catch-up-84, 1,000 | 1,886 | 1,893 | 1,986 | 2,226 | 1,942 | 1,935 | 1,871 | 1,830 | 2,000 | | catch-up-50, 100 | 132 | 132 | 132 | 397 | 243 | 193 | 171 | 161 | 127 | | catch-up-50, 1,000 | 1,162 | 1,167 | 1,193 | 1,307 | 1,169 | 1,191 | 1,114 | 1,108 | 1,217 | | catch-up-100, 100 | 244 | 244 | 260 | 784 | 480 | 396 | 348 | 321 | 245 | | catch-up-100, 1,000 | 2,242 | 2,265 | 2,361 | 2,632 | 2,348 | 2,341 | 2,212 | 2,236 | 2,366 | | mixed, 1,000 (999 idle + 1 deep K=100) | 43 | 37 | 31 | 590 | 296 | 199 | 150 | 124 | 32 | `P=1` is the tag-query floor, but it pays one round-trip per new log during catch-up. The full window (`P=84`) minimizes catch-up round-trips, but it charges every idle secret the full-window tag cost on every sync. Doubling is the middle ground: it matches `P=1` at steady state, stays close to `P=1` on tag queries during catch-up, and collapses deep catch-up round-trips geometrically. In the mixed scenario, doubling uses 1,126 tag queries vs 1,100 for `P=1`, but needs 7 round-trips instead of 101; compared with `P=84`, it avoids the 84,084-tag idle tax while staying within 5 round-trips of the full-window catch-up path. - Unit tests cover scan shape, doubling behavior, batched-round semantics, and returned-log equivalence. - Benchmark tests report tag queries, round-trips, and modeled blocking time for steady-state, catch-up, mixed, and unconstrained scenarios.
AztecBot
added
ci-draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port of #24275 to
next.This carries the constrained recipient tag sync optimization forward: constrained scans start with a small probe, double while every probed index hits, and stop at the first missing constrained tag. It also ports the recipient log sync benchmark coverage added by the source PR.
Conflict resolution
The automatic cherry-pick conflicted in
yarn-project/pxe/src/tagging/constants.tsbecausenexthad reducedUNFINALIZED_TAGGING_INDEXES_WINDOW_LENtoMAX_PRIVATE_LOGS_PER_TX, while the source PR restored the+20headroom and addedINITIAL_CONSTRAINED_PROBE_LEN. I kept the source PR's updated constant/comment block because the constrained sync optimization is what makes the larger window cheap again.origin/port-to-next-stagingwas not present on the remote when this was handled, so the local port branch was based onorigin/next, matchingscripts/backport_to_staging.sh's fallback when the staging branch does not exist.Testing
JEST_MAX_WORKERS=1 yarn workspace @aztec/pxe test src/tagging/recipient_sync/sync_tagged_private_logs.test.ts src/tagging/sender_sync/utils/load_and_store_new_tagging_indexes.test.tsNotes
./bootstrap.sh build yarn-project,yarn build, andyarn workspace @aztec/pxe buildwere attempted. They are blocked before PXE by generatednoir-protocol-circuits-typesartifacts in this local setup after bootstrapping Noir via released binaries, with missing generated modules such asprivate_kernel_reset_data.jsandtypes/index.js. The focused PXE suites above pass.Created by claudebox · group:
slackbot· Slack thread