perf: synchronous filter fast path on a dense-handle core (2.3× on 50k filter settle) - #487
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… queries Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tale window visible Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…placing Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One resolution helper per root shape: a flat root answers from its visible tree, a grouped root from group-index leaf membership. Nothing reads it yet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every consumer of `CompiledRowMetadata.filterPasses` now asks the structure that owns the answer: the committed root's membership for OLD verdicts, a freshly computed verdict for NEW ones. The field is still written; nothing reads it. Behavior unchanged. `filterVerdict` gains a memo on the plan's existing evaluation cache entry, so a producer that evaluates a row and then asks for its verdict still costs one accessor pass — the per-row work budgets are pinned exact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`CompiledRowMetadata.filterPasses` and `CompiledAggregateLeaf.filteredLeaf` are deleted, and `refilterRecordMetadata` with them. A row's verdict is its membership in the root's visible structure — the flat visible tree, or the group index's leaf trees — and a NEW verdict is computed by the producer that places the row, never stored. The point of the exercise: a filter-only change now reconstructs NO record. The rows HAMT carries by identity exactly as the sort fast path's does, and a flip is expressed purely by where the row sits in the new visible tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… seam Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gation `iterateEntries` walked the tree with `yield* iterateEntries(node.left)`, so every element leaving a leaf bubbled out through one generator frame per tree level — ~17 at 50,000 rows, ~850,000 resumptions to walk 50,000 entries. Measured in isolation at 50k: 30.04ms for the delegating shape, 1.77ms for an explicit-stack walk of the same tree. On the real 50k filter-metadata commit that one function body was the largest single frame in the profile. Rewrites both order-statistic-tree walks (persistent and transient) and the HAMT's walk in `persistent-map` as explicit-stack generators. The contract is unchanged: lazy, in-order, same yielded values, and the transient walk still checks the draft's liveness on first resumption and before every element. Each carries a comment with the measurement, because the delegating version is shorter and reads like the obvious simplification. Also reroutes the five callers that walk a tree to completion into an array — `filter-rebuild` (both), `sort-rebuild`, `row-store`, `create-local-row-model` — to the already-shipped `range(0, size)`, which does the same walk without suspending at all (1.05ms at 50k). The lazy callers are left alone deliberately: `transaction-draft`'s visible walk breaks on the first unaffected entry, and the cooperative-transition and distinct-values walks are iterators stepped across scheduler slices. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion cache A filter-only plan change leaves every cached per-row field valid: rowId and sourceOrder are guard fields re-checked against the live input, groupPath and aggregateLeaves and sortKeys are functions of facets the classifier holds identical (groups, derivations including accessor identity, sort), and the row object is the map key. So the next plan takes the previous plan's whole evaluation cache BY REFERENCE — one assignment — instead of walking every row to refill a store with value-identical copies. The one filter-dependent field is the cache entry's verdict memo, which H1 left behind when it removed `filterPasses` from the metadata itself. It is now tagged with the plan that wrote it, so an adopting plan never reads a verdict its own filters did not produce and runs exactly the accessor pass it ran before. The tag costs one property write inside a write that already happens; there is no new per-row work anywhere. `sortKeyCarries` stops incrementing on this path — the walk it counted was a 100%-carry walk, i.e. precisely the redundant work removed — so `evaluationCacheAdoptions` pins the replacement instead. Measured at 50k rows (Node, four interleaved A/B rounds, load ~7-8): median settle 100.5ms -> 89.4ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ion) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…OW tracking Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hip, guarded ops Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tions Dense generations now run refilter and reorder on the slot lane: survivors resolve by denseKey against a slot-indexed array with zero identity strings (only entrants compute an identity, because measurements and tombstones stay string-keyed in both lanes — Amendment I §3), the duplicate check and next membership share one bitset exactly like the builder ingest, and measured leavers retire in OLD-SEQUENCE order so tombstone ticket assignment — which is observable through cap eviction — matches the string lane bit for bit. Both interim Task-2 throws are gone; their tests became real dense-path tests, the only sanctioned existing-test edits. Ride-alongs from the Task 2 review: retainMeasurement rejects malformed dense keys before the bit test (a fractional key's &31 truncation would read a different row's bit), apply's remove/move/update arms reject an operation whose denseKey drifted from the entry's stamped slot, and replace() documents its deliberate lane exit. Pins: a seeded lane-equivalence oracle (200 rows, 30 mixed steps) comparing geometry, retention, and lane-independent work counters after every step; the ticket-order/cap-eviction pin (slots deliberately anti-ordered vs the sequence); and the Amendment §3 slot-reuse trap (new identity on a reused slot ingests at estimate; the old measurement returns only for the old identity). Both pins verified by mutation: slot-ordered leavers fail the ticket pin; a slot-keyed measurement store fails the §3 trap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ot-pooled refs Task 5 of the dense layout seam (Amendment I): the row-layout controller now feeds layout-core's dense lane from the row model's internal slot seam. - replacementSourceOf declares denseCapacity and stamps every entry's denseKey when the snapshot supplies the ɵ slot seam; the visible set is materialized through chunked bulk range walks (one maxUnitsPerSlice-sized read per chunk, lazy) instead of per-row O(log n) rowAt descents. The string lane keeps the per-row rowAt shape verbatim — structural snapshot wrappers (react's bounded-read guards) legitimately refuse wide range spans, and only ɵ-supplying real model snapshots take the bulk walk. - Incremental change operations and prepareWindow estimate updates are slot-stamped (before-first resolution so removes still resolve); staged measurement replay passes the slot to retainMeasurement, and a staged measurement whose row was permanently removed drops that one generation to the string lane (the amendment's wholesale escape hatch) so identity-keyed retention survives — pinned by the existing removed-then-reinserted test. - Data-row refs are pooled by slot: one frozen ref per bound (slot, rowId), reused across sources and window publications. Verified: every ref comparison goes through identityOf/sameRef, none by allocation identity. - Dense-contract refusals keep the honest fallback signals: refilter and reorder dispatch throws land in refilterFallbackCount and reorderFallbackCount; apply-path throws restart via the existing replacementStartCount-observable convention. Ride-along (comments only, from the Task 3 quality review): the string lane-pin note at both dense dispatches, the trust-boundary note in both dense docblocks, and cross-references between the lane-equivalence oracle and the leaver ticket-order pin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An end-to-end react pin drives a 200-row grid through filter-on -> narrow -> widen -> filter-off and asserts the DENSE refilter path ran (refilterPathCount +4, refilterFallbackCount 0), the mounted index is actually dense (layout-core's unkeyed-op refusal as a lane probe), and a measured row's height survives a flip-out/flip-in. Mutation-checked: unstamping denseKey in the dense source fails the probe; a string-lane refilter source fails the counters. Ride-along hardening: a DENSE build whose chunked source read throws (a spread-based snapshot wrapper carrying the seam with its own bounded-read guard) now takes the amendment's string-lane escape hatch for one generation instead of the generic failure path; the next full replacement re-decides dense. Pinned with a bounded-restart test. API reports regenerated: the three optional @internal slot-seam members on PretableRowModelSnapshot, mirrored in core and react — nothing else. Docs api-surface guard green with no table updates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add `readonly slot: number` to CompiledRowInput (Amendment J §1, dense handle for columnar cells; unread this task). Thread the field through every production call site: cooperative-transition.ts stamps it from the record, transaction-draft.ts's createRecord already took a slot param, and rebuildRowStoreForQuery (dead code, zero callers) gets it mechanically from the carried record. buildRowStore and replaceFlatRowsDraft need a slot before evaluate() runs (evaluate can throw via a user accessor), but their real slot for a brand-new row is only resolved by input.slots.allocate() AFTER evaluate succeeds — allocating earlier would leak allocator capacity (a monotonic high-water mark; release() cannot undo it) on a throwing accessor. Both keep that exact original ordering/side effects and pass a harmless placeholder (-1, or the carried previous slot when known) into evaluate()'s input only, since nothing reads the field yet. Test fixtures across 6 row-model test files and one react test file needed `slot` added to CompiledRowInput-shaped literals; no assertions changed. CompiledRowInput does not appear in any governed public package (core/react/ui/stream-adapter) or their .api.md reports — row-model is `"private": true` and outside the `pnpm api` gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
evaluateFilter's per-row operator dispatch becomes compileFilterPredicate: one monomorphic (value) => boolean closure per runtime filter, built at plan construction with operand normalization hoisted (between bounds min/maxed once, date operands collapsed to UTC day-ms once, text needles lowercased once, selection operands coerced into a Set once). Predicate semantics now exist exactly once, in the compile step; #filterVerdict walks the construction-time #compiledPredicates array parallel to #runtimeQuery.filters — no #byId lookup per row. Exhaustive pinned-literal sweep over all 31 (column type, operator) pairs in FILTER_OPERATORS, boundary inclusivity included; mutation- checked (exclusive lower between bound fails exactly the boundary cases). Full suite 602 green, zero existing-test edits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The filter-only rebuild's O(n) walk now takes each row's verdict from bulkFilterVerdictScan: per filter, in filter order, the columnar cell for (column, slot) answers when present, and a hole falls back to the live accessor through the shared #readColumnValue seam AND writes through — the store's only writer (Amendment J §3 revised). One-pass-per-slot for cell locality; short-circuit exactly like the per-row .every, so a failing row may leave later filters' cells unfilled (holes refill lazily). The per-row filterVerdict stays untouched for k-sized and grouped callers. Also closes the Task 3 review's stale-cell laundering hole: setDerivations' plan-REUSE branch now resets the columnar store, because derivationsEqualForPlan ignores UNREFERENCED columns' accessors and one intermediate filter-only adoption would otherwise put a new-accessor plan on a store still holding the old accessor's cells. Work counters: columnarVerdictScans (one per rebuild) and columnarCellFills (per hole filled); a second filter-only commit on unchanged data is pinned at ZERO fills. New tests: seeded randomized columnar-vs-per-row equivalence oracle (updates, slot reuse, setRows, newly-referenced column mid-script), the laundering sequence, and accessor-failed shape parity between scan and per-row paths. All four covering mutations verified red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The filter rebuild's verdict pass is now ONE bulkFilterVerdictSweep call per rebuild (walk, plan resolution, predicate/column/vector hoisting all inside the plan; assert-free trusted cell reads over walk-produced slots), and the columnar store caches SCAN-NORMALIZED cells (text lowercased, dates as day-ms, enum/boolean coerced) filled once, with normalized predicate twins compiled per plan. isEmpty/isNotEmpty stay on raw accessor reads — emptiness is a raw-value property the normalized forms lose. Re-measured paired at 50k: STILL FLAT (+0.9/+0.4ms medians, controls in band); traced verdict share ~15.4% vs ~17.1%. The bench scripts' single cold-store commit makes the fill the measured interaction, so the warm-path win (zero-fill repeat commits, test-pinned) is invisible to the settle metric. Details appended to the results doc. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keeps compiled predicates and slot threading. See docs/superpowers/specs/2026-08-24-columnar-verdicts-results.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
blove
enabled auto-merge (squash)
August 25, 2026 17:43
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
api-extractor's declaration-reference parser rejects the ɵ start; the api:check gate treats the warning as an error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Vercel preview readyPreview: https://pretable-1g2a2ep8b-cacheplane.vercel.app Updated automatically by the |
This was referenced Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Filter-only
setQueryon ungrouped data now completes synchronously end to end, on a row-model core rewritten around dense integer handles. Three arcs, every step measured against same-run TanStack controls:The filter fast path (F/G/H cycles). A filter-only change is classified and handled as a membership diff over an already-sorted set: verdict diff, O(flipped) updates, a linear merge of survivors with sorted flip-ins, and a bulk tree build (
orderIsProven+ derived byId). No record is rebuilt — membership IS the verdict (filter-membership.ts). The renderer permutes row heights instead of re-ingesting them (journal reason"refilter"), and a latent blank-viewport-during-replacement bug found along the way is fixed.The dense-handle core (M0–M2). Every row gets a lifetime slot; revisions carry
recordsBySlot(chunked COW slot vector) andvisibleSlots(membership bitset) alongside the persistent structures, so the rebuild's O(n) passes are array-resident instead of paying string-hash + HAMT traversal per row. Old snapshots stay valid under slot reuse because each revision owns its chunk table (mutation-pinned).The dense layout seam (Amendment I).
RowHeightIndexgains a dense lane: slot-keyed refilter/reorder, bitset membership, chunked source walks, pooled row refs. Measurements and tombstones stay string-identity-keyed — slots are lifetime-bound and reused; retention outlives the row (mutation-pinned both directions). Two one-generation string-lane escape hatches keep hostile inputs from livelocking.A columnar verdict cache (Amendment J) was also built, measured flat twice, and reverted per the repo's no-unmeasured-budget standard — the compiled filter predicates and slot-threaded inputs from that arc are kept (strictly simpler per-row shape). The full record is in
docs/superpowers/specs/2026-08-24-columnar-verdicts-results.md.Numbers (50k S2, medians of 3, TanStack same-run controls in band throughout)
TanStack same-run reference: ~50–58ms settle. Traced share of the remaining window: rebuild body ~19%, verdict pass ~17%, HAMT ~12%, render/commit ~15%, layout ~7% — no dominant lever remains.
The trade, stated plainly
The synchronous path blocks the main thread ~90ms at 50k where the old cooperative path blocked 0 (while settling ~120ms sooner and never blanking). At 3k the block is gone entirely (0 long tasks). The already-merged sort path blocks longer (~250ms) than this filter path does. A size gate (sync below ~10k, cooperative above) remains a follow-up option if keystroke-latency-at-scale becomes the priority; the cooperative path is retained in full as the fallback and the grouped path.
API surface
Six lines total: three optional
@internal ɵmembers onPretableRowModelSnapshot(the renderer seam), mirrored incore.api.md/react.api.md. Everything else is internal. The change-journal reset-reason union gains"refilter".Follow-ups (filed as issues after merge)
rebuildRowStoreForQuery(row-store.ts, zero callers).Verification
Full designs, plans, and measured results for every arc live in
docs/superpowers/{specs,plans}/on this branch. All work assertions are mutation-hardened (reviewers re-ran the mutations). Gates at HEAD after rebase onto693f01ed: build, api (zero drift), typecheck, lint, fullpnpm test.🤖 Generated with Claude Code