perf: v0.2.5 spike follow-ups — hydrate verify, streaming hash, JSONL policy, embed batching, vector polish

[valiantone]

What

Performance and correctness follow-ups from the native helper spike (#48, benched in #84), batched as the v0.2.5 release. This PR is opened as a draft and filled in ticket by ticket; each ticket's scope is implemented, tested, and checked against the unified acceptance criteria below.

Scope — tickets closed by this PR

• Closes Single-read verified hydration: drop redundant range re-read #87 — Single-read verified hydration. memory.hydrate_memory_detailed reads the range, then provenance.verify_range re-reads it to hash it. Hash the already-read bytes (or add a streaming verify API accepting in-memory bytes); expected ~+16% on verified hydration of large ranges, plus one fewer range copy.
• Closes Streaming range hash for large verified ranges (constant memory) #88 — Streaming range hash for large verified ranges. Chunked seek+hash in the provenance/adapter layer above a documented size threshold; peak RSS becomes O(chunk) instead of O(range) (spike B1: 20 MB vs 219 MB at 100 MB, 1.26x faster).
• Closes JSONL inspection validation policy: per-line json.loads costs 5.5x at 100 MB #89 — JSONL inspection validation policy. Spike B2: per-line json.loads is 5.5x of scan cost at 100 MB. ⚠️ Decision point (semantics owner: Lightweight file inspectors for CSV, JSONL, and Parquet metadata #53 successor): (a) validate only sampled lines, (b) make full validation opt-in via flag, or (c) full validation for small files / capped for large. The chosen policy gets documented in docs/api.md and covered by tests.
• Closes Batch embed_text trigram hashing (37-78% of parse_bundle time) #90 — Batch embed_text trigram hashing. 37–78% of parse_bundle time is per-trigram hashlib.md5 calls (embed.py:44). Batch into one hash update over a packed buffer or a cheaper rolling hash. Constraint: hotmem-hash-v1 vectors stay bit-identical, or the embedding model string is versioned with a migration note.
• Closes Vector index polish: chunked search_by_ids binds, single FTS pass, shared unpack_embedding #92 — Derived vector index polish. Chunk search_by_ids binds (safe on SQLite builds with the 999-variable limit, whatever oversample is configured); single FTS pass per accelerated search (candidate union + BM25 share one fts_search); reuse embed.unpack_embedding instead of _unpack_blob; log swallowed Chroma delete/clear errors at warn level instead of blanket-silencing.
• (Branch also carries the changelog scaffolding for the release.)

Unified acceptance criteria (holistic, across all tickets)

Correctness — the contracts that must not move:

1. Result parity everywhere. /v1/search responses remain byte-identical with and without the vector index (golden additive test stays green); verified hydration produces digests identical to the current double-read path on every range size; hybrid ranking order is unchanged for stores within the exactness window.
2. No API/behavior change except the explicit JSONL inspection validation policy: per-line json.loads costs 5.5x at 100 MB #89 policy decision. Any change to FileInspection guarantees is deliberate, documented in docs/api.md, and versioned in the CHANGELOG. include_archived/TTL/archived predicates behave identically on every new code path.
3. Dependency-free default preserved. All improvements are pure Python stdlib; chromadb stays optional ([vector] extra); zero new required dependencies or wheels.

Performance — measured against the committed spike baselines (bench/native_spike/results.json, B1/B2/B3):

4. Hydrate/verify speedup demonstrated: single-read (+Streaming range hash for large verified ranges (constant memory) #88 streaming above threshold) ≥ the spike's ~1.16–1.26x at 100 MB on the benchmark fixtures; no regression at small range sizes.
5. Constant memory for large ranges: verified hydration of a 100 MB range peaks at O(chunk) RSS (spike target ≈ 20 MB), not O(range).
6. Inspection cost reduction: under the chosen JSONL inspection validation policy: per-line json.loads costs 5.5x at 100 MB #89 policy, 100 MB JSONL inspection improves materially toward the spike's scan-only baseline (319 ms vs 1740 ms), without weakening guarantees beyond the documented policy.
7. Bundle parse speedup: embed_text batching yields a measurable parse_bundle reduction on the spike bundle trees (fixtures reusable), with hotmem-hash-v1 vectors bit-identical (or model string versioned + migration note).

Robustness:

8. search_by_ids is parameter-limit safe for any configured oversample (chunked binds), with a regression test covering >999 ids.
9. Staleness/fallback semantics unchanged: index staleness, graceful fallback, rebuild-marker safety (fingerprint-before-read) all keep their existing tests green; swallowed backend errors become visible warnings.

Quality gates (every commit):

10. Full suite green on Python 3.11–3.14 plus the vector CI job; ruff check + ruff format --check clean on src/ tests/ examples/.
11. Each ticket's own acceptance criteria (in Single-read verified hydration: drop redundant range re-read #87–Vector index polish: chunked search_by_ids binds, single FTS pass, shared unpack_embedding #92) individually satisfied and cross-referenced from its closing commit message.
12. CHANGELOG updated under [Unreleased] per ticket; docs (api.md, architecture.md) updated wherever user-visible behavior or policy changes.

Release

• Milestone v0.2.5; version bump to 0.2.5 happens as its own commit at release time (auto-tag cuts v0.2.5 on merge to main), not incidentally in feature commits.

References

• Spike: Native helper spike: Rust, C, or WebAssembly fast-path primitives #48 · Bench PR: bench: native helper spike — C/WASM vs Python fast-path primitives (#48) #84 · Vector index: Optional derived vector index #49/feat: optional derived vector index (#49) #85 · Inspectors: Lightweight file inspectors for CSV, JSONL, and Parquet metadata #53

---

Implementation status — all five tickets landed

Ticket	Commit	Delivered	Tests (all proven intent-purposed)
#92	864d655	900-bind chunking w/ canonical merge order; single FTS pass on both paths; shared unpack_embedding; warn-logged Chroma failures	>999-id parity vs full scan; FTS call-count == 1 on accelerated + fallback
#90	fe7602e	bounded per-gram lru_cache (65 536) + int.from_bytes digest conversion; hotmem-hash-v1 bit-identical	golden equivalence vs verbatim pre-#90 algorithm (ASCII/unicode/random, 159 cases); cache-engagement assertion
#87	c053082	provenance.verify_bytes; hydrate hashes already-read bytes — one read per range	SpyAdapter read-once assertion (proven failing on double-read code); error-surface parity via existing mismatch/truncation/missing tests
#88	ebd8ce2	read_range_chunked (optional local-adapter capability, no protocol change); streaming verify > 8 MiB threshold	digest parity + path engagement across the threshold boundary (exactly 8 MiB = simple path); truncated semantics preserved
#89	9b2ded7	advisory policy: sampled-by-default, assurance declared via metadata["validation"], opt-in validation="full" (+ CLI flag); row_count untouched	window-line reported in both modes; beyond-window only under full; assurance declaration; mode validation

Measured (this machine, min-of-N):

• embed_text 9.7 KB: 6.26 ms → 1.07 ms (5.9x, cold cache) — attacks the spike's 37–78% of parse_bundle.
• JSONL inspection 11.6 MB: full 115 ms → sampled 34 ms (3.4x; spike baseline ~5.5x at 100 MB where validation dominates).
• Verified hydration: one read_range per range (was two) — spike B1: ~+16% at 100 MB.
• Streaming verify: O(chunk) RSS for ranges > 8 MiB (spike B1: 20 MB vs 219 MB at 100 MB).

Decision record (#89, ratified by @valiantone 2026-08-28): inspection is advisory and never authorizes import/hydration/snapshot/provenance decisions; default validation="sampled" with declared assurance in metadata["validation"]; full validation preserved as opt-in. Documented in docs/architecture.md.

Atlas alignment: #87/#88 preserve digest + ProvenanceError semantics exactly (no new public semantics); #90 keeps hotmem-hash-v1 bit-identical (no model-version bump needed — the golden test is the compatibility proof); #92 keeps the index optional/rebuildable with zero result-semantics change; #89 implements the advisory-vs-authoritative boundary verbatim.