bench: native helper spike — C/WASM vs Python fast-path primitives (#48) - #84
Merged
Conversation
…ion #48 Benchmark C (self-contained SHA-256 checksum, JSONL scanner) and WASM (wasmtime line scanner) candidates against the real Python hot paths (verify_range double-read, JSONLInspector._stream, parse_bundle) on a deterministic 384MB corpus. All parity checks pass; recommendation: no native helper yet — pure-Python fixes deliver more (single-read verify, streaming hash, validation policy, embed batching). Also documents the _stream chunk-boundary line-offset bug discovered by parity testing. Bench-only; no production code changes.
This was referenced Aug 28, 2026
valiantone
added
area:benchmarks
Contributor
Local review + follow-up ticketsReviewed pre-merge: additive-only (bench/native_spike + .gitignore), all parity checks in Follow-ups filed for the README recommendations and review findings:
All on milestone M5: Performance + Inspectors. Verdict: safe to merge. |
valiantone
approved these changes
Aug 28, 2026
valiantone
added a commit
that referenced
this pull request
Aug 28, 2026
_stream computed line_start = offset + pos in (carry+chunk) coordinates, overstating file offsets by the carry length for every line after a chunk-spanning line. Affected unsupported_reason offsets and byte_ranges (feeding FileInspection -> API/MCP/CLI). Offsets now use base + pos with base = offset - len(carry), verified for exact parity against the spike's C scanner (bench/native_spike). Adds a chunk-boundary regression test; discovered by the native helper spike (#48, #84).
valiantone
added a commit
that referenced
this pull request
Aug 28, 2026
* feat: optional derived vector index (#49) - VectorIndex protocol + NullVectorIndex default (zero vector deps) - Optional ChromaVectorIndex backend (lazy import, hotmem[vector] extra) - Index supplies oversampled cosine candidates; SQLite re-scores with the identical hybrid formula, preserving the /v1/search ranking contract - Rebuild reads SQLite only (embeddings reused, never recomputed, no file I/O); rebuild marker records the store fingerprint for staleness - Admin endpoints: POST /v1/vector-index/rebuild (emits index.rebuilt), GET /v1/vector-index/status, DELETE /v1/vector-index - hotmem serve --vector-index {none,chroma}; CI vector job; docs + changelog * fix: snapshot vector-index rebuild fingerprint before reading rows (#49) The rebuild marker now records the store fingerprint taken BEFORE the all_rows() read. A mutation landing mid-rebuild (concurrent insert) thus changes the live fingerprint and the marker reads stale, so search takes the deterministic fallback until the next rebuild. Snapshotting after the read could bless an index that silently misses rows inserted during the rebuild. Adds a regression test simulating the mid-rebuild insert. * release: v0.2.4 — optional derived vector index (#49) * chore: refresh uv.lock for v0.2.4 version bump * fix: correct JSONL _stream line offsets for chunk-spanning lines (#86) _stream computed line_start = offset + pos in (carry+chunk) coordinates, overstating file offsets by the carry length for every line after a chunk-spanning line. Affected unsupported_reason offsets and byte_ranges (feeding FileInspection -> API/MCP/CLI). Offsets now use base + pos with base = offset - len(carry), verified for exact parity against the spike's C scanner (bench/native_spike). Adds a chunk-boundary regression test; discovered by the native helper spike (#48, #84). * chore(bench): spike review nits — corpus byte accounting, report rows, formatting (#91) - gen_corpus.py: malformed-line accounting used a hardcoded 38 instead of the actual 41 bytes (40 + newline); counter now derives from the line. The insertion threshold is unaffected (drift began after insertion), so the committed manifest stays valid for regenerated corpora. - run_bench.py: report rows column now takes the first arm with a measured row count instead of blindly the first arm; reformatted. --------- Co-authored-by: valiantone <zrjohn@yahoo.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Benchmark spike comparing C and WebAssembly native-helper candidates against the real Python hot paths for HotMem's file-native primitives, with a data-backed recommendation: no native helper yet — pure-Python fixes deliver more.
Closes #48 (spike deliverables: comparison, recommendation, packaging analysis).
Why
Issue #48 asks which native helper surface (Rust/C/WASM), if any, is worth implementing first, gated on ≥3x speedup on the 100MB+ checksum path + clean packaging + graceful fallback. The spike measures real HotMem paths —
provenance.verify_range(the verified-hydration double-read),JSONLInspector._stream/inspect(),bundle.parse_bundle, and whole-file manifest checksums — rather than synthetic throughput.Changes
All new files under
bench/native_spike/(17 files, ~7k lines; zero production code changes):gen_corpus.py(deterministic 384MB corpus, seed 48 — swap-shaped JSONL with a manifest-committed malformed line, bundle trees, binary files to 100MB),bench_worker.py(subprocess-isolated cells),run_bench.py(orchestrator with hard parity assertions + markdown report),benchlib.py(graceful-unavailable loaders,/proc-based RSS, fadvise cold cache, memory guards)c_checksum/(self-contained SHA-256; pread + mmap range-hash via ctypes),c_jsonl/(line scanner + strict RFC 8259 validator),wasm_parser/scanner.wat(wasmtime scanner, host-orchestrated I/O)results.json(committed full run, 5 runs/arm, warm+cold),README.md(methodology, tables, packaging matrix, recommendation, follow-ups),manifest.json(corpus fingerprints for reproducibility).gitignore: corpus + build artifactsRust was scoped to packaging analysis only (no toolchain on the bench box; C + WASM cover both candidate primitives).
Results (100MB arms, median warm)
embed_text= 37–78% of parse time; algorithmic fix, not nativeAll parity checks pass (digests, row counts, first-bad-line, sample boundaries, manifest sets), and the optional-boundary check proves graceful degradation with
HOTMEM_SPIKE_DISABLE_NATIVE=1.Bonus finding: parity testing exposed a real bug —
_streamoverstates line offsets bylen(carry)when lines span read-chunk boundaries (src/hotmem/inspectors/jsonl_inspector.py:129), demonstrated on corpus data (+330 bytes on the malformed-line offset). Documented with the one-line fix; out of spike scope.Recommended follow-ups (pure Python, zero new deps)
_streamoffset bug (correctness)json.loadsis 5.5x of scan cost)embed_textbatching (bundle-load bottleneck)Verification
ruff check bench/cleansrc/,tests/,pyproject.tomlunmodified;wasmtimeis a bench-only venv depcd bench/native_spike && python gen_corpus.py --profile reduced && python run_bench.py