perf: v0.2.5 spike follow-ups — hydrate verify, streaming hash, JSONL policy, embed batching, vector polish

CHANGELOG.md

@@ -4,6 +4,58 @@ All notable changes to HotMem will be documented in this file.
 
 Format follows [Keep a Changelog](https://keepachangelog.com/).
 
+## [Unreleased]
+
+### Changed — JSONL inspection validation policy (#89)
+- Inspection is **advisory** and now declares its assurance level:
+  `FileInspection.metadata["validation"]` is `sampled` (default — only the
+  declared sample window is parsed) or `full`. A malformed line beyond the
+  sampled window is reported only under `validation="full"`; `row_count`
+  semantics are unchanged. Full validation remains available via
+  `inspect_file(..., validation="full")` and `hotmem inspect
+  --full-validation`. Measured: 34 ms vs 115 ms per 11.6 MB file
+  (~5.5x at 100 MB per the spike baseline). Authoritative verification of
+  canonical content is unaffected — provenance checksums, not inspection.
+
+### Changed — streaming verification for large ranges (#88)
+- `provenance.verify_range` hashes ranges above `STREAM_VERIFY_THRESHOLD`
+  (8 MiB) while streaming through the new optional adapter capability
+  `LocalFilesystemAdapter.read_range_chunked` — O(chunk) memory instead of
+  O(range), identical digest and `ProvenanceError` semantics. Adapters
+  without the capability and ranges at/below the threshold keep the simple
+  single-read path.
+
+### Changed — single-read verified hydration (#87)
+- Verified hydration now hashes the bytes it already read instead of
+  re-reading the range through `provenance.verify_range` — one read per
+  range (spike B1: ~+16% at 100 MB). Digest and `ProvenanceError`
+  semantics are unchanged; new internal `provenance.verify_bytes` helper
+  carries the identical checksum contract.
+
+### Changed — embed_text trigram hashing (#90)
+- Trigram hashing now runs through a bounded per-gram cache
+  (`lru_cache`, 65 536 entries) replacing per-call md5 + hex parsing;
+  vectors are **bit-identical** to the previous `hotmem-hash-v1` output
+  (golden equivalence test over ASCII/unicode/random corpora). Measured
+  ~5.9x faster per embedding on realistic text — directly attacks the
+  37–78% of `parse_bundle` time the spike attributed to `embed_text`.
+
+### Changed — derived vector index polish (#92)
+- `search_by_ids` binds candidate ids in chunks of 900, so any configured
+  `oversample` stays under SQLite's legacy 999-variable cap; chunk results
+  are merged into the identical canonical order.
+- The accelerated search path runs a single FTS pass — candidate unioning
+  and BM25 normalization share one query (also true of the fallback path).
+- Swallowed Chroma delete/clear failures are now logged at warn level; the
+  derived index reuses `embed.unpack_embedding` for the blob format.
+
+### Performance follow-ups from the native helper spike (#87–#92)
+
+Work in progress — see PR for the unified acceptance criteria covering:
+single-read verified hydration (#87), streaming range hash (#88), JSONL
+inspection validation policy (#89), `embed_text` trigram batching (#90), and
+derived-vector-index polish (#92).
+
 ## [0.2.4] - 2026-08-28
 
 ### Added — Optional derived vector index (#49)

docs/architecture.md

@@ -34,6 +34,14 @@ HotMem combines deterministic text embeddings, keyword overlap, and importance
 to rank local memories. Read-only inspectors provide lightweight metadata for
 CSV, JSONL, and Parquet files without turning the runtime into a query engine.
 
+Inspection is **advisory**: it never authorizes import, hydration, snapshot,
+or provenance decisions. JSONL validation is sampled by default (lines inside
+the declared sample window only) and the result declares its assurance level
+via `metadata.validation` (`sampled` | `full`); pass `--full-validation`
+(CLI) or `validation="full"` to parse every line. Authoritative verification
+of canonical memory content always happens through provenance checksums, not
+through inspection.
+
 An optional derived vector index can accelerate candidate retrieval. The index
 is disposable and rebuildable from SQLite, sits in front of the canonical
 hybrid ranker rather than replacing it, and search falls back to the

src/hotmem/cli.py

@@ -342,25 +342,39 @@ def openapi(output: str | None, fmt: str):
     type=int,
     help="Max sample rows to preview (CSV/JSONL).",
 )
+@click.option(
+    "--full-validation",
+    "full_validation",
+    is_flag=True,
+    help="JSONL: validate every line instead of only the sampled window. "
+    "Inspection is advisory; full validation costs ~5x on large files.",
+)
 @click.option(
     "--json",
     "as_json",
     is_flag=True,
     help="Emit raw JSON (bypasses the renderer, for scripting).",
 )
-def inspect(uri: str, count_rows: bool, sample_size: int, as_json: bool):
+def inspect(uri: str, count_rows: bool, sample_size: int, full_validation: bool, as_json: bool):
     """Inspect a local file's structure and provenance without ingesting it.
 
     Lightweight metadata-only inspection for CSV, JSONL, and Parquet files.
     Never copies file contents into the database — returns URI, size, checksum,
     columns, and an optional bounded sample. Unsupported formats and remote
-    schemes fail with a clear error.
+    schemes fail with a clear error. Inspection is advisory: JSONL validation
+    covers the sampled window unless --full-validation is passed, and the
+    result declares its assurance level.
     """
     from hotmem.inspectors import UnsupportedFormatError, inspect_file
     from hotmem.storage import UnsupportedSchemeError
 
     try:
-        inspection = inspect_file(uri, count_rows=count_rows, sample_size=sample_size)
+        inspection = inspect_file(
+            uri,
+            count_rows=count_rows,
+            sample_size=sample_size,
+            validation="full" if full_validation else "sampled",
+        )
     except UnsupportedFormatError as err:
         raise click.ClickException(str(err)) from err
     except UnsupportedSchemeError as err:
@@ -380,6 +394,7 @@ def inspect(uri: str, count_rows: bool, sample_size: int, as_json: bool):
         rows=data["row_count"],
         checksum=str(data["checksum"])[:12] + "…",
     )
+    click.echo(f"validation: {data['metadata'].get('validation', 'sampled')} (advisory)")
     if data["columns"]:
         click.echo(f"columns: {', '.join(data['columns'])}")
     if data["delimiter"]:

src/hotmem/db.py

@@ -34,6 +34,10 @@
 
 _trace = get_tracer("db")
 
+# Ids bound per query in search_by_ids: stays under SQLite's legacy
+# 999-variable limit even with the embedding parameter plus headroom (#92).
+_SEARCH_BIND_CHUNK = 900
+
 # Single source of truth for the memories table column order. Drives INSERT
 # statement generation and SQLite-to-SQLite import projection so the three
 # write paths cannot drift.
@@ -871,22 +875,39 @@ def search_by_ids(
         ``search_with_cosine``, so ids that expired, were archived, or were
         deleted since indexing are filtered out here. Used by the accelerated
         search path (#49) to re-score vector-index candidates in SQLite.
+
+        Ids are bound in chunks of ``_SEARCH_BIND_CHUNK`` so any configured
+        ``oversample`` stays under SQLite's legacy 999-variable cap; chunk
+        results are merged and re-sorted into the same canonical order.
         """
         if not memory_ids:
             return []
         archived_clause = "" if include_archived else " AND promotion_state != 'ARCHIVED'"
-        placeholders = ", ".join("?" for _ in memory_ids)
-        rows = self._conn.execute(
-            f"""SELECT id, identifier, fact_text, fact_summary, importance,
-                       metadata_json, source, created_at,
-                       cosine_sim(embedding, ?) AS cosine_score
-                FROM memories
-                WHERE id IN ({placeholders})
-                  AND {_ttl_live()}{archived_clause}
-                ORDER BY cosine_score DESC, id ASC""",
-            (query_embedding, *memory_ids),
-        ).fetchall()
-        return [dict(r) for r in rows]
+        results: list[dict[str, Any]] = []
+        for start in range(0, len(memory_ids), _SEARCH_BIND_CHUNK):
+            chunk = memory_ids[start : start + _SEARCH_BIND_CHUNK]
+            placeholders = ", ".join("?" for _ in chunk)
+            rows = self._conn.execute(
+                f"""SELECT id, identifier, fact_text, fact_summary, importance,
+                           metadata_json, source, created_at,
+                           cosine_sim(embedding, ?) AS cosine_score
+                    FROM memories
+                    WHERE id IN ({placeholders})
+                      AND {_ttl_live()}{archived_clause}""",
+                (query_embedding, *chunk),
+            ).fetchall()
+            results.extend(dict(r) for r in rows)
+        # Same total order as the per-chunk SQL ORDER BY: ids are unique, so
+        # (cosine DESC, id ASC) is deterministic. NULL cosine (no embedding)
+        # sorts last, matching SQLite's DESC default.
+        results.sort(
+            key=lambda r: (
+                r["cosine_score"] is None,
+                -(r["cosine_score"] or 0.0),
+                r["id"],
+            )
+        )
+        return results
 
     def all_rows(self, *, include_embedding: bool = False) -> list[dict[str, Any]]:
         """Return all memory rows as dicts (for snapshot export)."""

src/hotmem/embed.py

@@ -21,6 +21,7 @@
 import hashlib
 import math
 import struct
+from functools import lru_cache
 
 from hotmem.trace import Timer, get_tracer
 
@@ -30,6 +31,18 @@
 EMBEDDING_MODEL = "hotmem-hash-v1"
 
 
+# Bounded cache of trigram -> (bucket, sign). Text reuses a small vocabulary
+# of character trigrams heavily (bundle corpora share word pools), so the
+# cache removes nearly all md5 calls while keeping vectors bit-identical:
+# same gram bytes -> same digest -> same bucket/sign (#90).
+@lru_cache(maxsize=65536)
+def _gram_bucket_sign(gram: str) -> tuple[int, float]:
+    h = int.from_bytes(hashlib.md5(gram.encode(), usedforsecurity=False).digest(), "big")
+    bucket = h % EMBEDDING_DIM
+    sign = 1.0 if (h >> 64) % 2 == 0 else -1.0
+    return bucket, sign
+
+
 def embed_text(text: str) -> list[float]:
     """Produce a deterministic embedding vector from text.
 
@@ -42,11 +55,7 @@ def embed_text(text: str) -> list[float]:
 
         # Hash overlapping trigrams into embedding buckets
         for i in range(max(1, len(text_lower) - 2)):
-            gram = text_lower[i : i + 3]
-            h = int(hashlib.md5(gram.encode(), usedforsecurity=False).hexdigest(), 16)
-            bucket = h % EMBEDDING_DIM
-            # Use upper bits for sign/magnitude
-            sign = 1.0 if (h >> 64) % 2 == 0 else -1.0
+            bucket, sign = _gram_bucket_sign(text_lower[i : i + 3])
             vec[bucket] += sign
 
         # L2 normalize

src/hotmem/inspectors/__init__.py

@@ -7,7 +7,7 @@
     contents into SQLite.
 
 Interface:
-    inspect_file(uri, *, count_rows=False, sample_size=5) -> FileInspection
+    inspect_file(uri, *, count_rows=False, sample_size=5, validation="sampled") -> FileInspection
     get_inspector(uri) -> FileInspector
     UnsupportedFormatError — raised when no inspector handles a format
 
@@ -78,6 +78,7 @@ def inspect_file(
     *,
     count_rows: bool = False,
     sample_size: int = 5,
+    validation: str = "sampled",
 ) -> FileInspection:
     """Inspect a backing file and return provenance + light metadata.
 
@@ -86,7 +87,15 @@ def inspect_file(
     raised for remote schemes; ``UnsupportedFormatError`` for unknown
     formats; otherwise a FileInspection (which may carry
     ``unsupported_reason`` for a recognized-but-malformed file).
+
+    ``validation`` is advisory-assurance control for JSONL (#89):
+    "sampled" (default) parses only the sample window and declares the
+    level via ``FileInspection.metadata["validation"]``; "full" parses
+    every line. Inspection is advisory — it never authorizes import,
+    hydration, snapshot, or provenance decisions.
     """
     adapter, meta = resolve_adapter(uri)
     inspector = _inspector_for_format(meta["format"])
-    return inspector.inspect(uri, adapter, meta, count_rows=count_rows, sample_size=sample_size)
+    return inspector.inspect(
+        uri, adapter, meta, count_rows=count_rows, sample_size=sample_size, validation=validation
+    )

src/hotmem/inspectors/base.py

@@ -81,6 +81,7 @@ def inspect(
         *,
         count_rows: bool = False,
         sample_size: int = 5,
+        validation: str = "sampled",  # JSONL-only; accepted for uniform dispatch
     ) -> FileInspection: ...
 
 

src/hotmem/inspectors/csv_inspector.py

@@ -35,6 +35,7 @@ def inspect(
         *,
         count_rows: bool = False,
         sample_size: int = 5,
+        validation: str = "sampled",  # noqa: ARG002 — JSON validation is JSONL-only
     ) -> FileInspection:
         head = adapter.read_range(uri, 0, min(_SNIFF_BYTES, meta["size"] or 0))
         text = head.decode("utf-8", errors="replace")

src/hotmem/inspectors/jsonl_inspector.py

@@ -5,6 +5,14 @@
     - Validate sampled lines are JSON; report the first malformed line offset
       in ``unsupported_reason`` instead of crashing (provenance, not outage).
     - O(file size) byte read, O(1) memory via buffered newline scanning.
+
+Validation policy (#89):
+    Inspection is ADVISORY — it never authorizes import, hydration, snapshot,
+    or provenance decisions. Default validation is "sampled": only lines
+    inside the ``sample_size`` window are parsed, and the result declares its
+    assurance via ``metadata["validation"]``. ``validation="full"`` keeps the
+    pre-#89 behavior (every line parsed) for consumers that need it.
+
 """
 
 from __future__ import annotations
@@ -18,6 +26,7 @@
 from .base import FileInspection
 
 _READ_CHUNK = 1 << 20  # 1 MiB read window — constant memory regardless of file size.
+_VALIDATION_MODES = ("sampled", "full")
 
 
 class JSONLInspector:
@@ -31,10 +40,15 @@ def inspect(
         *,
         count_rows: bool = False,
         sample_size: int = 5,
+        validation: str = "sampled",
     ) -> FileInspection:
+        if validation not in _VALIDATION_MODES:
+            raise ValueError(
+                f"unknown validation mode {validation!r}; expected one of {_VALIDATION_MODES}"
+            )
         path = _resolve(uri)
         row_count, sample_rows, byte_ranges, unsupported_reason = _stream(
-            path, count_rows=count_rows, sample_size=sample_size
+            path, count_rows=count_rows, sample_size=sample_size, validation=validation
         )
 
         columns = _infer_columns(sample_rows)
@@ -51,7 +65,7 @@ def inspect(
             has_header=None,
             sample=sample_rows or None,
             byte_ranges=byte_ranges or None,
-            metadata={},
+            metadata={"validation": validation},
             unsupported_reason=unsupported_reason,
         )
 
@@ -67,16 +81,21 @@ def _stream(
     *,
     count_rows: bool,
     sample_size: int,
+    validation: str = "sampled",
 ) -> tuple[int | None, list[dict[str, Any]], list[tuple[int, int]], str | None]:
     """One streaming pass: count lines, collect a bounded sample, validate JSON.
 
     Counts newlines in fixed-size chunks (cheap, allocation-free) and samples
     the first ``sample_size`` complete records by capturing byte offsets.
+
+    ``validation="sampled"`` parses only lines inside the sample window
+    (declared advisory assurance, #89); ``"full"`` parses every line.
     """
     row_count = 0 if count_rows else None
     sample_rows: list[dict[str, Any]] = []
     byte_ranges: list[tuple[int, int]] = []
     unsupported_reason: str | None = None
+    full_validation = validation == "full"
 
     line_index = 0
     line_start = 0
@@ -91,7 +110,7 @@ def _stream(
                 if carry:
                     if count_rows and carry.strip():
                         row_count = (row_count or 0) + 1
-                    if unsupported_reason is None:
+                    if unsupported_reason is None and (full_validation or line_index < sample_size):
                         unsupported_reason = _validate(carry, line_index, line_start)
                     _handle_line(
                         carry,
@@ -114,7 +133,7 @@ def _stream(
                 line_len = len(complete)
                 if count_rows and complete.strip():
                     row_count = (row_count or 0) + 1
-                if unsupported_reason is None:
+                if unsupported_reason is None and (full_validation or line_index < sample_size):
                     unsupported_reason = _validate(complete, line_index, line_start)
                 _handle_line(
                     complete,

src/hotmem/inspectors/parquet_inspector.py

@@ -59,6 +59,7 @@ def inspect(
         *,
         count_rows: bool = False,  # noqa: ARG002 — num_rows comes from the footer
         sample_size: int = 0,  # noqa: ARG002 — no row sampling (metadata-only)
+        validation: str = "sampled",  # noqa: ARG002 — JSON validation is JSONL-only
     ) -> FileInspection:
         size = meta["size"]
         unsupported = self._validate_magic(adapter, uri, size)

src/hotmem/memory.py

@@ -39,7 +39,7 @@
     BackingFileMissingError,
     ChecksumMismatchError,
     ProvenanceError,
-    verify_range,
+    verify_bytes,
 )
 from hotmem.storage import get_adapter
 from hotmem.trace import Timer, get_tracer
@@ -267,8 +267,10 @@ def hydrate_memory_detailed(
             raise ProvenanceError("truncated", source_uri, expected=expected_checksum)
 
         # On-demand checksum verification (skipped if no checksum stored or verify=False).
+        # Single-read verification (#87): hash the bytes already read — the
+        # digest and error semantics are identical to verify_range's re-read.
         if expected_checksum and verify:
-            verify_range(adapter, resolved_uri, offset, length, expected_checksum)
+            verify_bytes(source_uri, data, expected_checksum)
             verified = True
 
     _trace.info(