JSONL inspection validation policy: per-line json.loads costs 5.5x at 100 MB

[valiantone]

Summary

Decide the JSONL inspection validation policy: _stream currently pays json.loads on every line, which dominates inspection cost.

Measured impact (spike #48, PR #84, B2)

At 100 MB: full per-line validation = 1,740 ms median; scan-only = 319 ms (~5.5x). Validation is the cost, not scanning. A C scanner clears 3.55x, but the spike's conclusion (#48) is no native helper yet — the pure-Python policy change captures most of the win.

Options (semantics decision needed)

1. Validate only the sampled lines (first sample_size) — cheap, weaker guarantees.
2. Make full validation opt-in via an inspect parameter / API flag — default changes semantics.
3. Keep full validation for small files, cap by bytes for large files.

Acceptance criteria

• Policy decided and documented (affects FileInspection.unsupported_reason / row_count guarantees and API/MCP/CLI consumers).
• Inspector behavior change covered by tests; default path performance measured against bench/native_spike/results.json baseline.

Context

Spike recommendation follow-up #4: bench/native_spike/README.md. Related: #48, #53 (closed — original inspector ticket).

[valiantone]

Atlas alignment context — 2026-08-28

This is the one follow-up that requires an explicit inspection-policy decision before default behavior changes.

Required boundary: distinguish advisory inspection from authoritative import, hydration, snapshot, and provenance verification. A faster scan or sampled validation may be appropriate only when the API/result clearly declares its reduced assurance. It must not be represented as full JSON validity, trusted row counts, or verification sufficient for a canonical memory import/hydration decision.

Decision gate: before changing the default, record which modes are authoritative versus advisory; define the returned assurance/unsupported-reason semantics; preserve a full-validation path; and ensure API, MCP, CLI, and callers cannot silently mistake sampled inspection for canonical verification. Tests and benchmark results should demonstrate both semantic clarity and the measured tradeoff.

Product alignment: HotMem remains the public local-first data/provenance foundation. Memoria v0.2 may use its JSONL operational memory; EMOS owns policy, governed execution, and promotion. This ticket does not authorize a global graph, enterprise control plane, automatic feedback, or an unverified latency claim.

Source: Atlas bounded-loop synthesis and HotMem native-helper spike #48.