Summary
Add a streaming (chunked seek+read+hash) path for verifying/ranging large byte ranges, switching on range size.
Measured impact (spike #48, PR #84, B1)
The py_streaming arm (1 MiB chunked hashlib updates, no whole-range bytes) is the best checksum arm overall: 1.26x faster than the current path at 100 MB (673 ms median) with peak RSS 20 MB vs 219 MB — memory is flat in range size instead of ~1–2x the range.
Proposal
In provenance.verify_range / the adapter layer, use a streaming hash above a size threshold instead of materializing the full range. Pure Python, zero new dependencies.
Acceptance criteria
- Large-range verification holds O(chunk) memory.
- Digest parity with existing path on all range sizes (spike B1 parity harness reusable).
- Threshold documented; small ranges keep the simple path.
Context
Spike recommendation follow-up #2: bench/native_spike/README.md. Related: #48, and the single-read ticket (follow-up #1).
Summary
Add a streaming (chunked seek+read+hash) path for verifying/ranging large byte ranges, switching on range size.
Measured impact (spike #48, PR #84, B1)
The
py_streamingarm (1 MiB chunkedhashlibupdates, no whole-rangebytes) is the best checksum arm overall: 1.26x faster than the current path at 100 MB (673 ms median) with peak RSS 20 MB vs 219 MB — memory is flat in range size instead of ~1–2x the range.Proposal
In
provenance.verify_range/ the adapter layer, use a streaming hash above a size threshold instead of materializing the full range. Pure Python, zero new dependencies.Acceptance criteria
Context
Spike recommendation follow-up #2:
bench/native_spike/README.md. Related: #48, and the single-read ticket (follow-up #1).