bench: streaming-read A/B harness for the 3c go/no-go (backlog#1144)

[houseme]

What

Adds the streaming-read A/B harness for the 3c go/no-go (rustfs/backlog#1144): should ecstore's sequential streaming reads (open_read_stream/open_full_read) be routed through io_uring, or keep delegating to StdBackend's buffered path that rides kernel readahead?

• examples/streaming_bench.rs — reads one file end-to-end with a chosen strategy, prints a CSV row (throughput is the decision metric). One measurement per invocation so the runner controls page-cache state.
• bench-streaming.sh — sweeps strategy × size × chunk × queue-depth in two cache legs (warm: pre-read; cold: drop_caches before each timed run), pre-creating files so cold runs are genuinely cold.

Why pipelined io_uring

io_uring's only lever on a sequential stream is depth (keeping N reads in flight to hide per-op latency). A single-depth io_uring read is expected to lose to buffered+readahead, so the strategies take a queue depth and the fair comparison is deep-pipelined io_uring vs buffered+readahead. O_DIRECT variants additionally avoid page-cache pollution.

Scope

Library code unchanged — the example is additive and does not touch the crate API. This PR is the measurement harness; the go/no-go decision and results table land on backlog#1144 (and, if Go, a separate implementation PR).

Verification

• clippy --example streaming_bench -- -D warnings clean.
• All four strategies run end-to-end under real io_uring (byte counts match the file size).
• Full sweep runs on the bench host (real kernel + dedicated disk); results posted to backlog#1144.

[houseme]

Review comments — please address these before merging.

I found three issues in the additive benchmark harness:

1. ensure_file silently truncates an existing path whose size differs, and metadata/File::create follow symlinks. The script is documented to run as root, so a wrong BENCH_DIR or direct CLI path can overwrite unrelated data. Refuse existing files, or create only inside a dedicated directory with no-follow semantics.

2. The CLI accepts invalid parameters without validation. chunk=0 reaches step_by(0), align=0 reaches next_multiple_of(0), and oversized queue-depth arithmetic can overflow or panic. Validate positive, power-of-two/alignment-compatible values and use checked arithmetic.

3. The benchmark verifies only the total byte count. A strategy can return the wrong contents or wrong offsets and still pass assert_eq!(total, size). Add an optional checksum/sample verification mode and cover partial-size/O_DIRECT boundary cases outside the timed measurement.

CI passing is useful, but it does not execute the sweep script or validate these runtime cases.

[houseme]

All three addressed in f6911ec. Each fix was exercised on the bench host (Linux 6.17, real io_uring, NVMe), not just type-checked.

1. ensure_file could truncate / follow a symlink

It no longer writes to an existing path at all. symlink_metadata (which does not resolve a symlink) classifies the path, and creation uses create_new — O_CREAT|O_EXCL, so it fails on a symlink — plus O_NOFOLLOW|O_CLOEXEC. An existing wrong-size or non-regular path is a hard error, not an overwrite. Read opens also carry O_NOFOLLOW, closing the stat→open window. A partially written file is removed rather than left to look like a "wrong size" on the next run.

Observed:

$ head -c 4096 /dev/urandom > victim.bin
$ streaming_bench std_buffered victim.bin 1048576 131072 1 4096
streaming_bench: victim.bin exists with 4096 bytes but 1048576 were requested;
refusing to truncate it — delete it or pick another path        # exit=2, victim still 4096 bytes

$ ln -s precious.txt link.bin && streaming_bench std_buffered link.bin 1048576 131072 1 4096
streaming_bench: link.bin exists and is not a regular file (is_symlink: true); refusing to touch it

A directory at the target path is refused the same way, and precious.txt is untouched.

2. Unvalidated CLI geometry

validate() now runs before any I/O: size ∈ 1..=i64::MAX, chunk ∈ 1..=1 GiB, qd ∈ 1..=4096 (the cap keeps (qd*2).next_power_of_two() inside the kernel's 32768-entry limit and far from overflow), align a power of two in 512..=1 MiB, and — for the O_DIRECT strategies — chunk % align == 0 so every read offset is block-aligned. The strategy is parsed into an enum rather than matched as a string. Failures exit 2 with a message and create no file.

chunk_bytes must be in 1..=1073741824, got 0
align must be a power of two in 512..=1048576, got 0
qd must be in 1..=4096, got 99999999999999
O_DIRECT strategies need chunk_bytes (131073) to be a multiple of align (4096), ...

3. Length-only assertion admitted wrong contents/offsets

You are right that assert_eq!(total, size) is vacuous. The file is now filled with an offset-addressable pattern — byte[i] = splitmix64(i / 8)[i % 8] — so any chunk can be checked against its absolute offset without reading the ones before it. BENCH_VERIFY=1 checks every byte a strategy delivers at its claimed offset, and rejects a short read; the io_uring paths verify inside the read task.

bench-streaming.sh runs an untimed preflight before the first measurement: all four strategies × queue depths 1 and 4 × two boundary geometries — 1 MiB + 4096 (block multiple, not a chunk multiple: partial tail) and 1 MiB + 4097 (O_DIRECT tail landing inside a partly valid block). A mismatch aborts the sweep. Verification is compiled out of the decision path in the sense that matters: the flag is off for every timed run, so it cannot perturb throughput. A BENCH_VERIFY=1 invocation is a correctness check, and its CSV row is documented as not a measurement.

That the check has teeth (flipping one byte at offset 700000, then re-running each strategy):

rc=1 std_buffered         :: content mismatch at byte 700000: got 0x00, want 0x1e
rc=1 std_odirect          :: content mismatch at byte 700000: got 0x00, want 0x1e
rc=1 uring_read_at        :: read: content mismatch at byte 700000: got 0x00, want 0x1e
rc=1 uring_read_at_direct :: read: content mismatch at byte 700000: got 0x00, want 0x1e

Without BENCH_VERIFY, that same corrupted file still produces a clean, fast CSV row — which is exactly the vacuous pass you described.

Verification. On the bench host (Linux 6.17, dedicated NVMe): cargo fmt --check and cargo clippy --example streaming_bench -- -D warnings clean; every negative case above executed with the shown exit codes and messages; bench-streaming.sh run end-to-end (preflight + warm and cold legs, drop_caches as root) — preflight reports all strategies verified byte-exact and the sweep emits its CSV.