diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..f9516d8 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,153 @@ + + +# Changelog + +All notable changes to `rustfs-uring` are documented here. + +The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project +aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +The crate is **not published to crates.io** (`publish = false`); consumers pin a git revision. +Because of that, entries are grouped under `[Unreleased]` until the first published release, +and each one names the commit that landed it so a pin can be matched to a feature. + +## [Unreleased] + +Everything below is in `main` and reachable by pinning the listed revision. + +### Added + +- **Sharded rings** — `UringDriver::probe_and_start_sharded(entries, shards)` runs `shards` + independent rings, each with its own driver thread, pending table, backpressure semaphore, + and eventfd. `probe_and_start(entries)` is unchanged and equals `..._sharded(entries, 1)`. + ([#6], `719b245`) + + A buffered read that hits the page cache completes *inline* inside `io_uring_enter`, so the + thread driving a ring performs that read's `memcpy`. A single-ring driver is therefore capped + at one core's memory bandwidth. Measured on a 16-core host: one thread pinned at 100% CPU, + throughput flat at ~5 GB/s regardless of read size, while a blocking-pool baseline reached + 50 GB/s. Independent rings scaled near-linearly (1/2/4 rings → 4890/8969/15806 MB/s), which is + what identified the driver thread as the ceiling. With 8 shards, 1 MiB reads went from + 4911 MB/s to 47361 MB/s, and 64 KiB reads at concurrency 32 from 124k to 345k IOPS — while + *keeping* io_uring's tail-latency advantage rather than trading it away. + + In-flight ops are capped at `entries` **per shard**, so the invariant that makes CQ overflow + structurally unreachable holds per ring; the driver admits up to `shards * entries` + concurrent reads. Every cancel-safety invariant holds per shard: a `ReadHandle` carries the + `tx`/`wake` of the shard that accepted it, so a cancel or a deferred submission necessarily + routes back to the ring whose pending table holds the op. + +- **Native aligned `O_DIRECT` positioned read** — `UringDriver::read_at_direct(file, offset, len, align)`. + The caller opens the fd with `O_DIRECT` and passes the device's logical block size; `offset` + and `len` need **not** be aligned. The driver reads a block-aligned superset range into a + block-aligned buffer and hands back exactly `[offset, offset + len)`. ([#3], `f577ba0`) + + Alignment padding, the bytes preceding the range, and the block-aligned tail never escape — a + `BitrotReader` expecting an exact shard length would flag padded output as corruption. The + buffer remains a `Vec`, over-allocated by `align - 1` with the read region starting at the + first aligned byte inside the allocation, so the cancel-safety ownership model is untouched. + Buffered reads are the degenerate `align == 1` case of the same geometry. + +- **Benchmark harnesses** (`examples/` + runner scripts), both hardened so they cannot pass + vacuously: files are created only when absent (`create_new` + `O_NOFOLLOW`, never truncating a + caller-supplied path even though the sweeps run as root), content is offset-addressable, and + `BENCH_VERIFY=1` checks every delivered byte at its claimed offset. + + - `streaming_bench` + `bench-streaming.sh` — sequential whole-file read across + `std_buffered` / `std_odirect` / pipelined `uring_read_at` / `uring_read_at_direct`, in warm + and cold page-cache legs. ([#4], `4299b92`) + - `concurrent_pread_bench` + `bench-concurrent-pread.sh` — many concurrent positioned reads on + one disk, the shape erasure-coded shard reads actually serve. Four strategies isolate the + cost of the per-read `open` and of the `spawn_blocking` hop. ([#5], `d03977a`) + +### Changed + +- **Async backpressure** — the driver's synchronous backpressure gate became a tokio + `Semaphore`. `submit` takes a permit with `try_acquire_owned()` on the fast path and, when + saturated, hands the acquire future to the `ReadHandle` to await on its first poll instead of + parking the calling runtime worker. ([#2], `2b7cae4`) + + The permit is stored in the pending entry, so it is released exactly when that entry is + dropped at the final CQE — never at future drop. Public API is unchanged: `read_at`'s + signature and its "in flight the moment it returns" behaviour hold on the fast path. + +- **eventfd-driven reaping** — the 200 µs busy-poll loop was replaced by a `poll(2)` on two + eventfds: one registered with the ring so the kernel signals every CQE, one signalled by + `submit`/shutdown so a new message wakes the loop immediately. A bounded heartbeat still runs + a loop turn so the drain deadline is checked and the NODROP overflow list is flushed. + ([#1], `ea6b920`) + +### Fixed + +- `aligned_geometry` rejects an `align` above the kernel read cap, foreclosing the + `align_offset == usize::MAX` path that would have made the driver's pointer arithmetic + undefined, and the `region_len + align - 1` allocation overflow. The driver's aligned-buffer + setup replaced a release-mode-only `debug_assert!` with a checked allocation size and a + runtime guard: an unsatisfiable alignment now fails the read instead of doing UB pointer + arithmetic. ([#3], `f577ba0`) +- `deliver()` skips the full-buffer `copy_within` on the buffered path (`align == 1`, where the + logical range already starts at byte 0), so a buffered read no longer pays a per-read + `memmove`. ([#3], `f577ba0`) +- The fast-path `submit` now surfaces an explicit driver-gone error through the result channel + when the driver has exited, instead of letting the caller infer one from a dropped oneshot. + ([#2], `2b7cae4`) + +## Pre-history + +Before this repository existed, the code was the Spike 0 cancel-safety prototype in +`rustfs/rustfs` under `experiments/`. It was audited line by line +([rustfs/backlog#1051]) and every confirmed finding was remediated there. The library was then +promoted into this repository with its history preserved (`60993c9`), given a real-io_uring CI +leg that the main workspace cannot run (`d5b28fc`), and had its crate metadata and README +polished (`39018c0`). + +The cancel-safety invariants that survived that audit — the driver's pending table owning the +buffer and fd from SQE submission until the CQE, drop-abandons-result-only, bounded shutdown +drain, abort-before-free on a driver panic — are the ones every change above is measured +against. See [`docs/DESIGN.md`](docs/DESIGN.md). + +## Decisions recorded, not implemented + +These were on the roadmap and were **closed by measurement or by design conflict** rather than +built. They are listed so nobody re-opens them without new evidence. + +- **Three read shapes / streaming reads through io_uring** — **NO-GO** ([rustfs/backlog#1144]). + io_uring's lever on a *sequential* stream is pipelining; kernel readahead already has it. Cold + reads are device-bound, so io_uring at best ties; on a warm page cache it *loses badly* + (11–41% of a buffered read) because a single sequential stream exploits neither batched + submission nor the absence of a blocking thread. Streaming reads stay on the std backend. +- **`AsyncFd` reaping without a driver thread** — **not done**. `Drop` cannot `await`, so the + bounded shutdown drain would have to break the public API. The dedicated driver thread plus + eventfd reaping already removed the busy-poll, which was the actual cost. +- **Process-wide singleton ring** — **redefined**. A singleton conflicts with per-disk + isolation: a stalled disk must not starve another disk's ring. Rings stay per-disk, and + sharding ([#6]) scales them within a disk instead. +- **Registered buffers (`register_buffers`)** — **deprioritized**. It conflicts with the + `Vec` ownership model the cancel-safety proof rests on, and end-to-end profiling showed + the S3 GET bottleneck is userspace copies, not the disk read + ([rustfs/backlog#1159]). See "When this crate helps" in the README. + +[#1]: https://github.com/rustfs/uring/pull/1 +[#2]: https://github.com/rustfs/uring/pull/2 +[#3]: https://github.com/rustfs/uring/pull/3 +[#4]: https://github.com/rustfs/uring/pull/4 +[#5]: https://github.com/rustfs/uring/pull/5 +[#6]: https://github.com/rustfs/uring/pull/6 +[rustfs/backlog#1051]: https://github.com/rustfs/backlog/issues/1051 +[rustfs/backlog#1144]: https://github.com/rustfs/backlog/issues/1144 +[rustfs/backlog#1159]: https://github.com/rustfs/backlog/issues/1159 +[Unreleased]: https://github.com/rustfs/uring/commits/main diff --git a/README.md b/README.md index 03f4af3..da0adf8 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,25 @@ # rustfs-uring [![CI](https://github.com/rustfs/uring/actions/workflows/ci.yml/badge.svg)](https://github.com/rustfs/uring/actions/workflows/ci.yml) -[![crates](https://img.shields.io/crates/v/rustfs-uring.svg)](https://crates.io/crates/rustfs-uring) -[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/rustfs/uring/blob/master/LICENSE) -[![docs.rs](https://docs.rs/rustfs-uring/badge.svg)](https://docs.rs/rustfs-uring/) +[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/rustfs/uring/blob/main/LICENSE) Cancel-safe async `io_uring` read backend for [RustFS](https://github.com/rustfs/rustfs) storage. -This crate is the io_uring integration that RustFS's P2 read path is built on. It started as the Spike 0 cancel-safety +This crate is the io_uring integration that RustFS's read path is built on. It started as the Spike 0 cancel-safety prototype ([rustfs/backlog#894](https://github.com/rustfs/backlog/issues/894)) and was hardened per the [#1048/#1051 audit](https://github.com/rustfs/backlog/issues/1051). It lives in its own repository so it can be verified in isolation — with a real io_uring CI leg that the main `rustfs/rustfs` workspace cannot run — before being wired into the storage layer. -> **Status:** read path only, Linux only. Not yet published to crates.io — depend on it via git until the P2 work (write -> path, eventfd/`AsyncFd` reaping, O_DIRECT, `LocalIoBackend` integration) lands. See [`docs/DESIGN.md`](docs/DESIGN.md). +> **Status:** read path only, Linux only. **Not published to crates.io** (`publish = false`) — depend on it by pinning a +> git revision. The read path is wired into `rustfs/rustfs` behind a runtime probe and is **off by default** +> (`RUSTFS_IO_URING_READ_ENABLE`). See [`CHANGELOG.md`](CHANGELOG.md) for what has landed and +> [`docs/DESIGN.md`](docs/DESIGN.md) for the invariants. + +```toml +[target.'cfg(target_os = "linux")'.dependencies] +rustfs-uring = { git = "https://github.com/rustfs/uring", rev = "" } +``` ## The ownership model it enforces @@ -31,8 +36,10 @@ proves and enforces the invariants any production io_uring integration must foll at the CQE, not at future drop; short reads on positioned reads are resubmitted to satisfy the whole-range contract; the probe file is opened via `O_TMPFILE`. -The full invariant list, the corrected fd-reuse mechanism, and the P2 design constraints are in [ -`docs/DESIGN.md`](docs/DESIGN.md). +Each invariant holds **per shard** (see below), because a shard is an independent instance of the same driver. + +The full invariant list, the corrected fd-reuse mechanism, and the design constraints are in +[`docs/DESIGN.md`](docs/DESIGN.md). ## Usage @@ -42,25 +49,81 @@ use std::sync::Arc; use rustfs_uring::UringDriver; # async fn demo() -> std::io::Result<()> { - // Probe a real IORING_OP_READ before accepting work; a restricted environment - // (seccomp/gVisor/old kernel) returns a ProbeFailure you degrade to the std - // backend on. + // Probe a real IORING_OP_READ before accepting work. A restricted environment + // (seccomp/gVisor/old kernel) returns a ProbeFailure whose + // `is_expected_restriction()` tells you to degrade to the std backend quietly. let driver = UringDriver::probe_and_start(64).expect("io_uring available"); let file = Arc::new(File::open("/data/object")?); - // Positioned read (pread semantics, whole-range). + // Positioned read (pread semantics, whole-range: short reads are resubmitted). let bytes = driver.read_at(Arc::clone(&file), 0, 65536).await?; // Dropping the returned future before it completes is safe — the driver owns // the buffer until the CQE. - driver.shutdown(); + let snapshot = driver.shutdown(); + assert_eq!(snapshot.delivered + snapshot.orphan_reclaimed, snapshot.submitted); # Ok(()) # } ``` +### Sharded rings + +A buffered read that hits the page cache completes *inline* inside `io_uring_enter`, so the thread driving a ring +performs that read's `memcpy`. One ring is therefore capped at a single core's memory bandwidth (~5 GB/s measured). +Give a disk several rings when its reads hit the cache: + +```rust +# use rustfs_uring::UringDriver; +// Four independent rings, each with `entries` SQ slots and its own driver thread. +// In-flight is capped per shard, so the driver admits up to `shards * entries` reads. +let driver = UringDriver::probe_and_start_sharded(64, 4)?; +# Ok::<_, rustfs_uring::ProbeFailure>(()) +``` + +`probe_and_start(entries)` is exactly `probe_and_start_sharded(entries, 1)`, so nobody grows threads by upgrading. +Rings stay per-disk: a stalled disk cannot starve another disk's rings. + +### `O_DIRECT` + +Open the fd with `O_DIRECT`, pass the device's logical block size, and let the driver do the alignment. `offset` and +`len` need **not** be aligned — it reads a block-aligned superset into a block-aligned buffer and hands back exactly +the range you asked for. Padding, the bytes before the range, and the block-aligned tail never escape. + +```rust +# use rustfs_uring::UringDriver; +# use std::{fs::File, sync::Arc}; +# async fn demo(driver: &UringDriver, file: Arc) -> std::io::Result<()> { +// `file` was opened with O_DIRECT; 4096 is the probed logical block size. +let bytes = driver.read_at_direct(file, 8_191, 100, 4096).await?; +assert_eq!(bytes.len(), 100); +# Ok(()) +# } +``` + +## When this crate helps — and when it does not + +These numbers come from the harnesses in this repository and from end-to-end profiling of RustFS +([rustfs/backlog#1159](https://github.com/rustfs/backlog/issues/1159)). They are reported as measured, including the +cases where io_uring loses. + +| workload | result | +| --- | --- | +| **Many concurrent positioned reads on one disk** (erasure-coded shard reads) | **Where it wins.** With sharded rings and a cached fd: 64 KiB at concurrency 128 → 361k IOPS vs 125k for a blocking-pool baseline, and p999 3.0 ms vs 13.5 ms. | +| **A single sequential stream** | **It loses.** Kernel readahead already does what pipelining would buy. Cold reads are device-bound; on a warm page cache io_uring reaches only 11–41% of a buffered read. Streaming reads should stay on the std backend. | +| **One read at a time (low concurrency)** | **It loses.** Per-op submission overhead exceeds a page-cache `memcpy`. | +| **End-to-end S3 GET** | **Roughly neutral today (−7% … +4%).** The disk read is not the bottleneck: a cached 1 MiB GET spends ~25% of CPU in `memcpy` and ~10% in `memset`, and 0% on device reads. Optimising the read path further only pays once those copies are gone. | + +Two traps this crate's own benchmarking fell into, documented so others do not repeat them: + +- A `76×` apparent speedup turned out to be a **behaviour regression**, not a win: the io_uring path had silently + stopped honouring RustFS's `fadvise(DONTNEED)` page-cache reclaim policy, so one leg served everything from cache + while the other read the device. Always check `disk_read` and page-cache deltas, not just throughput. +- Microbenchmarks of the read path measured a page-cache-hit regime that production *deliberately avoids* for large + reads. Isolated-path gains do not transfer end-to-end for free. + ## Testing This is a Linux-only crate; on a non-Linux host `cargo check` only builds the empty stub. @@ -77,15 +140,37 @@ cargo test -- --nocapture --test-threads=1 ``` The harness fails on either a non-degrading leg 1 or a vacuous-pass leg 2, so a skipped suite can never masquerade as -real coverage. +real coverage. The 15 acceptance tests are the cancel-safety contract: buffer conservation under a mixed +drop/keep stress across shards, an orphaned op reclaimed only at its CQE, bounded shutdown drain, `O_DIRECT` returning +exact unaligned ranges, and backpressure deferring rather than blocking a runtime worker. + +## Benchmarks + +Both harnesses refuse to overwrite or follow a symlink at a caller-supplied path (the sweeps run as root), fill their +files with an offset-addressable pattern, and check every delivered byte against it under `BENCH_VERIFY=1` — throughput +alone cannot tell a correct strategy from one reading the wrong offsets. + +```bash +# Sequential whole-file read: buffered vs O_DIRECT vs pipelined io_uring, warm and cold cache. +./bench-streaming.sh + +# Many concurrent positioned reads on one disk — the shape shard reads actually serve. +# Isolates the cost of the per-read open and of the spawn_blocking hop. +./bench-concurrent-pread.sh +``` + +## Roadmap -## Roadmap (P2) +- **Write path.** Untouched today; PUT still goes through the blocking pool, and profiling suggests the win there may + exceed the read path's. +- **`register_files`.** Would remove the per-op fd lookup. Lower value now that the consumer caches descriptors. +- **`SQPOLL`.** Eliminates `io_uring_enter` under sustained load, at the cost of a kernel polling thread per ring — + which multiplies by shards and by disks. Only for high-end deployments. -- eventfd + tokio `AsyncFd` reaping to replace the poll loop (must still flush the NODROP overflow list). -- O_DIRECT aligned buffers and the three read shapes. -- `LocalIoBackend` integration in `rustfs/rustfs` behind runtime probing. -- Per-disk probe cache and runtime errno degradation latch. -- Registered buffers (P3) with the content-hygiene invariant, and the write path (P4). +Closed by measurement, not built — see [`CHANGELOG.md`](CHANGELOG.md#decisions-recorded-not-implemented): streaming +reads through io_uring (NO-GO), `AsyncFd` reaping without a driver thread (would break the public API), a process-wide +singleton ring (conflicts with per-disk isolation), and registered buffers (conflicts with the `Vec` ownership +model, and the bottleneck is elsewhere). ## License