feat(driver): native aligned O_DIRECT positioned read (backlog#1102)#3
Merged
Conversation
…#1102) Add `read_at_direct(file, offset, len, align)` for files opened with O_DIRECT. The caller's offset/len need no alignment: the driver reads the block-aligned superset range into a block-aligned buffer and hands back exactly [offset, offset + len). - Geometry (`aligned_geometry`) is computed once and validated in `submit`: a non-power-of-two alignment, or an aligned range past MAX_RW_COUNT, is rejected with InvalidInput before the kernel sees it. - The buffer is over-allocated by `align - 1` and the read region starts at the first aligned byte inside the allocation, so the SQE pointer, the kernel offset and the read length are all block-aligned. `Vec<u8>` stays the buffer type, so the cancel-safety ownership model is untouched. - A short read resubmits from `offset + nread` into `buf[pad + nread ..]`, which stays block-aligned; a non-block-multiple return means the file tail, so the driver stops and delivers. - `deliver()` slices out only the logical range: alignment padding, the bytes before the range, and the block-aligned tail never escape (a BitrotReader expecting an exact shard length would flag padded output as corruption), and the delivered length is clamped to what the kernel actually wrote, so the zero-filled remainder stays hidden (content hygiene, rustfs/backlog#1062). A buffered read is the `align == 1` degenerate case of the same geometry, so the existing paths are unchanged — all 12 prior tests still pass. Adds `direct_read_returns_exact_unaligned_ranges`: real O_DIRECT over real io_uring, covering unaligned starts, block-straddling ranges, the unaligned file tail, and the rejected non-power-of-two alignment. The filesystem in both run-docker.sh legs supports O_DIRECT, so the assertions are actually exercised (the test says so explicitly when a filesystem refuses O_DIRECT). 13 tests pass under real io_uring; clippy -D warnings clean. Co-Authored-By: heihutu <heihutu@gmail.com>
There was a problem hiding this comment.
Pull request overview
Adds a new additive driver API to support aligned O_DIRECT positioned reads while preserving the existing cancel-safety/backpressure ownership model, and updates tests/docs to validate that only the logical requested range is returned (no alignment padding leakage).
Changes:
- Introduce
read_at_direct(...)and direct-read alignment geometry (aligned_geometry) to read a block-aligned superset range and deliver only[offset, offset + len). - Refactor pending-read bookkeeping to track pad/head/want/region_len/align and adjust short-read resubmission + delivery slicing accordingly.
- Add an
O_DIRECTintegration test and update design documentation to reflect the new direct-read behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/driver.rs |
Adds read_at_direct, alignment geometry, aligned buffer handling, and logical-range delivery for O_DIRECT reads. |
tests/cancel.rs |
Adds an O_DIRECT test ensuring exact logical ranges are returned and misaligned align is rejected. |
docs/DESIGN.md |
Documents the completed O_DIRECT aligned-read milestone and its invariants. |
Address Copilot review on #3: - aligned_geometry now rejects align > MAX_READ_LEN, foreclosing the align_offset == usize::MAX path (which would make the driver's ptr::add UB) and the region_len + align - 1 allocation overflow. - The driver's aligned-buffer setup replaces the release-mode-only debug_assert with a checked allocation size and a runtime guard: an unsatisfiable align_offset now fails the read instead of doing UB pointer arithmetic. - deliver() skips the full-buffer copy_within on the buffered path (align == 1, start == 0) — no per-read memmove for the common case. - Test open_direct only degrades to skip on the real O_DIRECT-refusal errnos (EINVAL/EOPNOTSUPP), retries EINTR, and panics on anything else so ENOENT/EMFILE/EACCES can't masquerade as a vacuous pass. Verified: clippy -D warnings clean; run-docker.sh both legs pass (leg 1 degrades gracefully, leg 2 runs real io_uring 13/13 including direct_read_returns_exact_unaligned_ranges). Co-Authored-By: heihutu <heihutu@gmail.com>
houseme
added a commit
to rustfs/rustfs
that referenced
this pull request
Jul 10, 2026
…acklog#1102) (#4649) fix(ecstore): keep O_DIRECT for eligible reads via native io_uring (rustfs/backlog#1102) Wire ecstore's io_uring read backend to rustfs-uring's native aligned O_DIRECT read (`read_at_direct`, merged in rustfs/uring#3), so an O_DIRECT-eligible read keeps BOTH io_uring's async submission AND O_DIRECT's page-cache bypass instead of trading one for the other. `UringBackend::pread_bytes` now selects the read shape by a tiered, per-disk-latched ladder — never a blanket downgrade: 1. io_uring latched off for this disk (backlog#1101) -> StdBackend. 2. O_DIRECT-eligible + native path usable -> pread_uring_direct: open the file O_DIRECT, probe the device alignment once (cached), and read via read_at_direct. Best path: async + no cache pollution. 3. O_DIRECT-eligible but the fs refused O_DIRECT earlier (latched) -> StdBackend aligned path (which itself degrades to buffered). 4. non-O_DIRECT read -> buffered pread_uring. Latching keeps a failure from being re-attempted per-read: an fs that refuses O_DIRECT (EINVAL/EOPNOTSUPP on open) latches the native direct path off for that disk; a restriction-class errno from the read latches io_uring off wholesale (backlog#1101). Any other error falls back for that one read without masking a genuine data problem as a permanent downgrade. The alignment comes from the existing probe_direct_io_align (statx STATX_DIOALIGN, default 4096) and is cached in a per-disk OnceLock so the probe runs at most once. This replaces the interim guard that routed every O_DIRECT-eligible read to StdBackend (which disabled io_uring on the hottest shard reads); the native path is now the default and StdBackend is only a fallback tier. Bumps the rustfs-uring pin to the merged #3 commit. Verified on a real Linux host (16-core, real io_uring + O_DIRECT): cargo check --tests, clippy -D warnings, and disk::local tests all pass (128 passed, 0 failed), including uring_preserves_o_direct_for_eligible_reads across unaligned ranges. Co-authored-by: heihutu <heihutu@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Third productionization step for rustfs/backlog#1102: a native aligned O_DIRECT positioned read in the driver.
API (additive, nothing else changes)
The caller opens the file with
O_DIRECTand passes the device's logical block size.offsetandlenneed no alignment — the driver reads the block-aligned superset range into a block-aligned buffer and returns exactly[offset, offset + len).Design
aligned_geometry) is computed once and validated insubmit: a non-power-of-two alignment, or an aligned range pastMAX_RW_COUNT, is rejected withInvalidInputbefore the kernel sees it.Vec<u8>— over-allocate byalign - 1and start the read region at the first aligned byte inside the allocation. The SQE pointer, the kernel offset and the read length are then all block-aligned. The cancel-safety ownership model is untouched.offset + nreadintobuf[pad + nread ..], which stays block-aligned. A non-block-multiple return means the file tail, so the driver stops and delivers.deliver()slices out only the logical range: alignment padding, the bytes before the range, and the block-aligned tail never escape — aBitrotReaderexpecting an exact shard length would flag padded output as corruption. The delivered length is clamped to what the kernel actually wrote, so the zero-filled remainder stays hidden (content hygiene, rustfs/backlog#1062).align == 1degenerate case of the same geometry, so the existing paths are unchanged.Verification
run-docker.shlegs;clippy --all-targets -D warningsandcargo fmt --checkclean.direct_read_returns_exact_unaligned_rangesuses realO_DIRECTover real io_uring, covering unaligned starts, block-straddling ranges, the unaligned file tail, and the rejected non-power-of-two alignment. Both legs' filesystems support O_DIRECT (verified), so the assertions are genuinely exercised — and the test says so explicitly if a filesystem ever refuses O_DIRECT, rather than passing vacuously.Follow-up
Wiring ecstore to open the fd
O_DIRECTand callread_at_direct(replacing the temporary "route O_DIRECT-eligible reads back to StdBackend" guard in rustfs/rustfs#4645).