feat(driver): async backpressure via tokio Semaphore, permit released at the CQE (backlog#1102)

[houseme]

What

Second productionization step for rustfs/backlog#1102: replace the hand-rolled Mutex+Condvar backpressure semaphore with tokio::sync::Semaphore, so acquiring a permit never parks a runtime worker.

#1 (eventfd reaping) is merged; this branch is rebased onto main and now contains only the backpressure change.

Why

Backpressure::acquire() blocked the calling thread whenever entries ops were already in flight. In an async caller (ecstore's UringBackend) that parks a tokio worker under saturation — exactly the concern behind rustfs/backlog#1060.

Design

• Fast path unchanged. submit takes the permit with try_acquire_owned(); when a permit is free (the common, unsaturated case) there is no allocation, no await, and the op is in flight the moment submit returns — byte-for-byte the old behavior.
• Saturated path never blocks. The acquire future is handed to the returned ReadHandle, which awaits it on its first poll and submits then.
• "Release at the CQE" is now type-enforced. The OwnedSemaphorePermit travels with Msg::Read into the Pending entry and is dropped exactly when that entry is removed at the final CQE — no manual release() to forget. A short-read resubmit keeps the entry, and thus the permit.
• The driver close()s the semaphore on exit, so a handle still awaiting a permit resolves with a driver-gone error instead of hanging.

Compatibility

• Public API unchanged: read_at/read_current still return ReadHandle; ReadHandle is still the awaitable. ecstore needs no change, and there is zero S3/client impact.
• One internal test assertion changed: cancel_stress no longer asserts submitted == OPS. That was an artifact of the blocking acquire, which forced every op to be submitted before its handle could be dropped. A handle dropped before its first poll now never acquires a permit, never submits, and never allocates a buffer — strictly better for memory — while the load-bearing conservation identity delivered + orphan_reclaimed == submitted still holds exactly.

Verification

• All 12 tests pass under real io_uring, both run-docker.sh legs (leg 1 seccomp-blocked → graceful degradation; leg 2 unconfined → real io_uring).
• New saturated_submit_defers_instead_of_blocking: saturates every permit with blocked pipe reads, then asserts read_at returns without submitting (this test would hang under the old blocking implementation), and that the deferred handle submits and returns correct bytes once a permit frees.
• cargo fmt --check + clippy --all-targets -D warnings clean.

Remaining in #1102

tokio AsyncFd reaping (removing the dedicated driver thread), O_DIRECT aligned buffers, the three read shapes, and the process-level singleton ring.

[Copilot]

Pull request overview

This PR productionizes async backpressure for the io_uring driver by replacing the blocking Mutex/Condvar backpressure with a tokio::sync::Semaphore, ensuring permit acquisition never parks a Tokio runtime worker while preserving the “release at CQE” invariant.

Changes:

• Replaced the blocking backpressure mechanism with tokio::sync::Semaphore, deferring submission to first poll when saturated and carrying OwnedSemaphorePermit into Pending for CQE-scoped release.
• Updated ReadHandle lifecycle to support deferred submission (WaitingPermit) and to only issue async-cancel on drop when actually submitted.
• Adjusted cancel-stress accounting and added a saturation regression test ensuring saturated read_at returns immediately without submitting.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/driver.rs	Implements async backpressure via Tokio semaphore; enforces CQE-scoped permit release and deferred submission under saturation.
tests/cancel.rs	Updates invariants for async backpressure and adds a saturation test to ensure submit does not block.
docs/DESIGN.md	Documents the new semaphore-based backpressure and CQE-release enforcement.