Repository navigation
feat(io): io_uring TCP path for HTTP/1.1 and HTTP/2 - #416
Merged
Merged
Conversation
Extend the io_uring TCP path with a multishot IORING_OP_ACCEPT on the async server's listen socket: one persistent SQE replaces the one-shot POLL_ADD + accept4 drain loop, and each completion IS an accepted, already CLOEXEC|NONBLOCK socket. - reactor: accept slots tag bit 1 of user_data (bit 0 is the RX-uring conn tag); the slot is persistent, freed on terminal completion or at reactor destroy. CWIST_ACCEPT_URING=0 refuses the arm. - Fallback is silent in both directions: no io_uring ring keeps the epoll path, and a terminal completion (e.g. -EINVAL on kernels without multishot accept) hands the listener back to the one-shot POLL loop. - TLS connections flow through the same accept into cwist_https_dispatch; BoringSSL stays synchronous on its pool threads (documented delta). Tests: tests/test_reactor_accept.c (functional run, knob-off refusal, forced-epoll refusal, terminal error) and tests/test_http_uring_hammer.c (8x200 keep-alive requests on default, CWIST_ACCEPT_URING=0, and CWIST_REACTOR_BACKEND=epoll). Both registered in make test; ASan/UBSan clean.
gg582
added a commit
that referenced
this pull request
Oct 19, 2026
* feat(io): multishot io_uring accept for HTTP/1.1 and HTTP/2 Extend the io_uring TCP path with a multishot IORING_OP_ACCEPT on the async server's listen socket: one persistent SQE replaces the one-shot POLL_ADD + accept4 drain loop, and each completion IS an accepted, already CLOEXEC|NONBLOCK socket. - reactor: accept slots tag bit 1 of user_data (bit 0 is the RX-uring conn tag); the slot is persistent, freed on terminal completion or at reactor destroy. CWIST_ACCEPT_URING=0 refuses the arm. - Fallback is silent in both directions: no io_uring ring keeps the epoll path, and a terminal completion (e.g. -EINVAL on kernels without multishot accept) hands the listener back to the one-shot POLL loop. - TLS connections flow through the same accept into cwist_https_dispatch; BoringSSL stays synchronous on its pool threads (documented delta). Tests: tests/test_reactor_accept.c (functional run, knob-off refusal, forced-epoll refusal, terminal error) and tests/test_http_uring_hammer.c (8x200 keep-alive requests on default, CWIST_ACCEPT_URING=0, and CWIST_REACTOR_BACKEND=epoll). Both registered in make test; ASan/UBSan clean. * fix(http): make cached keep-alive timeout atomic under TSan
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the io_uring TCP path (issue #406) with a multishot accept for the async server: one persistent
IORING_OP_ACCEPT+IORING_ACCEPT_MULTISHOTSQE on the listen socket replaces the one-shotPOLL_ADD+accept4()drain loop. Each completion is an accepted, alreadySOCK_CLOEXEC | SOCK_NONBLOCKsocket, dispatched straight into the HTTP pool / HTTPS dispatcher.Note on scope: exploration showed the read side of #406 mostly already existed — the Linux reactor defaults to io_uring (with silent epoll fallback) and the RX-uring path (
cwist_reactor_recv_arm, issue #179) already doesIORING_OP_RECVfor HTTP/1.1 keep-alive reads. The missing piece on the connection path was accept, which this PR lands.Design
cwist_reactor_accept_arm()(privatereactor_rx.h): arms one multishot accept per listen fd. The accept slot is a persistent registration (unlike the recycled one-shot ev_ctx slots) and tags bit 1 ofuser_data(bit 0 is the RX-uring conn tag; malloc alignment keeps both clear). The run loop's CQE dispatch gains a branch for it;cqe->res >= 0is the new fd, and a completion withoutIORING_CQE_F_MOREis terminal: the slot is unlinked, the callback is notified once with-errno(or-ECANCELED), and the slot is freed. Slots still live at destroy are freed there.async_server.c:async_accept_dispatch()now holds the shared per-socket handoff (fd_forget, TCP_NODELAY/QUICKACK, HTTPS-vs-HTTP dispatch), used by both the accept4 loop and the uring callback.cwist_async_server_loop()arms the multishot accept first and falls back to the existingcwist_reactor_add(... async_accept_cb ...)arm on failure.CWIST_ACCEPT_URING(0disables; default on when the reactor has a ring). Documented indocs/runtime-config.mdanddocs/ko/runtime-config.md;make env-checkpasses.Fallback behavior (all silent)
CWIST_REACTOR_BACKEND=epoll) → existing epoll reactor, unchanged.CWIST_ACCEPT_URING=0or the arm fails → one-shot POLL + accept4 loop.-EINVALon a kernel withoutIORING_ACCEPT_MULTISHOT) → terminal completion hands the listener back to the one-shot POLL loop, once.Tests
tests/test_reactor_accept.c(registered astest_reactor_accept, runs 3 ways): functional accept of 64 connections on one arm (fd flags verified),CWIST_ACCEPT_URING=0arm refusal, forced-epoll arm refusal, and a terminal-error run (accept on a non-listening socket → exactly one negative callback).tests/test_http_uring_hammer.c(registered astest_http_uring_hammer): forked cwist app server, 8 client threads x 200 sequential keep-alive requests = 1600 requests; run on the default (io_uring + multishot accept),CWIST_ACCEPT_URING=0, andCWIST_REACTOR_BACKEND=epollconfigurations.make SANITIZE=address,undefined WERROR=1 test_reactor_accept test_http_uring_hammer test_reactor_wake test_reactor_timer test_rx_epoll test_https test_http2 test_http test_linux_writer_fast— all pass, zero sanitizer/leak reports (io_uring behaved normally under ASan here, kernel 6.12).make -j12 WERROR=1 test_reactor_wake test_reactor_timer test_rx_epoll test_http test_http2 test_https test_linux_writer_fast— all pass (both backends where the test supports it).make env-checkandmake api-checkpass (no public API added).Benchmarks (local, Linux 6.12, single 5 s runs — indicative, not gates)
benchmarks/web-frameworks/cwist_bench, wrk-t4 -c64 -d5skeep-alive:CWIST_ACCEPT_URING=0CWIST_REACTOR_BACKEND=epollConnection churn (
ab -n 30000 -c 64, one connection per request, 2 runs each): multishot accept ≈14.2k req/s vs ≈13.8k with the knob off (~+3%, consistent across runs); epoll ≈14.6k.Reading: at keep-alive steady state the accept path is cold, so the change is neutral (within noise) there — its value is fewer syscalls and no per-batch re-arm under connection churn. The epoll backend being slightly ahead of io_uring overall on this box is a pre-existing characteristic of the reactor (read path untouched by this PR), not caused by this change.
Not done (deferred, explicit)
IORING_OP_RECVinto the connection stash; registered buffers conflict with the stash's realloc-on-growth lifecycle and were judged too invasive for this pass.writer_fastspeculativesend/sendmsg+ the parkedPOLLOUTresume. AnIORING_OP_SENDresume is a possible follow-up.benchmarks/webserver+https_gatesgates: not run locally; the wrk/ab numbers above are the locally feasible subset.Closes #406