Close the transfer socket and set CLOEXEC on received listener fds - #960
Close the transfer socket and set CLOEXEC on received listener fds#960ShanireZ wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Fixes FD leaks during graceful upgrade FD transfer by ensuring the accepted transfer connection is always closed and that received listener fds are marked FD_CLOEXEC atomically, preventing unintended inheritance across exec().
Changes:
- Wrap the accepted
accept()connection fd inOwnedFdso it’s closed on all exit paths (including early returns). - Call
recvmsg()withMSG_CMSG_CLOEXECso SCM_RIGHTS-received listener fds getFD_CLOEXECatomically. - Add a regression test validating both the
FD_CLOEXECbit and that the accepted transfer socket is not left open.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // The accepted connection is only needed during the transfer itself. | ||
| assert_eq!( | ||
| 0, | ||
| unix_socket_fds_bound_to(SOCK), | ||
| "the accepted transfer socket was left open" | ||
| ); | ||
| }); |
There was a problem hiding this comment.
Good catch — adopted, the test now closes what it receives.
For the record, since it is relevant to this PR specifically: the two neighbouring tests
(test_send_receive_fds, test_serde_via_socket) leak the received fds too, but that is not a
reason to keep doing it in a PR about fd leaks.
One thing worth flagging while checking this: I initially thought the change caused
close_unclaimed_tests::closes_only_unclaimed_fds to start failing. It does not — that test is
already flaky on unmodified main in my environment. Measured 20 runs per configuration on a
12-core container:
| configuration | closes_only_unclaimed_fds failures |
|---|---|
unmodified main |
5/20 |
| this PR, before adopting your suggestion | 2/20 |
| this PR, after adopting it | 5/20 |
A pristine clone of main on its own gives 7/20, so the rate is unrelated to this branch.
The cause is independent of this PR: the test closes an fd and then asserts that the number is
invalid (fcntl(drop_fd, F_GETFD) == -1 with EBADF), but fd numbers are process-wide and are
reused immediately, so any test thread that opens a descriptor in that window takes the number back.
Filed separately as #963 rather than folded into this PR.
get_fds_from() leaks two kinds of file descriptors on every graceful
upgrade:
- The connection returned by accept() is a bare RawFd that is never
closed; only listen_fd is. Every completed upgrade therefore leaks
one connected unix socket for the lifetime of the process.
- recvmsg() is called with MsgFlags::empty(), so the listening sockets
received over SCM_RIGHTS do not have FD_CLOEXEC set. Those fds are
held for the lifetime of the process, so they are inherited by every
child it later execs.
The second one also interacts with the unclaimed-fd cleanup added in
"Close unclaimed inherited listening sockets on graceful upgrade": an fd
that reaches a process through exec() is not in the Fds table, so
listen_addresses() cannot close it. Deployments that start the new
generation by forking from the old one (needed when the service manager
tracks a cgroup) receive each listening socket twice, and only one of the
two copies is visible to that cleanup.
Take ownership of the accepted connection with OwnedFd so it is closed on
every path out of the function, including the early return from cmsgs()?,
and pass MSG_CMSG_CLOEXEC so the received descriptors get FD_CLOEXEC
atomically rather than through a follow-up fcntl().
The added test covers both defects independently: reverting
MSG_CMSG_CLOEXEC makes it report the missing flag, and reverting the
OwnedFd makes it find the accepted socket still open.
Signed-off-by: Shanire <shanire86@gmail.com>
37ede26 to
f94e445
Compare
Refs #959
get_fds_from()leaks two kinds of file descriptors on every graceful upgrade:accept()is a bareRawFdthat is never closed; onlylisten_fdis. Every completed upgrade leaks one connected unix socket for the lifetime ofthe process.
recvmsg()is called withMsgFlags::empty(), so the listening sockets received overSCM_RIGHTSdo not haveFD_CLOEXEC. They are held for the rest of the process's lifetimeand are therefore inherited by every child it later
exec()s.The second one also interacts with the cleanup added in "Close unclaimed inherited listening
sockets on graceful upgrade": an fd that reaches a process through
exec()is not in theFdstable, solisten_addresses()cannot close it.This takes ownership of the accepted connection with
OwnedFdso it is closed on every pathout of the function (including the early return from
cmsgs()?), and passesMSG_CMSG_CLOEXECso the received descriptors getFD_CLOEXECatomically rather than via afollow-up
fcntl(), which would race with a concurrentfork().Setting CLOEXEC cannot affect daemonization
Worth stating explicitly, since that is the obvious thing to worry about:
FD_CLOEXEConlytakes effect on
exec(), and nothing in this workspace execs. Daemonization goes throughdaemonix::Daemonize, which forks —fork()copies the descriptor table regardless ofFD_CLOEXEC, so the daemon child keeps the received listeners exactly as before. A repo-widesearch finds no
Command::new,exec*orCommandExtoutside of tests.The test covers both defects independently
test_receive_does_not_leak_fdschecks theFD_CLOEXECbit on the received fd, and countsthe descriptors in this process that refer to a unix socket bound to the transfer path
(via
/proc/net/unix, so it is not affected by unrelated fds opened by tests running inparallel). Reverting either half of the fix makes it fail, with distinct messages:
MSG_CMSG_CLOEXECfd received over SCM_RIGHTS is missing FD_CLOEXECOwnedFdthe accepted transfer socket was left openVerification
Commands taken from
.github/workflows/build.yml, run againstmain(0046038) with thispatch applied:
cargo fmt --all -- --checkcargo check --workspace(1.97.1)cargo +1.85.0 check --workspace --exclude pingora-foundationscargo clippy --all-targets --all -- --allow=unknown-lints --deny=warningscargo test -p pingora-core --lib --no-fail-fastmain(connectors::l4::tests::test_conn_timeout,test_bind_to_port_range_on_connect); two-way diff of the failure sets is emptycargo auditcargo macheteThe full workspace test suite was not run end to end here: it needs openresty as a test
backend, which this environment does not have. The change is confined to
pingora-core, andthat crate's full lib test suite is in the table above with a baseline comparison.
Scope note
get_fds_from()also creates its listening socket withSockFlag::SOCK_NONBLOCKbut withoutSOCK_CLOEXEC, andaccept_with_retry_timeout()usesaccept()rather thanaccept4().Those descriptors are closed before the function returns, so they only expose a narrow window
to a concurrent
fork(). I left them alone to keep this focused on the two descriptors thatactually outlive the call — happy to add them if you'd rather close the window too.