Skip to content

Close the transfer socket and set CLOEXEC on received listener fds - #960

Open
ShanireZ wants to merge 1 commit into
cloudflare:mainfrom
ShanireZ:fd-leaks-in-get-fds-from
Open

Close the transfer socket and set CLOEXEC on received listener fds#960
ShanireZ wants to merge 1 commit into
cloudflare:mainfrom
ShanireZ:fd-leaks-in-get-fds-from

Conversation

@ShanireZ

Copy link
Copy Markdown

Refs #959

get_fds_from() leaks two kinds of file descriptors on every graceful upgrade:

  • The connection returned by accept() is a bare RawFd that is never closed; only
    listen_fd is. Every completed upgrade leaks one connected unix socket for the lifetime of
    the process.
  • recvmsg() is called with MsgFlags::empty(), so the listening sockets received over
    SCM_RIGHTS do not have FD_CLOEXEC. They are held for the rest of the process's lifetime
    and are therefore inherited by every child it later exec()s.

The second one also interacts with the cleanup added in "Close unclaimed inherited listening
sockets on graceful upgrade"
: an fd that reaches a process through exec() is not in the
Fds table, so listen_addresses() cannot close it.

This takes ownership of the accepted connection with OwnedFd so it is closed on every path
out of the function (including the early return from cmsgs()?), and passes
MSG_CMSG_CLOEXEC so the received descriptors get FD_CLOEXEC atomically rather than via a
follow-up fcntl(), which would race with a concurrent fork().

Setting CLOEXEC cannot affect daemonization

Worth stating explicitly, since that is the obvious thing to worry about: FD_CLOEXEC only
takes effect on exec(), and nothing in this workspace execs. Daemonization goes through
daemonix::Daemonize, which forks — fork() copies the descriptor table regardless of
FD_CLOEXEC, so the daemon child keeps the received listeners exactly as before. A repo-wide
search finds no Command::new, exec* or CommandExt outside of tests.

The test covers both defects independently

test_receive_does_not_leak_fds checks the FD_CLOEXEC bit on the received fd, and counts
the descriptors in this process that refer to a unix socket bound to the transfer path
(via /proc/net/unix, so it is not affected by unrelated fds opened by tests running in
parallel). Reverting either half of the fix makes it fail, with distinct messages:

reverted failure
MSG_CMSG_CLOEXEC fd received over SCM_RIGHTS is missing FD_CLOEXEC
the OwnedFd the accepted transfer socket was left open

Verification

Commands taken from .github/workflows/build.yml, run against main (0046038) with this
patch applied:

Gate Result
cargo fmt --all -- --check pass
cargo check --workspace (1.97.1) pass
cargo +1.85.0 check --workspace --exclude pingora-foundations pass (MSRV unaffected)
cargo clippy --all-targets --all -- --allow=unknown-lints --deny=warnings pass
cargo test -p pingora-core --lib --no-fail-fast 544 passed / 2 failed — the same two that fail on unmodified main (connectors::l4::tests::test_conn_timeout, test_bind_to_port_range_on_connect); two-way diff of the failure sets is empty
cargo audit unaffected — no dependency change
cargo machete not run (not installed locally); no manifest is touched

The full workspace test suite was not run end to end here: it needs openresty as a test
backend, which this environment does not have. The change is confined to pingora-core, and
that crate's full lib test suite is in the table above with a baseline comparison.

Scope note

get_fds_from() also creates its listening socket with SockFlag::SOCK_NONBLOCK but without
SOCK_CLOEXEC, and accept_with_retry_timeout() uses accept() rather than accept4().
Those descriptors are closed before the function returns, so they only expose a narrow window
to a concurrent fork(). I left them alone to keep this focused on the two descriptors that
actually outlive the call — happy to add them if you'd rather close the window too.

Copilot AI lite review requested due to automatic review settings August 14, 2026 05:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes FD leaks during graceful upgrade FD transfer by ensuring the accepted transfer connection is always closed and that received listener fds are marked FD_CLOEXEC atomically, preventing unintended inheritance across exec().

Changes:

  • Wrap the accepted accept() connection fd in OwnedFd so it’s closed on all exit paths (including early returns).
  • Call recvmsg() with MSG_CMSG_CLOEXEC so SCM_RIGHTS-received listener fds get FD_CLOEXEC atomically.
  • Add a regression test validating both the FD_CLOEXEC bit and that the accepted transfer socket is not left open.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +546 to +552
// The accepted connection is only needed during the transfer itself.
assert_eq!(
0,
unix_socket_fds_bound_to(SOCK),
"the accepted transfer socket was left open"
);
});

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — adopted, the test now closes what it receives.

For the record, since it is relevant to this PR specifically: the two neighbouring tests
(test_send_receive_fds, test_serde_via_socket) leak the received fds too, but that is not a
reason to keep doing it in a PR about fd leaks.

One thing worth flagging while checking this: I initially thought the change caused
close_unclaimed_tests::closes_only_unclaimed_fds to start failing. It does not — that test is
already flaky on unmodified main in my environment. Measured 20 runs per configuration on a
12-core container:

configuration closes_only_unclaimed_fds failures
unmodified main 5/20
this PR, before adopting your suggestion 2/20
this PR, after adopting it 5/20

A pristine clone of main on its own gives 7/20, so the rate is unrelated to this branch.

The cause is independent of this PR: the test closes an fd and then asserts that the number is
invalid (fcntl(drop_fd, F_GETFD) == -1 with EBADF), but fd numbers are process-wide and are
reused immediately, so any test thread that opens a descriptor in that window takes the number back.
Filed separately as #963 rather than folded into this PR.

get_fds_from() leaks two kinds of file descriptors on every graceful
upgrade:

  - The connection returned by accept() is a bare RawFd that is never
    closed; only listen_fd is. Every completed upgrade therefore leaks
    one connected unix socket for the lifetime of the process.

  - recvmsg() is called with MsgFlags::empty(), so the listening sockets
    received over SCM_RIGHTS do not have FD_CLOEXEC set. Those fds are
    held for the lifetime of the process, so they are inherited by every
    child it later execs.

The second one also interacts with the unclaimed-fd cleanup added in
"Close unclaimed inherited listening sockets on graceful upgrade": an fd
that reaches a process through exec() is not in the Fds table, so
listen_addresses() cannot close it. Deployments that start the new
generation by forking from the old one (needed when the service manager
tracks a cgroup) receive each listening socket twice, and only one of the
two copies is visible to that cleanup.

Take ownership of the accepted connection with OwnedFd so it is closed on
every path out of the function, including the early return from cmsgs()?,
and pass MSG_CMSG_CLOEXEC so the received descriptors get FD_CLOEXEC
atomically rather than through a follow-up fcntl().

The added test covers both defects independently: reverting
MSG_CMSG_CLOEXEC makes it report the missing flag, and reverting the
OwnedFd makes it find the accepted socket still open.

Signed-off-by: Shanire <shanire86@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

WIP We are working on this feature internally

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants