[Rastrian]

Summary

Two production bugs found during the 2026-08-31 incident, both fixes protocol-coupled and included here. No wire-protocol changes — the frame layer (Spore.Shared.Delimited) is untouched, so Rust bore interop is preserved.

Root cause

1. Client: --retry was a no-op for half-open connections

Spore.Client.loop/4 read the control connection with Delimited.recv(d), i.e. a :infinity timeout. The control plane only ever receives "Heartbeat" frames (every 500ms). When the connection dies without a FIN/RST reaching the client — NAT/conntrack flush, silent host loss — the socket looks perfectly healthy, recv blocks forever, and neither the :eof nor the error clause that triggers reconnect/3 can ever fire.

Fix: bound every control-loop read by a heartbeat timeout — application env :heartbeat_timeout_ms, default 5000 ms (the server beats every 500 ms, so the default tolerates ten missed beats). On read timeout the socket is closed (no FIN is coming) and the loop goes through reconnect/3 when retry is enabled, or returns {:error, :heartbeat_timeout} when it is not. Data-plane connections are untouched (short-lived, heartbeatless by design).

2. Server: control sockets leaked to GC on handler failure paths

Spore.Server.handle_connection/4 returned :ok on its failure paths — eof, handshake read timeout, decode error, unexpected message, refused Hello — without closing the socket, leaving cleanup to port GC. Under a storm of doomed connections (clients retrying against a server that had just dropped their tunnels) this piles sockets up in CLOSE-WAIT and leaks FDs.

Fix: a single cleanup point — :gen_tcp.close(socket) in an after block — guarantees the close on every exit path (normal return, raise, exit/1). Redundant inline closes on the auth-failure paths were removed. handle_accept_connection/2 gets the same guarantee. Closing again after Shared.pipe_bidirectional already closed the socket is a verified no-op.

3. Hardening: accept loop survives EMFILE

{:ok, socket} = accept_conn(...) meant one {:error, :emfile} crashed the whole accept loop. Now it logs, waits 1 s and retries.

Evidence summary (production, v0.2.8, 2026-08-31)

• Server: "removed stale connection" x2 → "client control connection lost" for both tunnels in the same second (correlated client-side NAT/conntrack flush) → server kept accepting on :7835 but "new client on port" never fired again; ss -tn showed the peers stuck in CLOSE-WAIT with Recv-Q 1 (unread hellos); process FD count 3844 after a day of repeated {:error, :emfile} from accept.
• Clients (k3s sidecars, --retry on): control TCP stayed ESTABLISHED (zombie half-open against a server that had moved on), no reconnect attempts, no logs for hours; only a pod restart recovered them.

What changed

• lib/spore/client.ex — heartbeat-bounded reads in the control loop, :heartbeat_timeout_ms env (default 5000), close + reconnect/return on timeout.
• lib/spore/server.ex — guaranteed control-socket close on every handler exit path; accept-loop retry on transient accept errors.

Test plan

TDD: the new tests were written first and verified red against the unfixed code ({:error, :timeout} from recv = leaked socket; no reconnect / no return on the client side), then green after the fixes. All use real :gen_tcp sockets over loopback — no transport mocks.

• test/spore/client_heartbeat_test.exs — fake server answers Hello then goes silent (the half-open state): with retry, the client reconnects (a fresh handshake is observed); without retry, listen/1 returns {:error, :heartbeat_timeout} instead of hanging.
• test/spore/server_test.exs — garbage first message / total silence / refused out-of-range Hello / a run of unknown messages: the peer must see a FIN (recv → {:error, :closed}) instead of timing out against a GC-abandoned socket. Regression guards: healthy session still gets a tunnel port, heartbeats and listener release; end-to-end Accept pipe still forwards and tears down.
• mix test: 138 passed (132 → 138). mix format --check-formatted: clean. Wire protocol unchanged (no new frames; Delimited untouched).

🤖 Generated with Claude Code