Skip to content

fix(dind): fail container start when it cannot be networked or published - #213

Merged
luthermonson merged 2 commits into
mainfrom
fix/dind-network-fail-closed
Oct 22, 2026
Merged

luthermonson merged 2 commits into
mainfrom
fix/dind-network-fail-closed

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Two Docker-shim bugs reported from ePHPm CI on 2026-10-22.

1. A container with no IP got no port forward, silently

ephpm run 39690226239 (job 119534309464, runner fast_linus on ducks): the postgres sibling ephpm-pg-39690226239-1 started with ip="". The ducks log shows why:

00:50:22.560Z WARN failed to setup network for dind container ... error="CNI setup for 27bdc817...: plugin type=\"loopback\" failed (add): interrupted system call"
00:50:22.573Z INFO container started ... ip=""

The CNI loopback plugin got EINTR from netlink while the job's two MySQL siblings were being wired up in the same second. The shim logged a warning, started the container anyway, and installPortForwards returned early on the empty IP, so 127.0.0.1:26241 never opened and the job waited 4 minutes. It happened once in 14 days of ducks logs.

Changes:

  • networking: retry a CNI ADD that fails with "interrupted system call" (3 attempts, CNI DEL between them), and release a failed ADD's allocations. The runner container's setup goes through the same path, so it gets the retry too.
  • dind: a network setup failure fails docker start with the error instead of starting a container with no network.
  • dind: port bindings are published before task.Start, and a binding that can't be honoured (no IP, listen error) fails the start the way docker run -p does ("port is already allocated"). Anything already set up is torn down, and the container is left in created.

2. The forwarder dialed the container with no timeout

forwardConn now uses net.DialTimeout (5s), closes the client when the dial fails, and logs the failure with listen and target addresses. The comment claiming the dial runs in the runner's netns is corrected: the per-connection goroutine isn't thread-locked, so it dials from the daemon's namespace.

This is hardening, not a confirmed fix for run 39710247641. The shim logs for stoic_newton and happy_raman (both on the desktop's Linux VM) show the MySQL container got an IP (10.88.0.27 / 10.88.0.34) and the forward was installed at start. The job's readiness step got a real MySQL handshake through 127.0.0.1:31641 at 02:17:53, about 8 minutes before pdo_mysql_proxy_write_read_roundtrip returned 500 at 02:26:12. A dial to a listening container on the same bridge connects or is refused at once, so the 30s+ hang is more likely inside the PDO → ePHPm [db.mysql] proxy → MySQL path than in the forwarder. The forwarder didn't log per connection before this change, so that can't be ruled out from these logs; with this change a failed dial will show up.

Tests

  • TestRetrySetup_*: EINTR is retried with cleanup; other errors aren't; gives up after 3 attempts.
  • TestInstallPortForwards_NoIPIsAnError / _NothingToPublish.
  • TestForwardConn_DialIsBounded: a dial to TEST-NET-1 gives up and closes the client.
  • go vet for windows and linux (CGO off); go test ./pkg/dind ./pkg/networking on Windows; the Linux pkg/dind and pkg/networking test binaries run in WSL, all pass. TestStartPortForwardProxy_OwnNetns skips without root, as before.

A dind sibling whose CNI setup failed was started anyway with no IP, and
its -p bindings were then skipped without a word. ephpm run 39690226239
waited 4 minutes on a postgres port that could never open: the CNI
loopback plugin had returned EINTR ("interrupted system call") from its
netlink calls while other containers were being wired up on ducks.

- networking: retry a CNI ADD that fails with EINTR (3 attempts, CNI DEL
  between them) and release a failed ADD's allocations.
- dind: a network setup failure now fails `docker start` with the error.
- dind: port bindings are published before the task starts, and one that
  cannot be honoured (no IP, listen failure) fails the start, as
  `docker run -p` does, instead of logging a warning.
- portforward: bound the dial to the container (5s) and log a failed dial,
  so a client accepted on 127.0.0.1 is closed at once rather than left on
  a silent connection for the kernel's ~2 minute connect timeout.
@luthermonson
luthermonson merged commit c986538 into main Oct 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant