Repository navigation
fix(dind): fail container start when it cannot be networked or published - #213
Merged
Merged
Conversation
A dind sibling whose CNI setup failed was started anyway with no IP, and
its -p bindings were then skipped without a word. ephpm run 39690226239
waited 4 minutes on a postgres port that could never open: the CNI
loopback plugin had returned EINTR ("interrupted system call") from its
netlink calls while other containers were being wired up on ducks.
- networking: retry a CNI ADD that fails with EINTR (3 attempts, CNI DEL
between them) and release a failed ADD's allocations.
- dind: a network setup failure now fails `docker start` with the error.
- dind: port bindings are published before the task starts, and one that
cannot be honoured (no IP, listen failure) fails the start, as
`docker run -p` does, instead of logging a warning.
- portforward: bound the dial to the container (5s) and log a failed dial,
so a client accepted on 127.0.0.1 is closed at once rather than left on
a silent connection for the kernel's ~2 minute connect timeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two Docker-shim bugs reported from ePHPm CI on 2026-10-22.
1. A container with no IP got no port forward, silently
ephpm run 39690226239 (job 119534309464, runner
fast_linuson ducks): the postgres siblingephpm-pg-39690226239-1started withip="". The ducks log shows why:The CNI loopback plugin got EINTR from netlink while the job's two MySQL siblings were being wired up in the same second. The shim logged a warning, started the container anyway, and
installPortForwardsreturned early on the empty IP, so127.0.0.1:26241never opened and the job waited 4 minutes. It happened once in 14 days of ducks logs.Changes:
networking: retry a CNI ADD that fails with "interrupted system call" (3 attempts, CNI DEL between them), and release a failed ADD's allocations. The runner container's setup goes through the same path, so it gets the retry too.dind: a network setup failure failsdocker startwith the error instead of starting a container with no network.dind: port bindings are published beforetask.Start, and a binding that can't be honoured (no IP, listen error) fails the start the waydocker run -pdoes ("port is already allocated"). Anything already set up is torn down, and the container is left increated.2. The forwarder dialed the container with no timeout
forwardConnnow usesnet.DialTimeout(5s), closes the client when the dial fails, and logs the failure with listen and target addresses. The comment claiming the dial runs in the runner's netns is corrected: the per-connection goroutine isn't thread-locked, so it dials from the daemon's namespace.This is hardening, not a confirmed fix for run 39710247641. The shim logs for
stoic_newtonandhappy_raman(both on the desktop's Linux VM) show the MySQL container got an IP (10.88.0.27 / 10.88.0.34) and the forward was installed at start. The job's readiness step got a real MySQL handshake through127.0.0.1:31641at 02:17:53, about 8 minutes beforepdo_mysql_proxy_write_read_roundtripreturned 500 at 02:26:12. A dial to a listening container on the same bridge connects or is refused at once, so the 30s+ hang is more likely inside the PDO → ePHPm[db.mysql]proxy → MySQL path than in the forwarder. The forwarder didn't log per connection before this change, so that can't be ruled out from these logs; with this change a failed dial will show up.Tests
TestRetrySetup_*: EINTR is retried with cleanup; other errors aren't; gives up after 3 attempts.TestInstallPortForwards_NoIPIsAnError/_NothingToPublish.TestForwardConn_DialIsBounded: a dial to TEST-NET-1 gives up and closes the client.go vetfor windows and linux (CGO off);go test ./pkg/dind ./pkg/networkingon Windows; the Linuxpkg/dindandpkg/networkingtest binaries run in WSL, all pass.TestStartPortForwardProxy_OwnNetnsskips without root, as before.