Repository navigation
fix(deploy): pin Cilium 1.20.2, restore airgap images, and harden install.sh - #674
Merged
Merged
Conversation
install.sh picked the gVisor tarball using `tar --zstd --help`, which exits 0 whenever tar knows the flag — even when tar merely execs an external `zstd` it cannot find. On a host with libzstd1 installed but no zstd CLI (a stock Ubuntu 26.04 image, among others) the probe passed and the extract then died: tar (child): zstd: Cannot exec: No such file or directory tar: Child returned status 2 tar(1) accepts --zstd whenever it was built with zstd support, but on most distros it shells out to the zstd binary rather than linking libzstd, so the flag being accepted says nothing about zstd being available. Replace both probes with a real (empty) round-trip. Verified on a host with zstd absent: the old probe exited 0 and chose gvisor.tar.zstd, reproducing the exact failure above, while the new probe exits non-zero and falls through to gvisor.tar.bz2. Also verified the three branches against the real upstream release tarball (checksum OK, runsc release-20261019.0 extracted).
create_env_file forwarded HTTP(S)_PROXY but only copied an existing NO_PROXY, so a shell that exported a proxy without a NO_PROXY left the service proxying everything with no exemption. Go resolves NO_PROXY on every outbound request, so the apiserver dialled its own kubelet through the proxy and every kubectl logs/exec failed: proxyconnect tcp: proxy error from 127.0.0.1:6443 while dialing host.orb.internal:7897, code 502: 502 Bad Gateway OrbStack's guest environment injects http_proxy/https_proxy/all_proxy with no no_proxy, so this reproduces on every OrbStack install. Always append the loopback and in-cluster ranges, merging with any value the caller already set so existing entries are preserved. The proxy is still forwarded, so image pulls and other egress keep working.
k8e-killall.sh only matched containerd-shim, so k8e-uninstall.sh left the Tandem datastore pair running. The unit uses KillMode=process, so `systemctl stop` reaps just the server; rqlited and tandem were reparented to init, kept listening on 127.0.0.1:4001/4002, and survived the `rm -rf /var/lib/k8e` that followed, holding deleted inodes. The next install then bootstraps a fresh data dir while the stale rqlited still owns the raft listeners, and bootstrapping fails hard: start managed database: rqlite data dir is not empty and holds no rqlite state; refusing to rebuild an empty database over it which then loops under Restart=always with the same fatal. Match the helpers by binary name instead of launch path. The old pattern required the datadir copy (`k8e/data/<hash>/bin/...`) to be the live one, but resolveBinary() runs these from $K8E_RQLITED_BINARY, PATH, /usr/local/bin or the data dir, whichever resolves first. Verified against a live cluster: the name-based pattern matches rqlited, tandem, containerd and every containerd-shim (14 processes) with no false positives, while the old one matched zero whenever the binaries had been resolved out of /usr/local/bin.
Cilium 1.20.0/1.20.1 cannot start on kernel v7.2, and the airgap image list had lost every cilium image outright. Kernel v7.2-rc1 added validation that bpf_set_retval's R1 must be a scalar (torvalds/linux b1f7f67b74c2). Cilium's helper probe emits a bare call at the head of the program, where R1 still points at the program context, so 1.20.0/1.20.1 die at startup on any 7.2 kernel — OrbStack included: failed to probe helper ... R1 is not a scalar progType=CGroupSock helper=FnSetRetval Upstream fixed the probe with CO-RE (67c619cb, backported to v1.20 in cilium/cilium#48213) and shipped it in 1.20.2. See cilium/cilium#48016, where Daniel Borkmann confirms "it's a Cilium bug, triggered by additional kernel asserts". Separately, the three quay.io/cilium/* images were dropped from hack/airgap/image-list.txt during the v1.35 bump (a054230) when the default CNI moved away from Cilium, and were never restored after Cilium came back (88c7e68). An airgap install therefore staged a cluster whose CNI could never start. Add them back at 1.20.2 — cilium-envoy, which is on its own independent version line (there is no v1.20.2 of it), is pinned to the tag the 1.20.2 chart references. Add TestCiliumAirgapCoverage/TestCiliumAirgapVersions so the images cannot silently drop out again, and so a downgrade below 1.20.2 fails the suite. Also bump the two remaining pins that still said 1.20.0: VERSION_CILIUM_CHART in hack/version.sh and the Gateway API CRD header comment.
`zig build` clones the CNI plugins into .cni-build/ and `make clean` removes it, but it was never ignored, so 30 MB of vendored source showed up as untracked noise in every git status. Tracked for symmetry with the other build outputs already listed here (.zig-cache, zig-out, bin, dist, build).
|
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four independent fixes found while bringing a fresh OrbStack VM (Ubuntu 26.04,
kernel 7.2.12) online with the revised
contrib/install.sh. Each commit isself-contained; the Cilium one is the only one that changes runtime behaviour
of the shipped product.
1. Cilium 1.20.0/1.20.1 cannot boot on kernel v7.2
A newly installed cluster came up with the node stuck
NotReadyandNetworkPluginNotReady. The agent died at startup:Not an OrbStack problem — it is a Cilium bug exposed by a kernel check.
Kernel v7.2-rc1 added validation that
bpf_set_retval's R1 must be a scalar(torvalds/linux
b1f7f67b74c2), while Cilium's helper probe emits a bare callat the head of the program where R1 still points at the program context.
Reproduced the kernel constraint directly with a raw
bpf(2)probe on thehost: R1 of
0/-1is accepted, any other value returnsEINVAL— which iswhere the misleading "not a scalar" message comes from.
Upstream fixed the probe with CO-RE (
67c619cb, backported to v1.20 incilium/cilium#48213) and shipped
it in 1.20.2. Daniel Borkmann confirms the diagnosis in
cilium/cilium#48016:
"it's a Cilium bug, triggered by additional kernel asserts."
Bumped the pin and verified the resulting cluster: agent
1/1 Running,datapath links loaded, node
Ready, all podsRunning, sandbox warm poolscheduling normally.
Also fixed a silent airgap regression: the three
quay.io/cilium/*imageswere dropped from
hack/airgap/image-list.txtduring the v1.35 bump(
a0542305) when the default CNI moved away from Cilium, and were neverrestored when Cilium came back (
88c7e68e). An airgap install thereforestaged a cluster whose CNI could never start. Added back at 1.20.2, with
cilium-envoypinned to the tag the 1.20.2 chart references (it is on its ownindependent version line — there is no
v1.20.2of it).Added
TestCiliumAirgapCoverage/TestCiliumAirgapVersionsso neither canregress quietly again: the images cannot drop out of the list, and a downgrade
below 1.20.2 fails the suite. Both failure modes were verified by temporarily
pinning 1.20.0 and watching the tests fail.
2.
k8e-uninstall.shleft rqlited/tandem runningReported as: after an install, the next install would not boot —
rqlite data dir is not empty and holds no rqlite state; refusing to rebuild an empty database over it, looping underRestart=always.k8e-killall.shonly matchedcontainerd-shim. The unit usesKillMode=process, sosystemctl stopreaps just the server;rqlitedandtandemwere reparented to init, kept listening on127.0.0.1:4001/4002, andsurvived the
rm -rf /var/lib/k8ethat followed while holding deleted inodes.The fresh bootstrap then could not bind the raft listeners and hard-failed.
Fixed by matching helpers by binary name rather than launch path. The old
pattern required the datadir copy (
k8e/data/<hash>/bin/…) to be the live one,but
resolveBinary()runs these from$K8E_RQLITED_BINARY,PATH,/usr/local/binor the data dir, whichever resolves first — so the old patternmatched zero processes whenever the resolved copy was not the datadir copy.
Verified against a live cluster: the new pattern matches
rqlited,tandem,containerdand every shim (14 processes) with no false positives.3. Proxy leaked into
k8e.service.envwith no exemptioncreate_env_fileforwardedHTTP(S)_PROXYbut only copied an existingNO_PROXY. OrbStack's guest environment injectshttp_proxy/https_proxy/all_proxywith nono_proxy, so every OrbStack install proxiedcluster-local traffic with no exemption and
kubectl logs/execfailed:Now always appends the loopback and in-cluster ranges, merging with any value
the caller already set. The proxy is still forwarded, so egress and image pulls
keep working. Verified both the "no NO_PROXY" and "caller already set NO_PROXY"
cases.
4.
install.shzstd probe accepted a tarball it could not extractReported as
tar (child): zstd: Cannot exec: No such file or directory. ThegVisor tarball was chosen with
tar --zstd --help, which exits 0 whenever tarknows the flag — even when tar merely execs a
zstdbinary that is notinstalled. On any host with
libzstd1but nozstdCLI (stock Ubuntu 26.04included) the probe passed and extraction then failed.
Replaced with a real round-trip probe. Verified on a host with
zstdabsent:old probe exits 0 and picks
gvisor.tar.zstd(reproducing the failure), newprobe exits non-zero and falls through to
gvisor.tar.bz2. All three brancheschecked against the real upstream release tarball.
Testing
go test ./pkg/deploy/...— pass, including the two new Cilium testssh -n contrib/install.sh— passgo vet ./pkg/deploy/ ./pkg/version/— cleank8e-killall.shbody extracted from the heredoc and run against alive cluster to confirm PID selection
Ready, cilium agent/operator/envoy
1/1, cert-manager, coredns, metrics-server, local-path-provisionerall
Running, sandbox warm pool scheduling