Skip to content

packages: add containerd-2.3 - #1023

Merged
qianxjcraig merged 1 commit into
bottlerocket-os:developfrom
qianxjcraig:containerd-2.3
Sep 2, 2026
Merged

packages: add containerd-2.3#1023
qianxjcraig merged 1 commit into
bottlerocket-os:developfrom
qianxjcraig:containerd-2.3

Conversation

@qianxjcraig

@qianxjcraig qianxjcraig commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Description of changes:

Introduces containerd 2.3.4 as a new package. 2.3 is the first LTS in the 2.x line (supported to ~April 2028); containerd 2.2 goes EOL 2026-11-06, about a month after EKS 1.37 GA so the k8s-1.37 variants can't ship on it. No variant adopts it in this PR.

Upstream changelogs:

Patch changes: none. All three patches from containerd-2.2 apply cleanly to 2.3.4 and carry forward unchanged.

Other changes: added export GO_MAJOR="1.26" to %build in the spec — containerd 2.3's go.mod declares go 1.26.3, newer than the SDK default.

Testing done

1. Binary versions (verified on a running node):

bash-5.2# containerd --version
containerd github.com/containerd/containerd/v2 2.3.4+bottlerocket db8809540e1a7a9da5d518876894933ff55692ab
bash-5.2# ctr --version
ctr github.com/containerd/containerd/v2 2.3.4+bottlerocket

2. Conformance testing: 26/26 variant/arch combos passing.

Phase Arch Instance Variants Tested Result
A x86_64 m5.xlarge aws-k8s-1.33, 1.33-fips, 1.34, 1.34-fips, 1.35, 1.35-fips, 1.36, 1.36-fips ✅ All passing
B aarch64 m6g.xlarge aws-k8s-1.33, 1.33-fips, 1.34, 1.34-fips, 1.35, 1.35-fips, 1.36, 1.36-fips ✅ All passing
C x86_64 g5.xlarge / g4dn.xlarge aws-k8s-1.33-nvidia, 1.34-nvidia, 1.35-nvidia, 1.36-nvidia, 1.36-nvidia-fips ✅ All passing
D aarch64 g5g.2xlarge aws-k8s-1.33-nvidia, 1.34-nvidia, 1.35-nvidia, 1.36-nvidia, 1.36-nvidia-fips ✅ All passing
Full conformance results per variant/arch
Variant Arch K8s Version Tests Instance
aws-k8s-1.33 x86_64 1.33 421/421 m5.xlarge
aws-k8s-1.33-fips x86_64 1.33 421/421 m5.xlarge
aws-k8s-1.33-nvidia x86_64 1.33 421/421 g5.xlarge
aws-k8s-1.34 x86_64 1.34 426/426 m5.xlarge
aws-k8s-1.34-fips x86_64 1.34 426/426 m5.xlarge
aws-k8s-1.34-nvidia x86_64 1.34 426/426 g4dn.xlarge
aws-k8s-1.35 x86_64 1.35 444/444 m5.xlarge
aws-k8s-1.35-fips x86_64 1.35 444/444 m5.xlarge
aws-k8s-1.35-nvidia x86_64 1.35 444/444 g4dn.xlarge
aws-k8s-1.36 x86_64 1.36 451/451 m5.xlarge
aws-k8s-1.36-fips x86_64 1.36 451/451 m5.xlarge
aws-k8s-1.36-nvidia x86_64 1.36 451/451 g5.xlarge
aws-k8s-1.36-nvidia-fips x86_64 1.36 451/451 g4dn.xlarge
aws-k8s-1.33 aarch64 1.33 421/421 m6g.xlarge
aws-k8s-1.33-fips aarch64 1.33 421/421 m6g.xlarge
aws-k8s-1.33-nvidia aarch64 1.33 421/421 g5g.2xlarge
aws-k8s-1.34 aarch64 1.34 426/426 m6g.xlarge
aws-k8s-1.34-fips aarch64 1.34 426/426 m6g.xlarge
aws-k8s-1.34-nvidia aarch64 1.34 426/426 g5g.2xlarge
aws-k8s-1.35 aarch64 1.35 444/444 m6g.xlarge
aws-k8s-1.35-fips aarch64 1.35 444/444 m6g.xlarge
aws-k8s-1.35-nvidia aarch64 1.35 444/444 g5g.2xlarge
aws-k8s-1.36 aarch64 1.36 451/451 m6g.xlarge
aws-k8s-1.36-fips aarch64 1.36 451/451 m6g.xlarge
aws-k8s-1.36-nvidia aarch64 1.36 451/451 g5g.2xlarge
aws-k8s-1.36-nvidia-fips aarch64 1.36 451/451 g5g.2xlarge

A version-skew run (aws-k8s-1.36 nodes against a 1.37 beta control plane, conformance:v1.37.0-beta.0) also passed with 0 failures.

3. Seccomp defaults (containerd#13409):
containerd 2.3.1 hardened the default seccomp socket policy to block AF_ALG, and Bottlerocket's patch 1001 re-allows io_uring in that same code — so the two interact. We ran three probes on a 2.3.4 node and confirmed AF_ALG is blocked under RuntimeDefault (errno=1, EPERM), works under Unconfined (proving the block is seccomp-only, not kernel), and that io_uring_setup(2) still succeeds under RuntimeDefault (fd=3) — patch 1001 survives the rebase. Note this makes AF_ALG unavailable to RuntimeDefault workloads on 2.3+; upstream behavior, no BR patch re-allows it.

Probe results
t1a  AF_ALG, seccompProfile=RuntimeDefault  -> AF_ALG=BLOCKED errno=1
t1b  AF_ALG, seccompProfile=Unconfined      -> AF_ALG=OK
t1c  io_uring_setup(2), RuntimeDefault      -> IO_URING=OK fd=3

4. Sandbox image resolution (containerd#13759): containerd 2.3.4 normalizes sandbox image references, and Bottlerocket pins sandbox = "localhost/kubernetes/pause:0.1.0", so the pinned local reference could have been rewritten. Confirmed it still resolves — the io.cri-containerd.pinned label shows CRI recognized it as the configured sandbox image, and journalctl -u containerd shows only local ImageCreate/ImageUpdate events with no registry pull attempts.

bash-5.2# ctr -n k8s.io images ls | grep pause
localhost/kubernetes/pause:0.1.0  ... io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned

5. CRI stats (containerd#12629):
containerd 2.3.0 added a background collector that computes UsageNanoCores for the kubelet Summary API (relates to Kubernetes KEP-2371). Verified on a cgroup v2 node under a stress --cpu 1 --vm 1 --vm-bytes 200M pod that the field is populated with a plausible rate.

pod  cpu usageNanoCores : 1999589326   (~2 cores; --cpu 1 --vm 1 runs two busy workers)
pod  mem workingSetBytes:  147410944   (~140 MiB resident of the 200 MiB allocation)
node cpu usageNanoCores : 2033703578

6. Shim lifecycle across a containerd restart (#12786 bootstrap protocol, #13803 start-response corruption, #13857 orphaned shims):
containerd 2.3.0 replaced the shim bootstrap mechanism and 2.3.4 carries two fixes to it, so shim re-adoption is new code on this path. Conformance never restarts containerd, so we tested it directly: with a single-container pod and a two-container pod pinned to one node, we restarted containerd.service and confirmed the shims survived and were re-adopted rather than recreated.

Note two Bottlerocket specifics: containerd.service sets KillMode=process, so systemd leaves the shim children running — that is the mechanism being exercised. And kubelet.service sets BindsTo=containerd.service, so kubelet restarts alongside containerd and the node is briefly NotReady; that is expected, not a regression.

Before / after on the node and from the cluster

containerd was restarted (MainPID 18443949). All six shim PIDs are unchanged and their elapsed times grew continuously across the restart, i.e. the processes were never respawned:

PID    namespace / id            etimes BEFORE -> AFTER
1985   k8s.io  d50a1f98...            135 -> 145
1988   k8s.io  56ad3439...            135 -> 145
2365   default control                128 -> 139     (host-containerd, unaffected)
2427   default admin                  128 -> 138     (host-containerd, unaffected)
3172   k8s.io  c4c4957f...             88 ->  99
3179   k8s.io  3b434f42...             88 ->  99
shim count: 6 -> 6   (no orphans, none lost)

Container identity and restart counts unchanged, and the marker file written at container start still holds its original value — proving the same container filesystems, not recreated ones:

                 BEFORE                          AFTER
shim-a  app      restarts=0 id=7643f9d9d3d84bbe  restarts=0 id=7643f9d9d3d84bbe
shim-b  c1       restarts=0 id=89e322964c86a1f6  restarts=0 id=89e322964c86a1f6
shim-b  c2       restarts=0 id=d68963704071bb96  restarts=0 id=d68963704071bb96
/tmp/marker      original-1787769572             original-1787769572

journalctl -u containerd showed no load failures or protocol errors, and a pod created after the restart ran successfully (post-restart-ok), confirming containerd was left fully functional.

7. SOCI snapshotter (#13071):
containerd 2.3.0 began passing two additional labels to snapshotters when unpacking image layers —containerd.io/snapshot/diff-id and containerd.io/snapshot/parent-chain-id. Bottlerocket ships soci-snapshotter 0.15.0 as an external proxy-snapshotter plugin (separate process over gRPC), so that label contract is a cross-process interface which the compiler cannot check. SOCI is also opt-in (settings.container-runtime.snapshotter = "soci", default overlayfs), so none of the tests above exercise it, and its failure mode is a silent fallback to overlayfs where pods still run and lazy loading is simply gone. Tested explicitly on a node booted with the SOCI setting.

SOCI verification

Setting applied and plugin healthy:

bash-5.2# cat /etc/containerd/selected-snapshotter
SELECTED_SNAPSHOTTER="soci"
bash-5.2# systemctl is-active soci-snapshotter
active
bash-5.2# ctr plugins ls | grep soci
io.containerd.snapshotter.v1    soci    linux/amd64    ok

The decisive check — SOCI served every snapshot and overlayfs served none, ruling out silent fallback:

bash-5.2# ctr -n k8s.io snapshot --snapshotter soci ls
KEY                                                                PARENT                     KIND
1041125a31414aee2d5b8c28a1198640853bdc5de8783fbfd396e741c4285205   sha256:fd8b371c477c...     Active
4d2d28a089bcafe987570b65796174b876703215075bdc745702d2c7a75012d0   sha256:1a4ca3b283ff...     Active
sha256:1a4ca3b283ff4b5a8b456ffe0d24e9f989457d36fcb7c18861ca0cb757dae144                       Committed
... (many more)

bash-5.2# ctr -n k8s.io snapshot --snapshotter overlayfs ls
KEY PARENT KIND        <-- empty

The test pod's sandbox is traceable to a SOCI-managed snapshot — containerd logged RunPodSandbox for name:"soci-a" ... returns sandbox id "4d2d28a089bcafe98...", and that same ID appears as an Active snapshot in the soci list above.

Neither journalctl -u soci-snapshotter nor journalctl -u containerd reported any snapshot/label errors; containerd logged Get image filesystem path "/var/lib/soci-snapshotter" for snapshotter "soci".

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

Signed-off-by: Craig Qian <qianxj@amazon.com>
@qianxjcraig
qianxjcraig merged commit b62359a into bottlerocket-os:develop Sep 2, 2026
7 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants