Pin the autoscaler provider to an immutable tag — :latest is why a built, merged fix shipped nothing

[izzywdev]

helm/fuzeinfra/values-contabo.yaml pins the autoscaler provider to a floating tag:

provider:
  # TODO: replace with the published image once fuzeinfra-contabo-ca-provider
  # has a tagged release (see cluster-autoscaler/contabo-externalgrpc).
  image: ghcr.io/izzywdev/fuzeinfra-contabo-ca-provider:latest
  # Bump with the provider image build SHA so Argo rolls the :latest pod.
  imageRollout: "canceldate-dateonly-fix-2026-09-02"

Why it is time

:latest means Kubernetes sees no change when the image is rebuilt, so a correct, merged, successfully-built fix does not deploy. It only rolls when a human remembers to change an unrelated annotation.

This bit us today, twice in one chain:

1. fix(ca): unwedge the Contabo elastic pool — attribute unregistered instances + stop cancelled ones holding slots #831 merged; the fix was right. It did not deploy for ~45 minutes — an unrelated Argo sync was stuck, and nothing about :latest made that visible.
2. fix(ca): parse Contabo's date-only cancelDate — cancelled instances were holding every elastic slot #832 (cancelDate parse) merged and ca-provider-image built successfully at 06:19:58Z. The pod was still 56 minutes old and still logging unparsable cancelDate. It took chore(ca): roll the provider pod onto the cancelDate fix #834, a one-line imageRollout bump, to actually ship it.

The failure mode is the expensive one: a deploy that looks complete and ships nothing. Same shape as the other four we hit today — the Argo sync that reported success 2s after firing, packages-publish green without building two packages, the provisioner exiting 0 having written nothing, and governance-sync announcing a push the server had rejected.

There is a real cost attached, not just tidiness: while the old binary was live, the autoscaler could not distinguish cancelled instances from live ones, so the elastic group read as full at MAX_SIZE=4 holding one real node, and ~27 runner pods stayed Pending.

What "done" looks like

1. ca-provider-image.yml publishes an immutable tag per build (commit SHA and/or semver), alongside :latest if you want to keep it for humans.
2. values-contabo.yaml pins that immutable tag.
3. Deploying = bumping the tag in Git — one change, reviewable, and the diff shows which build is going to prod.
4. imageRollout can then be deleted. Its only purpose is to force a roll when the tag cannot change; with a real tag the tag itself is the trigger.
5. Optional: a CI check that the pinned tag exists in GHCR before merge — a typo'd tag currently fails at pull time in prod, not at review time.

Note

imageRollout is a pod-template annotation (checksum/provider-image) whose only job is to differ. It is never resolved against anything, so its value is a label for humans — which is why a stale-looking value there is harmless today, and why it stops being needed at all once the image tag is real.