fix(snapshot azure): only send Azure credentials to Azure Container Registry - #1153
Conversation
…egistry
snapshot azure took the registry host from each Web App's own
linuxFxVersion and built an authenticated azcontainerregistry client for
it, with no check that the host was an Azure Container Registry. A
registry that answers with an ordinary 401 Bearer challenge makes the SDK
fetch an AAD token for the containerregistry.azure.net audience and POST
it to <host>/oauth2/exchange, so any host named in an app's container
configuration could collect that token.
Anyone with Microsoft.Web/sites/config/write on a scanned resource group
could set that host. The built-in Website Contributor role grants it and
carries no Microsoft.ContainerRegistry actions at all, so this turned a
role with no registry access into one that could read every registry the
snapshot service principal could reach.
Resolution is now layered, and only the ACR path attaches the credential:
- a digest-pinned reference already carries the fingerprint, so it is
read straight from the reference with no registry call
- an Azure Container Registry login server is read as before, with the
Azure credential
- any other host is read through digest.OciSha256 with no credentials
The third case also fixes an existing failure. A non-ACR host previously
errored out (the exchange returns 405 on ghcr.io and 404 on Docker Hub),
and because one app's error cancels the whole snapshot, a single app on a
third-party registry broke the entire environment report. Those apps now
resolve correctly when the image is public, and produce an error naming
the app and host, and pointing at --digests-source logs, when it is not.
Reading the digest from the App Service API instead would avoid the
registry entirely, but no such field exists: digest and sha256 appear
nowhere in the Microsoft.Web/sites specs from 2023-01-01 through
2026-07-15, and siteContainers is configuration only. App Service
re-pulls on restart and scale-out, so instances of one app can run
different digests and no single field could describe them.
Also drops a redundant GetManifestProperties call that re-read the digest
already returned by GetManifest, validates every fingerprint through
digest.ValidateDigest, and adds the parseImageName tests that were absent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @AlexKantor87's task in 4m 59s —— View job Review — round 4
No blocking findings. Everything from round 3 is addressed, and the three things I raised then were each handled in a way that made the code smaller rather than larger. Three minor/follow-up notes posted inline. Static analysis
I could not run Round-3 items, verified
Things I checked that hold up
Notes posted inline
Also, not worth an inline: Tiny nit: the mismatch error at What's good hereThe test work is the strongest part, and it's noticeably stronger than round 3.
The write-ups on each thread continue to correct your own claims where they didn't hold (the |
Review found that the suffix check could be talked into disagreeing with the client about which host it was naming, which reopened the leak this branch exists to close. net.SplitHostPort does not validate the port, so for "myregistry.azurecr.io:443@attacker.example" it returns a host of "myregistry.azurecr.io" and the suffix check passes, while url.Parse reads the same string as userinfo plus a host of attacker.example. The Azure credential therefore went to the attacker. With the parser removed as a mutation the request is visible in the test output: Get "https://myregistry.azurecr.io:***@attacker.example/v2/repo:tag/manifests/tag" parseImageName is replaced by reference.ParseNormalizedNamed, which rejects all three smuggling forms. Classification runs on reference.Domain, and the reference handed to a resolver is the canonical form that was classified, so the two cannot disagree about which registry is contacted. Normalising also removes the need for a special case for references with no registry host: "nginx:latest" becomes docker.io/library/nginx:latest and takes the anonymous arm as an ordinary case. Those references previously errored out and, because one app's error cancels the run, took the whole snapshot with them. digest.OciSha256Anonymous replaces OciSha256 with empty credentials on the anonymous path. Empty credentials leave DockerAuthConfig nil, which makes containers/image fall back to credential discovery, so a runner holding a docker login for a host could present it to a registry named in an app's configuration. Proven with a planted DOCKER_CONFIG: nil sent Basic bGVha2VkdXNlcjpsZWFrZWRwYXNz, a non-nil empty config sent Basic Og==. The digest-pinned short-circuit added earlier on this branch is removed. It skipped the existence check the old code got from GetManifest. In its place GetImageFingerprint now holds the registry to a pinned reference, erroring when the returned digest differs from the pinned one, because docker.GetDigest reports the registry's header without cross-checking it. Also: %w instead of %s so callers can inspect the wrapped error, a shorter error message, trailing-dot hosts trimmed before the suffix check, and a func-var seam so tests assert which resolver was chosen and with what reference. All new tests verified red against the specific bug each one covers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… the reference once Review round 2 found that deriving tagOrDigest by slicing the canonical reference stripped the algorithm prefix from a digest-pinned reference, so ACR was handed a bare hex string, read it as a tag, and returned MANIFEST_UNKNOWN. One app's error cancels the run, so a single digest-pinned ACR app took down the whole environment report. main did not have that bug for digest-only references. Following that up found an unhandled panic on the path this branch opens. ociSha256 ended with: return strings.Split(digest.String(), "sha256:")[1], nil docker.GetDigest returns whatever Docker-Content-Digest says, and go-digest accepts sha384 and sha512, for which Split yields one element and [1] panics. OciSha256Anonymous exists to be pointed at a registry taken from app configuration, so one hostile registry and one response header killed the whole kosli snapshot azure process. Fixed with Algorithm()/Encoded(), which also covers kosli attest artifact. The ACR arm now parses its header the same way instead of trimming a prefix. Reference handling is consolidated into a pure planImageFingerprint returning a fingerprintPlan, so every value crossing the boundary is table-testable without a seam. Normalising once also fixes tag+digest references, which containers/image refuses when a reference carries both: those previously cancelled the snapshot on the anonymous arm. A non-sha256 pin is now rejected at parse time, since ValidateDigest accepts only 64 hex and such a pin can never produce a Kosli fingerprint. The unreachable trailing-dot trim is removed. net.SplitHostPort is kept: reference.Domain can produce IPv6 literals, for which strings.Cut yields "[" and "[2001". Every new guard verified red by mutation, including the panic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…the ACR arm Round 3 review pointed out that hardening the registry-digest parsing had left two independent copies of it, in internal/digest and internal/azure, and that the copy without coverage was the one that could drift. There were in fact three: planImageFingerprint had its own algorithm check for the pin supplied by app configuration. internal/digest now exports Sha256FingerprintFromDigest for a raw wire string and Sha256Fingerprint for an already-parsed digest, so a typed value never has to be turned back into a string to be checked. All three call sites use them, godigest.SHA256 appears once in the codebase, and internal/azure no longer imports go-digest. Dropped the ValidateDigest call in the ACR arm. godigest.Parse already rejects non-hex, uppercase, wrong-length and empty input, so after the algorithm check the value is exactly 64 lowercase hex and that guard could not fail. The coverage half of the finding was measurable: mutating the ACR arm to drop the helper entirely left the suite passing, so its error branches had no coverage. acrImageFingerprint now takes client options, nil in production, and tests pass a transport pointed at a fake TLS registry. That covers sha512, sha384, unparseable and missing-header responses plus the success path, with no package-level state. Both mutations that previously proved nothing now fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…iling app
Round 4 review. Three points, all one-liners, all on the security
boundary or its diagnosability.
fingerprintFromACR was iota, so fingerprintPlan{} claimed the
credential-bearing arm. Nothing reached the dispatch with a zero plan
because err is checked first, but the field that decides whether an AAD
token is sent should fail closed. The order is swapped so an unset or
partially built plan resolves anonymously, and
TestPlanImageFingerprint's require.Equal on the error cases now asserts
something meaningful instead of asserting the rejected plan claims ACR.
One app's error cancels the whole run, and the error did not say which
app. It is now wrapped once in the goroutine that collects it rather
than at each of the seven return sites, so every path is covered by one
line, including the zip and logs paths. The PR description previously
claimed the error named the app; it did not.
Sha256Fingerprint took a godigest.Digest, which is a string type, and
called Algorithm()/Encoded() on it without validating. Handing it
"sha256:hello" returned "hello" with a nil error, and "sha256:" returned
an empty fingerprint. Both callers are validated so this was not live,
but it is exported from the package responsible for fingerprint
integrity. It now calls Validate() first.
The pinned-digest mismatch message reports plan.reference, matching the
arms rather than mixing the configured and canonical forms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An earlier `go mod tidy` in this branch added 274 lines of unrelated /go.mod hashes to go.sum, for modules that appear nowhere in go.mod (cloud.google.com, buf.build, bitbucket.org and others). Running tidy again removes them, so they were an artefact of the module graph state at the time rather than anything this change needs. go.sum is now identical to master. go.mod keeps only the two intended promotions: distribution/reference and opencontainers/go-digest move from indirect to direct, because internal/azure and internal/digest now import them directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A mutation audit of this branch killed 52 of 69 mutants. The survivors included the claim the branch exists to make: neither end of the credential boundary was pinned. OciSha256Anonymous could be rewired to pass a nil DockerAuthConfig, which re-enables credential discovery from auth files and helpers, and no test failed. The test asserted the fields of a helper that nothing had to call. The exported function cannot be driven end to end either, because it forces HTTPS and deliberately does not skip TLS verification, so a better test was not available. So the policy is now a typed choice rather than a caller-supplied SystemContext: ociSha256 takes a credentialSource, and credentialContext turns it into a context. noCredentials is the zero value, so a lookup that fails to state what it wants presents nothing rather than the host's credentials, and "hand it the wrong context" is no longer expressible. credentialContext has a table test covering all five combinations. The two-line delegation in each exported function is still not pinned; that needs a global seam or a trusted-cert registry. On the Azure side, anonymousFingerprint could be swapped for the credential-discovering OciSha256 because every test replaces it. Its production identity is now asserted by pointer. The ACR arm was never driven past credential construction, so a wrong domain reaching the registry client was invisible. It is now resolved end to end against a fake registry via a host-rewriting transport that records the host it was asked for, which also catches a hardcoded one. The pinned-digest cross-check was satisfied by comparing a single character, because the fixtures differed in every position; there is now a near-miss fixture and the check is exercised on both arms. fingerprintDockerService, the only production caller, had no coverage at all: inverting its digests-source condition and replacing the resolver with a constant both survived, and both now fail. Also: imageFingerprintSource and fingerprintPlan.source are deleted. Classification is a pure call on the domain, so storing the answer was derived state, and the fail-closed zero value added last round was guarding a path that cannot occur. Two comments that overstated what the code does are corrected. The manifest response body is closed. The app name is no longer repeated in wrapped errors. Test helpers parse srv.URL rather than trimming a prefix off it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| err = unzip(packagePath, destDir, logger) | ||
| if err != nil { | ||
| return AppData{}, fmt.Errorf("failed to unzip downloaded package for app [%s]: %v", *app.Name, err) | ||
| return AppData{}, fmt.Errorf("failed to unzip the downloaded package: %v", err) |
There was a problem hiding this comment.
Good change — but one sibling on the zip path still names the app, so it now reads twice.
downloadAppPackage (azure_apps.go:225) returns failed to download package for app [%s]: %s, and the goroutine at azure_apps.go:110 wraps every NewAppData error with app [%s]: . A non-200 from Kudu therefore surfaces as:
app [payments-api]: failed to download package for app [payments-api]: 404 Not Found
Same one-line fix as here — drop the app name from the inner message, since the wrap covers every path:
return fmt.Errorf("failed to download the package: %s", resp.Status)Cosmetic only; the wrap itself is the right call and puts the name on the ACR/anonymous errors that previously had none.
| } | ||
|
|
||
| AcrClient, err := azcontainerregistry.NewClient(registryUrl, credentials, nil) | ||
| acrClient, err := azcontainerregistry.NewClient("https://"+plan.domain, credentials, clientOptions) |
There was a problem hiding this comment.
Not introduced here, but this arm is now isolated enough that it's cheap to fix: a fresh ClientSecretCredential is built per image.
NewAzureClient (azure_apps.go:593) already constructs one from the same three fields and then discards it after building the app-service factory. azidentity caches the AAD token on the credential instance, so N ACR-backed apps in a resource group — resolved concurrently in the goroutine loop at azure_apps.go:90 — means N credentials and N independent containerregistry.azure.net token acquisitions, none of which can reuse each other's.
Holding it on the struct alongside AppServiceFactory collapses that to one, and also moves the "bad service principal" failure to client construction rather than to whichever app happens to be resolved first:
type AzureClient struct {
Credentials AzureStaticCredentials
AppServiceFactory *armappservice.ClientFactory
credential azcore.TokenCredential
acrClientOptions *azcontainerregistry.ClientOptions
}Worth checking against TestGetImageFingerprintUsesACRForACRHost, which relies on the empty tenant id failing inside the arm — that assertion would need to move to client construction.
| // | ||
| // This is the rule for the OCI and Azure Container Registry lookups. The older | ||
| // DockerImageSha256 and RemoteDockerImageSha256 paths still parse digests by | ||
| // hand and are not covered by it. |
There was a problem hiding this comment.
Naming the uncovered paths in the doc comment is the right call. One of the two is closer to a live bug than the comment suggests, so it's worth an issue rather than just a note.
RemoteDockerImageSha256 ends with strings.TrimPrefix(digestHeader, "sha256:") (digest.go:420). For a Docker-Content-Digest of sha512:ccc…, TrimPrefix is a no-op and the function returns the literal string "sha512:ccc…" as the fingerprint. Its only production caller — internal/cloudrun/registry.go:62 — guards on hex == "", which a non-empty non-hex string passes, so the value goes to the Kosli API as a fingerprint. Not a panic like the ociSha256 case you fixed, and Artifact Registry won't do it in practice, but it's silently wrong rather than an error.
Sha256FingerprintFromDigest closes it in one line:
return Sha256FingerprintFromDigest(res.Resp.Header.Get("docker-content-digest"))That also picks up the empty-header case, which currently returns ("", nil). Fair to leave for a follow-up given the scope discipline on this branch — just worth filing rather than only commenting.
The problem
kosli snapshot azurereads each Web App's ownlinuxFxVersionand built an authenticatedazcontainerregistryclient for whatever registry it named, with no check that it was an Azure Container Registry. On a 401 Bearer challenge the SDK fetches an AAD token for thecontainerregistry.azure.netaudience and POSTs it asaccess_tokento<host>/oauth2/exchange. So any host named in an app's container configuration could collect that token, and the default--digests-source acrwas enough.Anyone with
Microsoft.Web/sites/config/writeon a scanned resource group can set that host. The built-in Website Contributor role grants it and carries noMicrosoft.ContainerRegistryactions, so this turned a role with no registry access into one that could read every registry the snapshot service principal could reach.--dry-rundid not prevent it, because the token is sent during discovery.Reported privately; tracked internally. Detail here is limited to what is needed to review the fix.
The fix
Only an Azure Container Registry login server gets the Azure credential:
.azurecr.io/.azurecr.cn/.azurecr.us, matched on a whole label)azcontainerregistrydigest.OciSha256AnonymousplanImageFingerprintidentifies the registry withreference.ParseNormalizedNamedrather than splitting the string by hand, and the reference handed to a resolver is the canonical form that was classified, so the two cannot disagree about which host is contacted.github.com/distribution/referenceandgithub.com/opencontainers/go-digestmove from indirect to direct;go.sumis unchanged frommaster.The credential policy is a typed choice, not a caller-supplied
SystemContext.credentialSourcehasnoCredentialsas its zero value, so a lookup that fails to state what it wants presents nothing rather than the host's credentials. This matters because containers/image falls back to credential discovery (~/.docker/config.json,~/.config/containers/auth.json, credential helpers) wheneverDockerAuthConfigis nil, and that fallback must not reach a registry named by an untrusted source.OciSha256's own behaviour is unchanged, so the ECR/Podman discovery path thatkosli fingerprint -t ocirelies on still works.What five review rounds changed, because it is most of the value here
Each of these was a real defect in an earlier version of this branch:
url.Parse:net.SplitHostPortdoes not validate the port, somyregistry.azurecr.io:443@attacker.exampleclassified as ACR while resolving toattacker.example. Fixed by parsing properly. Regression test:TestGetImageFingerprintRejectsSmuggledACRHost.DockerAuthConfignil. Proven with a plantedDOCKER_CONFIG: nil sentBasic bGVha2VkdXNlcjpsZWFrZWRwYXNz, a non-nil empty config sentBasic Og==.strings.Split(d.String(), "sha256:")[1]panics when a registry answers with a sha384/sha512 digest, whichgo-digestaccepts. One hostile registry and one response header killed the whole process. This also affected 17 other commands that reachOciSha256; for them the behaviour changes from panic to a clean error.sha256:, so ACR read a bare hex string as a tag and 404'd, and one app's error cancels the whole run.mainhandled that case.internal/digest, used from three call sites.It also fixes an existing bug
References with no registry host (
DOCKER|nginx:latest,DOCKER|myuser/myimage:tag) previously errored:GetManifest(ctx, "", "")returnsparameter name cannot be empty. Because one app's error callscancel(), a single such app broke the entire environment report. Normalisation makes them ordinary anonymous-arm cases.Private third-party registries still cannot be resolved and produce an error naming the app and host, pointing at
--digests-source logs. Deliberately an error rather than a silent fallback: changing a customer's fingerprint source without being asked is wrong for a compliance tool.Why not read the digest from Azure instead
That would remove the registry call altogether. No such field exists.
digestandsha256appear zero times in theMicrosoft.Web/sitesOpenAPI specs from2023-01-01through2026-07-15, and zero times inarmappservicev2.3.0 or v5.1.0.siteContainersexists from2023-12-01but is configuration only. The reason is structural: App Service re-pulls on every restart and scale-out, so instances of one app can legitimately run different digests.Making
logsthe default was also considered. It removes the leak but is not a sound control: it fails open silently, returning an empty fingerprint with anilerror when noDigest:line is found, and the API returns only the last lines of the log, so the pull line ages out on long-running apps.Testing
Full suite, baselined.
go test ./... -shortfails in four packages (cmd/kosli,internal/digest,internal/gitview,internal/requests) which need the local Kosli server on:8001and registry network access. The identical command on unmodifiedmainproduces an identical failure set. Ininternal/digestthe only failure isTestRemoteDockerImageSha256, which needs a registry on:5001and fails the same way onmain; every test added here passes, including under-race -count=2.make test_integrationwas not run locally because it starts or resets a server stack; CI covers it.Every guard is mutation-verified. Mutations that now fail a test include: reverting the parser to a hand-rolled split;
isACRLoginServerreturningtrue; swapping the dispatch arms; handing the resolverrepoPathinstead of the reference; strippingsha256:from the digest; keeping a tag alongside a digest; accepting a non-sha256 pin; a nilDockerAuthConfig; restoring theSplit(...)[1]panic; swappinganonymousFingerprintfor the credential-discovering resolver; pointing the ACR client at a hardcoded host; swappingrepoPathandtagOrDigest; comparing one character of the pinned digest; skipping the cross-check on the ACR arm; dropping theGetManifesterror; inverting the digests-source condition; and replacing the resolver with a constant.Known residuals, stated rather than implied
OciSha256Anonymousis not pinned. Making it ask forcallerOrHostCredentialsstill passes. Pinning it needs a global seam or a trusted-cert registry; the typed source makes the fail-open-by-omission direction impossible, which is the part that matters.isACRLoginServeraccepts any ACR, not only one the customer owns. ACR names are globally registrable, so an attacker can name their own*.azurecr.io. They do not receive the token, because Microsoft operates that endpoint, but they can make the victim's snapshot authenticate to their registry and serve arbitrary manifests. The stronger fix is cross-checking against the subscription's registries via ARM, which needs a new permission; raised separately.Acceptheaders, so for a multi-arch image the ACR arm may report a platform manifest where the anonymous arm reports an index. Pre-existing asymmetry, now visible because there are two arms.AppData.DigestsSourcestill reportsacrfor anonymously-resolved digests. The server validates the enum (models/environments.py:370,schemas/azure_web_app_snapshot.json:35), so a new value needs a server change deployed first.Pre-existing issues found while testing, not fixed here
Raising separately rather than widening a security fix:
DockerImageSha256has the sameSplit(..., "@sha256:")[1]panic shape;RemoteDockerImageSha256fails open on a missing digest header;extractImageFingerprintAndStartedTimestampFromLogspanics on a shortDigest: sha256:line;NewAppDatadereferencesSiteConfig.LinuxFxVersionwithout a nil check; and one app's error cancelling the whole snapshot deserves an abort-vs-partial decision of its own.Checklist
cmd/kosli/testdata/output/docs/mintlify/goldens do not coversnapshot azure.🤖 Generated with Claude Code