Skip to content

feat: harden dockerfiles - #1474

Merged
tt-cll merged 4 commits into
mainfrom
tt/images
Oct 5, 2026
Merged

tt-cll merged 4 commits into
mainfrom
tt/images

Conversation

@tt-cll

@tt-cll tt-cll commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Description

Hardens all production images: digest-pinned bases and distroless final stages

Images

  • All six production images (aggregator, executor, indexer, pricer, verifier, devenv fakes) pin their base images by digest and run their final stages from gcr.io/distroless/static-debian12:nonroot as the distroless nonroot user (65532:65532): no shell, no apk, no apk upgrade security-patch surface. The distroless base ships the CA certificates the services need.
  • The verifier keeps tini as PID 1, now statically linked and copied from the builder: namespace init ignores SIGSTOP, so the service must stay a child process for quiescing to reach it.
  • The aggregator stages its /app/migrations symlink in the builder

ccv quiesce pause|resume (cli/quiesce)

  • The final images have no pkill, which the e2e suite (chain-status, policy-hook, jobqueue, replay, reorg, chaos) and the staging runbook used to freeze the verifier during CLI mutations and curse replay. The new subcommand finds the verifier service by scanning /proc for the lowest non-self PID whose comm matches the CLI binary's own name, then sends SIGSTOP/SIGCONT.
  • verifiercli.Pause/Resume (plus the best-effort variants) now exec ccv quiesce ... instead of pkill -STOP/-CONT -f verifier; WithProcessMatch/DefaultProcessMatch are gone

Build context and updates

  • .dockerignore: VCS dirs, .build/, docs/, changelog/, local keystores (**/keystore.json), and .env* files never enter a build context; build/devenv/* is excluded except the fakes module, which the fakes image builds.
  • .github/dependabot.yml: weekly digest bumps for the pinned bases across all six Dockerfiles, grouped into a single PR.

Runbook (docs/migration/staging-migration-plan.md): in-container wget probes replaced with kubectl port-forward + curl, and pkill -STOP -f verifier replaced with ccv quiesce pause.

Breaking for anything that exec-ed a shell in these images, referenced the per-service users, or probed them with wget from inside the container. See the changelog entry for the migration steps.

Testing

  • Full CI matrix on this PR drives verifiercli.Pause/Resume through ccv quiesce against the distroless verifier image: chain-status, policy-hook, jobqueue, replay, reorg, and chaos suites
  • cli/quiesce unit tests: lowest-PID selection, self-exclusion, no-match error, and a kill seam asserting the target PID and signal (tests never signal their own process).
  • Built the devenv fakes image end-to-end locally from the new Dockerfile verifies the .dockerignore negation keeps the fakes module files in the context, the pinned digests resolve to the right images, and the distroless final stage runs.

Checklist

@tt-cll
tt-cll marked this pull request as ready for review September 25, 2026 17:30
@tt-cll
tt-cll requested review from a team as code owners September 25, 2026 17:30
Copilot AI lite review requested due to automatic review settings September 25, 2026 17:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

Only minor nit-level comment and changelog-link cleanups remain; no blocking issues were identified.

Review effort: Lite
Findings: 1 Low severity

Open (1)
What changed in this PR

Hardens production Docker images with digest-pinned distroless bases and replaces shell-based verifier pausing with ccv quiesce.

Changes:

  • Migrates six images to nonroot distroless runtime stages.
  • Adds tested process discovery and SIGSTOP/SIGCONT quiescing.
  • Updates E2E tooling, runbooks, changelog, Docker contexts, and Dependabot configuration.
File Summary
verifier/​Dockerfile Distroless verifier image with static tini.
pricer/​Dockerfile Distroless nonroot pricer image.
indexer/​Dockerfile Distroless nonroot indexer image.
executor/​Dockerfile Distroless nonroot executor image.
docs/​migration/​staging-migration-plan.md Updates probes and quiesce instructions.
cmd/​verifier/​run_ccv_cli.go Registers quiesce commands.
cli/​quiesce/​quiesce.go Implements process discovery and signaling.
cli/​quiesce/​quiesce_test.go Tests quiesce behavior and PID selection.
changelog/​2026-09-25_distroless_images_and_quiesce.md Documents migration and compatibility changes.
build/​devenv/​tests/​e2e/​verifiercli/​committee.go Updates resume helper documentation.
build/​devenv/​tests/​e2e/​verifiercli/​client.go Uses quiesce instead of pkill.
build/​devenv/​fakes/​Dockerfile Adds a distroless fakes image.
aggregator/​Dockerfile Adds a distroless image and staged migration symlink.
.github/​dependabot.yml Configures grouped weekly Docker digest updates.
.dockerignore Excludes unnecessary and sensitive build-context files.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread changelog/2026-09-25_distroless_images_and_quiesce.md Outdated
carte7000
carte7000 previously approved these changes Sep 25, 2026
@tt-cll
tt-cll enabled auto-merge September 28, 2026 11:00
Comment thread aggregator/Dockerfile
@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1.7
FROM golang:1.26.6-alpine AS builder
FROM golang:1.26.6-alpine@sha256:3889b425f035be855a72fb4755265311293b6d414521f0a519d819df32222d83 AS builder

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you drop a comment why this hash?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

// A best-effort helper for cleanup paths is ResumeBestEffort.
func (c *Client) Resume(ctx context.Context) error {
_, err := c.Exec(ctx, "pkill", "-CONT", "-f", c.processMatch)
_, err := c.CLI(ctx, []string{"ccv", "quiesce"}, "resume")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if there is no shell, how this works?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker exec runs the static verifier binary directly (/bin/verifier ccv quiesce pause). The binary scans /proc itself from Go (os.ReadDir/os.ReadFile in cli/quiesce/quiesce.go), finds the service PID by comm, and sends SIGSTOP with syscall.Kill. pkill/sh were only ever a workaround for not having a signaling path in the binary.

bukata-sa
bukata-sa previously approved these changes Oct 2, 2026
@tt-cll
tt-cll dismissed stale reviews from bukata-sa and carte7000 via 23be759 October 2, 2026 16:24
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

Code coverage report:

Package main tt/images Diff
github.com/smartcontractkit/chainlink-ccv/aggregator 47.86% 47.86% +0.00%
github.com/smartcontractkit/chainlink-ccv/bootstrap 65.67% 65.67% +0.00%
github.com/smartcontractkit/chainlink-ccv/cli 54.58% 54.90% +0.32%
github.com/smartcontractkit/chainlink-ccv/cmd 40.85% 40.85% +0.00%
github.com/smartcontractkit/chainlink-ccv/common 53.60% 53.60% +0.00%
github.com/smartcontractkit/chainlink-ccv/executor 42.14% 42.14% +0.00%
github.com/smartcontractkit/chainlink-ccv/indexer 35.39% 35.39% +0.00%
github.com/smartcontractkit/chainlink-ccv/integration 62.55% 62.55% +0.00%
github.com/smartcontractkit/chainlink-ccv/internal 0.00% 0.00% +0.00%
github.com/smartcontractkit/chainlink-ccv/migration 78.70% 78.70% +0.00%
github.com/smartcontractkit/chainlink-ccv/pkg 84.62% 84.62% +0.00%
github.com/smartcontractkit/chainlink-ccv/pricer 0.00% 0.00% +0.00%
github.com/smartcontractkit/chainlink-ccv/protocol 67.04% 67.04% +0.00%
github.com/smartcontractkit/chainlink-ccv/tools 39.19% 39.19% +0.00%
github.com/smartcontractkit/chainlink-ccv/verifier 34.55% 34.55% +0.00%
Total 50.60% 50.60% +0.00%

Files added (in tt/images):

  • github.com/smartcontractkit/chainlink-ccv/cli/quiesce/quiesce.go

Comment thread .github/dependabot.yml
@@ -0,0 +1,19 @@
version: 2

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this hav to use the GATI github token, o/w it won't trigger CI? Or is dependabot a special thing that doesn't need it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's provided by GH itself so shouldn't need GATI because there are no cross-repo calls. We'll find out 🙂

@tt-cll
tt-cll added this pull request to the merge queue Oct 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026
@tt-cll
tt-cll added this pull request to the merge queue Oct 5, 2026
Merged via the queue into main with commit f6daab8 Oct 5, 2026
57 checks passed
@tt-cll
tt-cll deleted the tt/images branch October 5, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants