Skip to content

Build only the MCP images a change can affect, instead of all four every time - #56

Merged
bowen-bd merged 1 commit into
learningmatter-mit:mainfrom
bowen-bd:ci/skip-unaffected-images
Oct 27, 2026
Merged

bowen-bd merged 1 commit into
learningmatter-mit:mainfrom
bowen-bd:ci/skip-unaffected-images

Conversation

@bowen-bd

Copy link
Copy Markdown
Collaborator

Summary

build (generative, linux/arm64) takes about two hours, every time — 115.7 min and 115.5 min on the last two main builds, against at most 33 min for any other image. On aarch64 there are no PyG wheels, so docker/Dockerfile.cuda compiles the extensions from source with CUDA. The GHA layer cache is already configured (cache-from/to: type=gha, mode=max); the cost is not a missing cache but a lock change invalidating the layer that does the compiling.

The cost was also being paid for changes that cannot possibly affect that image. The workflow's paths: filter is per-workflow, so any venv/*/uv.lock matched it. Two open pull requests bump venv/mlip/uv.lock and venv/fairchem/uv.lock, and both rebuild the generative image — twice — for a dependency it does not install.

This narrows the matrix by diffing the change. An image installs its own uv projects from their locks, so only a change to those locks can alter it.

Measured

leg main build 1 main build 2
generative arm64 115.7 min 115.5 min
generative amd64 23.3 32.6
mlip amd64 21.0 13.7
fairchem amd64 15.8 15.4
the other four 7.4 – 11.9 7.5 – 15.8

Changes

  • docker/render.py matrix --changed PATH (repeatable, or - for stdin) narrows the matrix to the images those paths can affect. select_images() attributes venv/<project>/{uv.lock,pyproject.toml} to the image that installs that project; a change to venv/adit/uv.lock selects generative, and to venv/mlip/uv.lock selects mlip.
  • build-images.yml gains a step that diffs against the base — origin/<base_ref> for a pull request, github.event.before for a push — and feeds the list to the matrix. workflow_dispatch with an explicit image list still wins over the diff.

Safety

The question is whether skipping an image can let a broken image through. The rule is built on one asymmetry: building too much only costs time, while building too little ships an image nobody checked.

So the filter narrows only when every changed path belongs to exactly one image:

Changed path Images built
venv/mlip/uv.lock mlip
venv/adit/uv.lock generative
docker/Dockerfile (or .cuda, images.json) all — every image's build recipe
src/**, pyproject.toml, anything else all — images install the repository itself
one attributable path plus one that is not all
empty diff (new branch, forced push, tag) all
base commit unavailable all

Each of those rows is a test in tests/test_images_and_manifests.py (TestMatrixSelection), along with the case that motivated the rule — a change to the mlip and fairchem locks must not select the generative image.

What is genuinely given up: nothing that the previous behaviour caught. The only inputs shared across images are the build recipes and the repository sources, and both select the whole matrix. A change restricted to one project's lock has never been able to alter a different image's contents.

Verification

  • tests/test_images_and_manifests.py: 34 passed, including eight new selection tests and a subprocess check that matrix --changed emits exactly the selected images.
  • docker/render.py servers --check, plugin-mcp --check and tools/sync_version.py --check remain current; ruff is clean.
  • On this repository's own open pull requests the matrix drops from 8 entries (4 images × 2 platforms) to 4, which is what removes the two-hour leg from them.

Note for this pull request's own CI

It changes .github/workflows/build-images.yml, which the filter treats as shared, so this run still builds every image once — about two hours, as before. That is the intended behaviour (a build-recipe change must rebuild everything) and doubles as a check that the unfiltered path still works.

The generative image compiles the PyG extensions from source with CUDA on
arm64, because PyG publishes no aarch64 wheels there. It takes about two
hours, every time: 115.7 and 115.5 minutes on the last two main builds, while
no other image exceeds 33.

It ran on every pull request anyway, because the workflow's path filter is
per-workflow, not per-image: any venv/*/uv.lock, including venv/mlip's,
matched. Two open pull requests that bump mlip and fairchem locks would
rebuild it twice for a dependency it cannot see.

An image installs its own uv projects from their locks, so diff the change and
drop the images it cannot affect. docker/render.py matrix --changed takes the
paths, attributes venv/<project>/{uv.lock,pyproject.toml} to the image that
installs that project, and keeps everything when any path is not attributable
to exactly one image — docker/, the workflow, and the sources the images copy
in all select the whole matrix. An empty or unavailable diff also builds
everything, so a new branch or a forced push stays on the safe side.

Building too much only costs time; building too little ships an image nobody
checked. That asymmetry is why the default is everything and why the tests pin
each unattributable case.
@bowen-bd
bowen-bd force-pushed the ci/skip-unaffected-images branch from 12f57b0 to 2f1fee6 Compare October 27, 2026 02:56
@bowen-bd
bowen-bd merged commit 66f26b7 into learningmatter-mit:main Oct 27, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant