ci: containerize CI pipeline with pre-built Docker image

[Brendan Walsh (BrendanWalsh)]

⛔ Prerequisite — this PR cannot merge as-is (unrelated to code quality)

pipeline.yaml references an Azure DevOps Docker Registry service connection named SynapseML MCR (top-level resources.containers, and again in the BuildCIImage job's Docker@2 login step). This service connection does not exist in the ADO project. Resource validation runs before any job condition is evaluated, so this fails the entire pipeline instantly, before a single job starts:

The pipeline is not valid. A service connection with name "SynapseML MCR" could not be found.

Confirmed on two real queued builds against this branch (230951450, 230953655) — both failed in ~0 timeline records. 9 jobs use container: ci, so every PR build and every master build will fail immediately until this is fixed.

Before merging, someone with ADO project-admin rights must:

1. Create a Docker Registry service connection named exactly SynapseML MCR, pointing at mmlsparkmcr.azurecr.io.
2. Grant it pipeline permissions for microsoft.SynapseML.
3. Confirm the pinned image synapseml/ci:ci-cc980b65de98 actually exists in that registry.

Summary

Containerizes the SynapseML CI pipeline using a pre-built Docker image so jobs start from a warm environment (JDK, conda env, SBT + resolved dependencies, Spark, test datasets) instead of installing everything on every run.

Metric (ADO Pipeline 17563, pre-rebase reference build)	Master (8-build avg)	Container (2 green builds)	Improvement
Wall clock	66.3m	26.5–27.5m	59% faster
Total compute	~1228 agent-min	~817 agent-min	33.5% savings
UnitTests avg	18.0m/job	13.1m/job	-4.9m/job
PythonTests avg	28.4m/job	19.6m/job	-8.8m/job

⚠️ These numbers are from an earlier revision, before the rebase/fixes below and before the SynapseML MCR blocker was discovered. No build has completed successfully against the current head — treat them as directional, not a guarantee.

What changed, by file

• tools/docker/ci/Dockerfile (new): Ubuntu 22.04, Temurin JDK 8, pinned Miniconda, SBT 1.10.11, Spark 3.5.0/hadoop3, pre-warmed SBT/Ivy caches, and a pre-fetched test-dataset tarball. HTTPS + SHA256-verified libssl1.1. Published to mmlsparkmcr.azurecr.io/synapseml/ci.
• .dockerignore: rewritten from a denylist to an allowlist — only environment.yml, project/, build.sbt, sonatype.sbt, tools/docker/demo/init_notebook.py, and docs/ are sent to the Docker build context.
• pipeline.yaml (bulk of the diff):
  • New resources.containers entry (container: ci) and new BuildCIImage job that builds/pushes the image only on trusted, non-PR master builds, with content-hash tagging and a hash-vs-CI_IMAGE_TAG guard.
  • container: ci added to all 9 containerizable jobs: Style, Publish, DatabricksCPUE2E, DatabricksGPUE2E, FabricE2E, PythonTests, RTests, WebsiteSamplesTests, UnitTests. (BuildDocker, ReleaseBranchCompat — needs JDK 17 — and BuildAndCacheSbt intentionally stay on the host.)
  • Because BuildCIImage is skipped on PR builds by design, every dependent job's condition changed from succeeded() to and(not(failed()), not(canceled()), ...) — otherwise a skipped BuildCIImage would skip all 9 downstream jobs on every PR.
  • PROJECT variable added to the UnitTests matrix so each leg compiles only its own module; PythonTests' install step now builds core then $(PACKAGE) explicitly.
  • templates/sbt_cache.yml / update_cli.yml / conda.yml replaced by templates/free_disk.yml (runs on the host via target: host even inside container jobs, to reclaim real VM disk).
  • SBT_OPTS tuned (-Xmx4G, -Dscala.concurrent.context.numThreads=8/maxThreads=8) and UnitTests timeout raised 80m → 100m.
• build.sbt: getDatasetsTask now checks $DATASET_CACHE/datasets-2023-04-03.tgz (baked into the image) before downloading from blob storage.
• core/.../test/base/TestBase.scala: adds a global 10-minute per-test timeout (failAfter, overridable via testTimeoutInSeconds), and declares implicit protected val testSignaler: Signaler = ThreadSignaler. This second part is the important one: ScalaTest's default DoNotSignal only reports a timeout after the test body returns on its own, so a genuinely hung test would never actually be interrupted. Verified empirically with a probe suite (3s limit, 12s sleeping body): 12001 ms before ThreadSignaler → 3004 ms after. Scoped only to TestBase; does not touch the separate, pre-existing TimeLimitedFlaky / testFun _ eta-expansion bug in the same file (that bug silently skips the test body in PartitionConsolidatorSuite and is tracked as its own fix).
• project/CodegenPlugin.scala: drops a redundant LocalRootProject / Compile / publishLocal from installPipPackage.
• templates/free_disk.yml (new), templates/databricks_e2e_steps.yml, .gitignore (adds pipeline.yaml.bak): supporting template/hygiene changes for the above.

Fixes applied since this PR was opened (rebase onto master 7be2767)

#	Issue found	Fix
A	BuildCIImage built and pushed on every PR build, including forks	Gated to and(succeeded(), isMaster, !isPR)
B	Jobs consumed the mutable ci-latest tag	Jobs now pin the digest tag ci-cc980b65de98, with a hash guard in BuildCIImage
C	UnitTests job timeout equaled its step timeout (couldn't distinguish job vs. step timeout)	Job timeout raised to 100 min, 90-min step timeout kept
D	New global per-test timeout couldn't actually interrupt a hung test	Added ThreadSignaler (see TestBase.scala above)

Known issues / residual risk in this diff

• Duplicated hardcoded tag: ci-cc980b65de98 is a literal in two places — resources.containers[0].image and variables.CI_IMAGE_TAG — with no automated link between them. BuildCIImage's hash guard only checks the variable, and that job never runs on PRs, so a PR that bumps tools/docker/ci/Dockerfile or its dependencies without manually updating both literals will pass PR CI and can silently desync on merge.
• resources.repositories: - repository: self / type: self: type: self isn't a documented value (git | github | githubenterprise | bitbucket). ADO appears to tolerate it in practice, but it's non-standard and unnecessary — self doesn't need to be declared under resources.repositories at all.
• RTests still downloads Spark at runtime (wget -q .../spark-3.5.0-bin-hadoop3.tgz) even though the image pre-downloads and unpacks the identical tarball specifically "for R tests" — this job doesn't get the caching win the image was built to provide it.
• PythonTests "Test Python Code" step silently dropped one of three ||-chained retry attempts (3 → 2), with no callout here. ~10 of the ~20 PythonTests/UnitTests matrix legs are marked FLAKY: "true".
• PythonTests' core matrix leg runs "project core" installPipPackage publishM2 twice back-to-back (once explicitly, once again because $(PACKAGE) == core). Harmless, just wasted time.
• Docker image permissions are mostly tightened to 755, but chmod -R 777 /opt/conda/envs/synapseml/lib/R/library is intentionally kept for the R package library (needed for UID-1001 write access at test time).

Testing

• Pre-rebase: 64/66 jobs green on an earlier revision (Databricks GPU timeout was pre-existing notebook flakiness, unrelated to containerization).
• Post-rebase (current head): not yet validated — both queued builds failed instantly at pipeline-resource validation due to the SynapseML MCR blocker above. Re-run once the service connection exists.
• sbt scalastyle Test/scalastyle: 0 errors. YAML safe_load + duplicate-key check: pass.

Dependencies

Independent, but works best alongside #2524, #2525, #2526, #2527, #2528 — all split from the original #2506 for independent review.

Open design question

Building/pushing the CI image from the same pipeline that consumes it still means image-push credentials live in the product pipeline (even though BuildCIImage is now gated to trusted master builds). A separate trusted pipeline that only publishes the image, with this pipeline only consuming a pinned digest, would remove that credential surface entirely. Not a blocker for this PR — a follow-up worth considering.

---

This is the core containerization PR, split from #2506.

[github-actions]

Hey Brendan Walsh (@BrendanWalsh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[Copilot]

Pull request overview

Containerizes most Azure DevOps CI jobs to run inside a pre-built Docker image, aiming to reduce end-to-end CI time by pre-baking toolchains, caches, and datasets.

Changes:

• Add a new CI Docker image definition and a pipeline job to build/retag it using a content-hash tag.
• Move major CI jobs (style/tests/publish) to run with container: ci, add disk cleanup, and scope compilation for unit tests.
• Add dataset cache support in build.sbt and introduce a global ScalaTest per-test timeout in TestBase.

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tools/docker/ci/Dockerfile	Defines the CI container image (JDK8, conda env, Spark, SBT warmup, datasets).
templates/free_disk.yml	Adds a reusable host disk cleanup step for container jobs.
pipeline.yaml	Introduces ci container resource, BuildCIImage job, and migrates many jobs to containers/scoped compilation.
build.sbt	Uses DATASET_CACHE to avoid re-downloading test datasets in CI.
core/src/test/.../TestBase.scala	Wraps all tests with a global failAfter timeout.
project/CodegenPlugin.scala	Removes redundant LocalRootProject publishLocal from installPipPackage.
.gitignore	Ignores pipeline.yaml.bak.

[Brendan Walsh (BrendanWalsh)]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.93%. Comparing base (0a2b0b4) to head (4d6cdc8).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2529      +/-   ##
==========================================
+ Coverage   84.66%   84.93%   +0.27%     
==========================================
  Files         335      335              
  Lines       17747    17747              
  Branches     1595     1595              
==========================================
+ Hits        15025    15073      +48     
+ Misses       2722     2674      -48     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Brendan Walsh (BrendanWalsh)]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Brendan Walsh (BrendanWalsh)]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Brendan Walsh (BrendanWalsh)]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Brendan Walsh (BrendanWalsh)]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
No pipelines were found matching this branch/path.

[Copilot]

Review details

Suppressed comments (2)

templates/databricks_e2e_steps.yml:4

• templates/databricks_e2e_steps.yml still runs sbt but no longer restores the shared sbt bootstrap/dependency caches (and the job only references this template). This will force Databricks E2E legs to cold-bootstrap dependencies, increasing runtime and reintroducing Maven Central/429 flakiness that the BuildAndCacheSbt prewarm is meant to avoid.

steps:
  - template: free_disk.yml
  - template: kv.yml
  - template: publish.yml

tools/docker/ci/Dockerfile:103

• The pre-downloaded datasets tarball is fetched at build time without any integrity verification. Since this blob is an external input that feeds tests, it’s safer to pin and verify its digest (similar to the libssl1.1 .deb above) to reduce supply-chain risk and make builds reproducible if the blob content ever changes.

# Pre-download test datasets (static tarball, ~50MB) to avoid downloading in every job
ENV DATASET_CACHE=/opt/datasets
RUN mkdir -p $DATASET_CACHE \
    && wget -q "https://mmlspark.blob.core.windows.net/installers/datasets-2023-04-03.tgz" \
         -O "$DATASET_CACHE/datasets-2023-04-03.tgz"

• Files reviewed: 10/11 changed files
• Comments generated: 1
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
No pipelines were found matching this branch/path.

[Copilot]

Review details

Suppressed comments (2)

pipeline.yaml:262

• Same locale-dependent hashing issue as in BuildAndCacheSbt: this sort -z is not pinned to a stable collation order, so the CI image content-hash tag can vary depending on agent locale. Pin sort to LC_ALL=C so CI_IMAGE_TAG stays deterministic.

        HASH=$({
          sha256sum .dockerignore environment.yml build.sbt sonatype.sbt tools/docker/ci/Dockerfile
          git ls-files -z -- project | sort -z | xargs -0 sha256sum
        } | sha256sum | cut -c1-12)

pipeline.yaml:126

• The CI image build hashes project/ inputs using sort -z without pinning the collation locale. sort is locale-dependent, so the computed content hash (and therefore CI_IMAGE_TAG validation) can change across agents/environments if LC_ALL differs, even when file contents are identical. Pin the sort locale to C to make the tag deterministic.

This issue also appears on line 259 of the same file.

        HASH=$({
          sha256sum .dockerignore environment.yml build.sbt sonatype.sbt tools/docker/ci/Dockerfile
          git ls-files -z -- project | sort -z | xargs -0 sha256sum
        } | sha256sum | cut -c1-12)

• Files reviewed: 10/11 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

tools/docker/ci/Dockerfile:53

• Spark tarball is downloaded and extracted without integrity verification. Since this image is meant to be a trusted, reproducible CI baseline, it would be safer to checksum-verify (or signature-verify) the archive before extracting it.

RUN wget -q "https://mmlspark.blob.core.windows.net/installers/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" -O /tmp/spark.tgz \
    && tar -xzf /tmp/spark.tgz -C /opt \
    && rm /tmp/spark.tgz

tools/docker/ci/Dockerfile:103

• The pre-fetched datasets tarball is downloaded without any integrity verification. A corrupted or tampered artifact would be cached into the CI image and could cause hard-to-debug test failures (or worse). Consider adding checksum verification for the tarball after download.

RUN mkdir -p $DATASET_CACHE \
    && wget -q "https://mmlspark.blob.core.windows.net/installers/datasets-2023-04-03.tgz" \
         -O "$DATASET_CACHE/datasets-2023-04-03.tgz" \

• Files reviewed: 10/11 changed files
• Comments generated: 1
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Automated-review adjudication + one correction to an earlier reply

Recording where I landed on the remaining automated review comments on this PR, including the
suppressed ones, so a reviewer doesn't have to re-derive it.

Correction to my earlier reply on the apt-get install thread. I said the image drops
privileges so the sudo wouldn't have worked anyway. That part was wrong — sudo is installed
in the image. The two substantive reasons still hold and are what actually justify the removal:
graphviz and doxygen are already installed in the image, and the apt lists are deleted in the
same layer, so an apt-get install inside the container would fail to resolve packages rather
than quietly no-op. Flagging it because a wrong reason left standing in a thread is worse than no
reason.

databricks_e2e_steps.yml — sbt cache restore removed, deliberately. Both Databricks jobs now
run with container: ci (pipeline.yaml:399 CPU, :436 GPU), and the image pre-fetches
dependencies into COURSIER_CACHE=/opt/.cache/coursier at build time. Restoring a pipeline-level
cache on top of that is redundant work, not a safeguard.

There is a real gap worth naming, though: this PR's own test_sbt_jobs_use_cache asserts that
every sbt job restores a cache, and it does not catch these two jobs. That isn't because they're
exempt — it's because their sbt "testOnly ..." invocation lives inside the step template, and the
test walks flatten(steps) on pipeline.yaml only, so anything behind a template is invisible to
it. I've left that as-is rather than widening the test's reach in a PR that's already blocked on
infrastructure, but the invariant is weaker than it reads.

Reviewed and dismissed, with reasons

• testWebsiteDocs won't find the conda env in the container — not correct.
  BuildUtils.activateCondaEnv (project/build.scala:72) is not a no-op on Linux; it returns
  Seq("conda", "run", "-n", "synapseml", "--no-capture-output") unless CONDA_DEFAULT_ENV
  already matches. The image has conda on PATH and the env materialized at
  /opt/conda/envs/synapseml.

Acted on in this round

• .dockerignore was not part of the image content hash, so a change to it could not trigger a
  rebuild (f7efec999d).
• apt-get install inside the containerized Publish job, plus a regression test that fails if
  one is reintroduced (c6d5f9aa99).
• Digest verification for the dataset tarball (0cbbdf0b3c) and then for Miniconda, Spark and sbt
  (8c82618c22), each checked against the upstream publisher's own published digest.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
No pipelines were found matching this branch/path.

[Copilot]

Review details

Suppressed comments (3)

tools/docker/ci/Dockerfile:41

• The Miniconda installer is downloaded and executed without any integrity verification. Since this image is used as a trusted CI base, it would be safer to pin and verify the installer with a SHA256 check (similar to the libssl1.1 download) to reduce supply-chain risk.

# Digest is the one published by Anaconda on https://repo.anaconda.com/miniconda/ ; this
# installer is executed as root, so verify before running it rather than after.
RUN wget -q https://repo.anaconda.com/miniconda/Miniconda3-py311_24.11.1-0-Linux-x86_64.sh -O /tmp/miniconda.sh \

templates/databricks_e2e_steps.yml:4

• Databricks E2E jobs run sbt (via this template), but this template no longer restores the shared sbt/Ivy/Coursier cache (sbt_cache.yml). That defeats the existing CI hardening that avoids cold-bootstrapping sbt (and related 429/rate-limit failures) and can significantly increase E2E wall time. Add the sbt cache template back (it can run inside the container; free_disk.yml can remain host-targeted).

steps:
  - template: free_disk.yml
  - template: kv.yml
  - template: publish.yml

tools/ci/tests/test_pipeline_yaml.py:692

• test_every_sbt_running_job_waits_for_the_prewarm_cache only inspects the job's direct steps entries. Jobs like Databricks E2E run sbt inside a referenced step template (templates/databricks_e2e_steps.yml), so this check can miss real sbt executions and allow cache regressions to slip through. Consider treating that template as sbt-running and verifying it restores sbt_cache.yml (or more generally, scanning referenced templates).

        runs_sbt = any(
            _INVOKES_SBT.search(t) or t.strip().startswith("sbt") or "sbt_retry.sh" in t
            for t in texts
        )
        templates = [

• Files reviewed: 10/11 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (3)

tools/docker/ci/Dockerfile:87

• The CI image overrides the torch/torchvision versions pinned in environment.yml (torch==2.1.2, torchvision==0.16.2) by installing older CPU wheels (2.1.0 / 0.16.0). This creates an implicit version downgrade that can lead to dependency mismatches and makes the image less reproducible vs the declared environment. Prefer matching the pinned versions while swapping only the build variant (+cpu).

         "https://download.pytorch.org/whl/cpu/torch-2.1.0%2Bcpu-cp311-cp311-linux_x86_64.whl" \
         "https://download.pytorch.org/whl/cpu/torchvision-0.16.0%2Bcpu-cp311-cp311-linux_x86_64.whl" \

templates/databricks_e2e_steps.yml:4

• This template runs sbt (sbt "testOnly $(TEST-CLASS)") but no longer restores the shared sbt caches. The repo’s CI convention (see templates/sbt_cache.yml header) is that every sbt-running job restores these caches after BuildAndCacheSbt to avoid cold-bootstrapping and 429 rate limiting. Re-add the sbt_cache template here so Databricks E2E jobs follow the same pattern.

steps:
  - template: free_disk.yml
  - template: kv.yml
  - template: publish.yml

tools/ci/tests/test_pipeline_yaml.py:690

• test_every_sbt_running_job_waits_for_the_prewarm_cache only scans each job’s inlined steps, so it misses sbt invocations that live inside referenced templates. In the current pipeline this means DatabricksCPUE2E/DatabricksGPUE2E can run sbt via templates/databricks_e2e_steps.yml without being validated for sbt_cache usage. Expand one level of templates when building texts so sbt usage in templates is detected.

        steps = job.get("steps", [])
        texts = flatten(steps)
        runs_sbt = any(
            _INVOKES_SBT.search(t) or t.strip().startswith("sbt") or "sbt_retry.sh" in t
            for t in texts

• Files reviewed: 10/11 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
No pipelines were found matching this branch/path.