fix: stop InternalCompat requesting an Internal version that was never published

[Rana Singh (ranadeepsingh)]

Fixes #2653. Follow-up to #2645, which merged with this check red.

Targeted at spark4.1 rather than master so the fix is validated where the failure actually reproduces — and against SynapseML-Internal's own spark4.1 branch (confirmed to exist, so the job's branch-matching does not silently fall back to Internal master).

Symptom

Run Internal Python tests (ExcludeAIFunc) fails on essentially every PR. All ~21 tests error at setup, and the only visible cause is:

pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED]
  Java gateway process exited before sending its port number

which hides the real error in the JVM that never reported a port:

java.lang.RuntimeException: [unresolved dependency:
  com.microsoft.azure#...-internal_2.13;1.1.3.0-python3.13-5-<sha>-20260817-1136-SNAPSHOT: not found]
	at org.apache.spark.util.MavenUtils$.resolveMavenCoordinates(MavenUtils.scala:539)

Root cause

Internal's version comes from sbt-dynver, which appends a live -<yyyyMMdd>-<HHmm> suffix whenever the working tree is dirty, recomputed from the wall clock on every sbt load. Internal's build.sbt sets dynverSonatypeSnapshots := true and dynverSeparator := "-", which is where the -SNAPSHOT suffix and the - separators come from.

Retarget Internal to this build runs two sed -i commands against build.sbt. That is what makes the tree dirty — for the rest of the job. Every subsequent sbt session then picks a different version.

Timeline from build 231488245 (step start times from the ADO timeline, versions from its log):

time	what runs	version
11:32	sbt packagePython publishM2	bakes 1132 into the generated package and publishes that same jar ✅
11:33–11:36	make_mlflow_models.py	uses baked 1132, resolves it fine (found … in local-m2-cache) ✅
11:36	sbt testPythonExcludeAIFunc re-runs CodeGen in a new session	rebakes as 1136 and pip-installs it
11:37+	pytest fixtures	read the baked coordinate and request 1136 — never published ❌

The fixtures read the coordinate from the installed package:

# SynapseML-Internal/src/test/python/setup_spark.py
from synapse.ml.ebm import __spark_package_version__ as __spark_internal_package_version__
...
f"com.microsoft.azure:synapseml-internal_2.12:{__spark_internal_package_version__}"

Two consequences worth calling out:

• make_mlflow_models.py succeeds — it is not the culprit, contrary to what the existing comment in this file says. It runs before the rebake.
• The existing "packagePython and publishM2 MUST share one sbt session" mitigation is necessary but not sufficient: it makes those two agree, but a later sbt session recomputes the version regardless.

Fix

Commit the retarget edits, so the tree is clean and dynver emits one stable version for the whole job.

The OSS side already demonstrates the target end state — its checkout is never edited, so its version carries no timestamp at all: 1.1.3-python3.13-102-bfba9c82-SNAPSHOT, in that very same build.

What the version becomes

No timestamp. Internal's version changes shape from

1.1.3.0-python3.13-5-<sha>-20260817-1132-SNAPSHOT   (dirty)
1.1.3.0-python3.13-6-<sha>-SNAPSHOT                 (clean, after this change)

-SNAPSHOT is retained because dynverSonatypeSnapshots := true and the distance is non-zero. This is the same shape SynapseML-Internal's own CI produces for its branches, since those checkouts are clean.

Verification

The fix rests on two properties of git describe --dirty (what dynver reads). Both checked against git directly rather than assumed:

state	git describe --long --tags --dirty
clean tree	v1.1.3.0-1-g0a826a4f
tracked file modified (the sed)	v1.1.3.0-1-g0a826a4f**+DIRTY**
after commit	v1.1.3.0-2-gdc0c4fca
+ gitignored target/ + untracked files	v1.1.3.0-2-gdc0c4fca

The last row is the one that matters: --dirty considers only modifications to tracked files, so sbt's own build output (.gitignore already excludes **/target/*) cannot reintroduce the drift mid-job. A git diff-index guard fails the step loudly if the tree is somehow dirty anyway.

The commit is local to the agent's checkout and is never pushed.

Also fixed

The diagnostic listing published versions hardcoded *-internal_2.12. On this branch, which builds _2.13, it printed nothing — exactly where a version mismatch most needed to be visible.

Scope

Internal Scala tests were never affected: sbt resolves through its own classpath and never goes through Ivy for the Internal artifact. That is why they pass in the same builds where the Python step fails.

The job stays continueOnError: true and is not a required status check. This restores a signal that has been permanently red; it does not change what gates merges.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Copilot]

Pull request overview

This PR fixes the InternalCompat CI failure mode where SynapseML-Internal’s sbt-dynver version drifts between steps (due to dirty working tree + wall-clock HHmm suffix), causing Internal Python tests to request a Maven coordinate that was never published.

Changes:

• Commit the build.sbt retarget edits during the InternalCompat job so sbt-dynver produces a stable version across later, separate JVM invocations.
• Add a guard that fails loudly if the Internal repo remains dirty after the commit (preventing silent reintroduction of version drift).
• Improve diagnostics by globbing Internal artifacts in ~/.m2 across Scala suffixes (rather than hardcoding _2.12).

Show a summary per file

File	Description
pipeline.yaml	Stabilizes the Internal build/version across CI steps by committing retarget edits and improves artifact diagnostics for Scala 2.13 branches.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Good catch - this was a real defect in the guard, and I verified it rather than assuming:

git commit --quiet --no-verify -am   (nothing staged)  -> exit 1
git diff-index --quiet HEAD --       (clean tree)      -> exit 0
git diff-index --quiet HEAD --       (dirty tree)      -> exit 1

The step runs under set -e (top of the same script), so an unconditional commit would have turned the already retargeted case into a hard step failure and made the step non-idempotent on retry.

Fixed in 7be7e32 by committing only when the tree is actually dirty. The preceding grep -qF checks already prove build.sbt holds the requested synapseMLVersion and Resolver.mavenLocal, so an unchanged tree is a valid state rather than an error - and either way dynver sees a clean tree and emits one stable version, which is the property the rest of the job depends on. The post-commit diff-index guard is unchanged and still fails loudly if the tree is dirty at that point.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[Rana Singh (ranadeepsingh)]

/azp run microsoft.SynapseML

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run microsoft.SynapseML

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[Copilot]

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0 new
• Review effort level: Lite