fix(core): include reference-only distribution categories by ranadeepsingh · Pull Request #2630 · microsoft/SynapseML

Rana Singh (ranadeepsingh) · 2026-08-14T20:55:50Z

Fixes #2010

Summary

DistributionBalanceMeasure now evaluates custom reference distributions over the typed union of observed categories and positive-probability reference categories. Reference-only categories receive zero observed count/probability; observed-only categories receive zero expected values. This fixes truncated distance metrics while preserving empty-map uniform behavior, explicit zeroes, output schema, public JVM signatures, copy behavior, and serialized parameter shape.

The implementation also validates null/non-finite probabilities, sums, numeric key conversion, and duplicate typed integral keys; handles empty inputs, empty sensitive-column lists, null observed values, and one-category chi-square support; and returns the correct right-tail p-value (0) for positive-infinite chi-square statistics.

Execution and compatibility

Lazy DataFrame/Spark SQL plan only: no RDDs, observed-support collection, eager Spark action, cache/persist, or unbounded driver materialization.
Sensitive columns are projected once into a long representation and aggregated over reduced category support.
Runtime output, transformSchema, copy, save/load, generated Python, and documented semantics agree.
A 100,000-row/51-support local smoke run completed in 1.289 seconds; the optimized plan used Union, Aggregate, and Generate, with no InMemoryRelation or Python UDF.
.pipelines/release-compat-prerequisites.txt is intentionally empty: Spark 4.1 sync chore: sync spark4.1 with master #2645 already contains every former prerequisite. Exact replay onto the live Spark 4.1 branch applies cleanly and preserves release ordering.

Validation

Exact merged-tree local validation passed:

JDK 11: core/compile, core/Test/compile, and codegen.
Balance suites: 47/47 tests.
Scala style: zero findings across main and test sources.
Generated-wrapper public Python regressions: 3/3.
Pipeline configuration tests: 40 passed, 21 intentionally skipped.
Pinned Black 22.3.0: 194 files unchanged.
Spark 4.1 / Scala 2.13 / JDK 17 replay: full core/Test/compile and 30/30 DistributionBalanceMeasureSuite tests.

The repository-wide core/testPython task remains unsuitable on this Windows-mounted worktree because unrelated test-data generation encounters filesystem I/O and WSL resolves the Windows Conda executable. The committed generated-wrapper tests were therefore executed directly against the locally built core jar.

Integration, review, and CI

Final PR head: 69138124920df5e475316eebb621a8266b4c1079.
Squash merge: a6fd536ad76eb1b60ac82f31a362ae624886c6ff, based on target 5e2ec081ced634325156a8bddb1311b9f90179b1.
Branch and merge trees are identical: aaab73f0caf6109496ed2d5b85df87b2a9461b63. After the squash merge, live fetch divergence is naturally 1 master-only / 5 branch-only commits, with zero content difference.
Exact-head Copilot review 4959161441 covered 7/7 files with zero comments. All three historical findings are fixed, replied to, resolved, and outdated; there are no active or suppressed-for-head findings.
Exact-head Azure build 231663816 completed with 64 succeeded jobs and one succeededWithIssues job. The sole warning was SSL failure while uploading dependency caches; its Test Python Code step succeeded with 123/123 tests. Azure published the aggregate and every job-level GitHub check as success.
Earlier exact-head retries 231655932 and 231660092 isolated a transient Maven Central resolution outage before tests; the final retry cleared it and completed every functional, Docker, CPU/GPU Databricks, Fabric, publish, style, Scala, Python, and R job.
Get-PrReadiness.ps1 reports complete=true: exact-head automated review covered, zero unresolved/suppressed findings, and zero failed, pending, or missing required checks.

Issue #2010 is closed as completed.

azure-pipelines · 2026-08-14T20:56:00Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Rana Singh (ranadeepsingh) · 2026-08-14T20:56:01Z

/azp run

github-actions · 2026-08-14T20:56:03Z

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

fix: Fix LightGBM crashes with empty partitions
feat: Make HTTP on Spark back-offs configurable
docs: Update Spark Serving usage
build: Add codecov support
perf: improve LightGBM memory usage
refactor: make python code generation rely on classes
style: Remove nulls from CNTKModel
test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

azure-pipelines · 2026-08-14T20:56:15Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR fixes DistributionBalanceMeasure in the core exploratory module so that custom reference distributions correctly contribute categories that appear only in the reference map (not observed in the dataset), ensuring distance metrics (e.g., JSD) are computed over the full intended support. It also updates documentation to describe the new/clarified support semantics and adds regression tests covering reference-only and observed-only categories, key typing, validation, and persistence.

Changes:

Materialize reference-only (positive-probability) categories into a small DataFrame, union with observed stats, and aggregate so metrics are computed on the union support.
Add validation for reference probabilities (range + sum-to-1 tolerance) and key conversion/uniqueness for integral sensitive columns; improve numeric decoding robustness for persisted params.
Expand Scala test coverage and clarify documentation for custom reference distribution behavior.

Show a summary per file

File	Description
docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md	Clarifies custom reference distribution semantics, including union support behavior for observed-only and reference-only categories.
core/src/test/scala/com/microsoft/azure/synapse/ml/exploratory/DistributionBalanceMeasureSuite.scala	Adds regression tests for reference-only/observed-only categories, ordering/typing alignment, validation failures, and persistence.
core/src/main/scala/com/microsoft/azure/synapse/ml/exploratory/DistributionBalanceMeasure.scala	Implements union-support alignment for custom references, validates distributions/keys, and ensures persisted numeric probabilities decode safely.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Files reviewed: 3/3 changed files
Comments generated: 1
Review effort level: Lite

Rana Singh (ranadeepsingh) · 2026-08-14T21:42:29Z

/azp run

azure-pipelines · 2026-08-14T21:42:42Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Review details

Files reviewed: 3/3 changed files
Comments generated: 0 new
Review effort level: Lite

## Summary Strengthen Jensen-Shannon distance verification with exact-value, symmetry, normalization, cancellation, zero-support, bounds, schema, and end-to-end assertions. Clarify the mathematical domain and smoothing semantics in the docs, and remove a stale chart whose embedded values used the old natural-log scale. ## Prompting Intent Independently make GitHub PR microsoft#2631 merge-ready by auditing its full mathematical and runtime behavior, API compatibility, edge cases, related PR microsoft#2630 interactions, review feedback, checks, generated bindings, Spark compatibility, and downstream documentation. Fix every valid issue without bundling the separate reference-support implementation. ## Linked Sources - GitHub issue microsoft#2006: microsoft#2006 - Pull request microsoft#2631: microsoft#2631 - Related reference-support pull request microsoft#2630: microsoft#2630 - Copilot review feedback: microsoft#2631 (review) - Jensen-Shannon definition: https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence ## Rationale Keep the production correction narrowly scoped to dividing Jensen-Shannon divergence by ln(2), preserving natural-log KL divergence and all public APIs. Use independently calculated constants and tolerant assertions to validate semantics across Spark/JVM versions. Document that the unit bound assumes valid probability distributions and that midpoint mixing, not additive smoothing, handles zero support. Leave reference-only category materialization and custom-distribution validation to microsoft#2630 to avoid duplicating unrelated work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-16T08:40:42Z

Coordination note for #2631 (5d91a6143f6f9623bcb4ac64f5d2605fcf7663a0): it changes JS distance from natural-log units to base-2 normalized units without bundling this PR's reference-support work. When #2630 rebases after #2631, its explicit JS constants should become:

reference-only example: 0.33841498603440373 (was 0.28174895710781067)
all-reference-categories-observed example: 0.1975751820353933 (was 0.1644921288538882)

The shared DistributionMetricsCalculator expectation will update automatically when the #2631 test-base change is present. All non-JS metric expectations remain unchanged.

* fix: normalize Jensen-Shannon distance to unit range ## Summary Normalize only Jensen-Shannon distance to base-2 units so its documented endpoints are exact: identical distributions produce 0 and disjoint distributions produce 1. Preserve natural-log KL divergence and every public API, add cancellation protection near zero, expand numerical regression coverage, and refresh the data-balance documentation and sample values. ## Prompting Intent Own GitHub issue #2006 end-to-end with an isolated numerical correction. Trace formulas, documentation, and tests; preserve unrelated relative-entropy metrics and API behavior; test endpoints, symmetric intermediate cases, symmetry, bounds, zero handling, and unchanged metrics; validate with JDK 11 guarded SBT commands and project code review; address PR checks and review comments. ## Linked Sources - GitHub issue: #2006 - Pull request review: #2631 - Data Balance Analysis docs: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md - Sample notebook: docs/Explore Algorithms/Responsible AI/Quickstart - Data Balance Analysis.ipynb - Independent code review: no significant findings ## Rationale Divide Jensen-Shannon divergence by ln(2) at the JS call site rather than changing the shared natural-log entropy helper. This is mathematically equivalent to base-2 relative entropy while explicitly preventing changes to KL divergence and other metrics. Clamp only tiny negative JS divergence caused by floating-point cancellation before the square root, preserving valid values and avoiding NaN near the lower bound. Keep endpoint tolerances tight enough to verify the mathematical result without relying on cross-platform bitwise floating-point equality. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(core): harden Jensen-Shannon distance coverage ## Summary Strengthen Jensen-Shannon distance verification with exact-value, symmetry, normalization, cancellation, zero-support, bounds, schema, and end-to-end assertions. Clarify the mathematical domain and smoothing semantics in the docs, and remove a stale chart whose embedded values used the old natural-log scale. ## Prompting Intent Independently make GitHub PR #2631 merge-ready by auditing its full mathematical and runtime behavior, API compatibility, edge cases, related PR #2630 interactions, review feedback, checks, generated bindings, Spark compatibility, and downstream documentation. Fix every valid issue without bundling the separate reference-support implementation. ## Linked Sources - GitHub issue #2006: #2006 - Pull request #2631: #2631 - Related reference-support pull request #2630: #2630 - Copilot review feedback: #2631 (review) - Jensen-Shannon definition: https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence ## Rationale Keep the production correction narrowly scoped to dividing Jensen-Shannon divergence by ln(2), preserving natural-log KL divergence and all public APIs. Use independently calculated constants and tolerant assertions to validate semantics across Spark/JVM versions. Document that the unit bound assumes valid probability distributions and that midpoint mixing, not additive smoothing, handles zero support. Leave reference-only category materialization and custom-distribution validation to #2630 to avoid duplicating unrelated work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * review(core): clarify Jensen-Shannon test fixtures and docs ## Summary Address the latest automated review by using explicit unused count values in the JS-distance expression fixture and rewriting the sample interpretation relative to the uniform reference distribution with corrected wording. ## Prompting Intent Resolve every valid active review comment on GitHub PR #2631, preserve the strong normalized-distance coverage, and keep the documentation mathematically consistent before rerunning required checks. ## Linked Sources - Pull request #2631: #2631 - Count-fixture review comment: #2631 (comment) - Notebook wording review comment: #2631 (comment) ## Rationale Counts are not read by the private JS-distance expression, so explicit zero placeholders communicate that fact without pretending probabilities are counts. The notebook now compares observed data directly with the configured uniform reference rather than using ambiguous balance terminology. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(core): remove JS distance table padding ## Summary Compact the JS Distance markdown row so the corrected mathematical description remains easy to review and edit without large whitespace-only spans. ## Prompting Intent Audit and address suppressed as well as active review feedback on GitHub PR #2631 before declaring the change ready. ## Linked Sources - Pull request #2631: #2631 - Final Copilot review with suppressed feedback: #2631 (review) ## Rationale Markdown tables do not require visual source alignment. Removing cell padding preserves rendered output while preventing noisy future diffs and directly addresses the suppressed review finding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Fix DistributionBalanceMeasure so custom comparisons cover the union of observed and positive-probability reference categories. Add typed validation, robust persisted-number decoding, one-category and empty-input handling, public Scala/Python regressions, and documentation. ## Prompting Intent Make GitHub issue microsoft#2010 and PR microsoft#2630 engineering-ready end to end. Preserve public signatures, serialized parameter shapes, output schema, empty-map uniform behavior, and existing observed-only semantics while covering reference-only categories, null and invalid inputs, supported category types, copy/save-load, generated Python, and SQL-native scalability. ## Linked Sources - Bug report: microsoft#2010 - Pull request: microsoft#2630 - Related Jensen-Shannon normalization: microsoft#2631 - Automated review finding: microsoft#2630 (comment) - Data balance documentation: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md ## Rationale Build a typed DataFrame from the already driver-resident reference parameter, then union and aggregate it with distributed observed statistics. This aligns support without collecting observed categories, RDD APIs, Python UDFs, or persistent caches. Positive-probability keys extend support, explicit zero entries remain inert, observed-only categories retain zero reference weight, and dynamic support cardinality keeps chi-square behavior correct while preserving compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-17T12:38:06Z

/azp run

azure-pipelines · 2026-08-17T12:38:20Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Review details

Files reviewed: 5/5 changed files
Comments generated: 0 new
Review effort level: Lite

## Summary Fix DistributionBalanceMeasure so custom comparisons cover the union of observed and positive-probability reference categories. Add typed validation, robust persisted-number decoding, one-category and empty-input handling, public Scala/Python regressions, documentation, and conflict-free Spark 4.1 replay. ## Prompting Intent Make GitHub issue microsoft#2010 and PR microsoft#2630 engineering-ready end to end. Preserve public signatures, serialized parameter shapes, output schema, empty-map uniform behavior, and existing observed-only semantics while covering reference-only categories, null and invalid inputs, supported category types, copy/save-load, generated Python, SQL-native scalability, and release-branch compatibility. ## Linked Sources - Bug report: microsoft#2010 - Pull request: microsoft#2630 - Related Jensen-Shannon normalization: microsoft#2631 - Automated review finding: microsoft#2630 (comment) - Spark 4.1 compatibility failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231496504 - Data balance documentation: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md ## Rationale Build a typed DataFrame from the already driver-resident reference parameter, then union and aggregate it with distributed observed statistics. This aligns support without collecting observed categories, RDD APIs, Python UDFs, or persistent caches. Normalize complex-param numeric values during Param validation so Python and persisted integral numbers are safe while preserving the Spark 4 branch's strict-map adaptation and clean patch replay. Dynamic support cardinality keeps chi-square behavior correct without changing public schema or signatures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-17T14:01:39Z

/azp run

## Summary Remove eager input and support-count actions from DistributionBalanceMeasure. Build observed and configured support in one SQL-native plan, preserve empty-input and category semantics with bounded sentinel rows, and add a regression proving transform does not launch Spark jobs or cache its input. ## Prompting Intent Continue PR microsoft#2630 through current-head automated review and address the reported double-scan performance regression without RDD APIs, driver collection, persistent caches, unbounded materialization, public signature changes, schema changes, or serialized parameter changes. Preserve reference-only, observed-only, null, integral, string, uniform, persistence, and Spark 4.1 behavior. ## Linked Sources - Bug report: microsoft#2010 - Pull request: microsoft#2630 - Current-head automated review: microsoft#2630 (review) - Performance finding: microsoft#2630 (comment) - Related Jensen-Shannon normalization: microsoft#2631 ## Rationale Avoid the suggested cache because input-size materialization and cache lifetime are unsafe for a reusable transformer. Instead, project supported string and integral sensitive columns into a long SQL representation, aggregate the input once, and perform row/support totals over the reduced category support. Small configured-reference and sentinel rows stay driver-bounded; sentinels preserve empty-input rejection lazily. This removes eager full scans while retaining all metric and compatibility contracts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Return a right-tail chi-square p-value of zero when the test statistic is positive infinity. Align the independent Scala calculator, strengthen Scala and generated-wrapper Python regressions, and clarify the documented interpretation. ## Prompting Intent Continue PR microsoft#2630 through current-head automated review, audit every new finding, and fix real metric defects without weakening tests or changing public signatures, output schema, serialized parameters, or supported category behavior. Verify the corrected metric through Scala, generated Python, Spark 3.5, and Spark 4.1 paths before pushing. ## Linked Sources - Bug report: microsoft#2010 - Pull request: microsoft#2630 - Current-head review: microsoft#2630 (review) - Chi-square p-value finding: microsoft#2630 (comment) - Data balance documentation: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md ## Rationale The implementation computes the upper-tail probability as `1 - CDF(score)`, whose limit is zero as the score approaches positive infinity. Returning one inverted the statistical meaning and reported maximum compatibility for an impossible observed category. Preserve the one-category degree-of-freedom special case at one, while making infinite statistics consistently produce zero across production, test calculators, Scala tests, Python wrappers, and documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Drain the release compatibility prerequisite manifest after the Spark 4.1 master sync incorporated every configured baseline. This prevents old master patches from being replayed over their Spark 4.1-adapted equivalents before pull request changes are validated. ## Prompting Intent Make PR microsoft#2630 fully engineering-ready by resolving the release-compatibility failure at its root. Verify each configured prerequisite against live master and Spark 4.1 history, preserve replay ordering, run the exact release replay and affected CI tests, and avoid unrelated pipeline changes. ## Linked Sources - Pull request microsoft#2630: microsoft#2630 - Spark 4.1 master sync microsoft#2645: microsoft#2645 - Azure Search auth prerequisite microsoft#2591: microsoft#2591 - Azure Search dependent change microsoft#2604: microsoft#2604 - LightGBM retry prerequisite microsoft#2612: microsoft#2612 - LightGBM IPv6 dependent change microsoft#2637: microsoft#2637 - Coverage baseline microsoft#2507: microsoft#2507 - Precision-recall AUC prerequisite microsoft#2635: microsoft#2635 - Metadata prerequisite microsoft#2632: microsoft#2632 - Failed current-head Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231542808 ## Rationale The release matrix currently validates only Spark 4.1, and sync microsoft#2645 contains every commit named by the manifest. Their source was deliberately adapted during the port, so exact reverse-patch detection cannot recognize them and sequential replay conflicts before the PR patch. An exact local replay showed the first two stale entries failing in turn; an empty manifest then applied the PR patch cleanly, compiled the complete Spark 4.1 test surface, and passed the 30-test distribution suite. Keeping the supported empty manifest is narrower and safer than weakening conflict detection or special-casing individual patches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Review details

Files reviewed: 7/7 changed files
Comments generated: 0 new
Review effort level: Lite

Rana Singh (ranadeepsingh) · 2026-08-18T05:42:33Z

/azp run

azure-pipelines · 2026-08-18T05:42:47Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Rana Singh (ranadeepsingh) · 2026-08-18T05:45:23Z

/azp run

azure-pipelines · 2026-08-18T05:45:34Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Rana Singh (ranadeepsingh) · 2026-08-18T06:30:24Z

/azp run

azure-pipelines · 2026-08-18T06:30:35Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Codecov Comments Bot (codecov-commenter) · 2026-08-18T07:23:25Z

Codecov Report

❌ Patch coverage is 98.52941% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.17%. Comparing base (43f31cb) to head (736437a).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...se/ml/exploratory/DistributionBalanceMeasure.scala	98.52%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2630      +/-   ##
==========================================
+ Coverage   86.47%   87.17%   +0.70%     
==========================================
  Files         337      337              
  Lines       19428    19523      +95     
  Branches     1889     1881       -8     
==========================================
+ Hits        16800    17019     +219     
+ Misses       2628     2504     -124

Files with missing lines	Coverage Δ
...se/ml/exploratory/DistributionBalanceMeasure.scala	`98.08% <98.52%> (+1.59%)`	⬆️

... and 23 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

## Summary Fix DistributionBalanceMeasure so custom comparisons cover the union of observed and positive-probability reference categories. Add typed validation, robust persisted-number decoding, one-category and empty-input handling, public Scala/Python regressions, documentation, and conflict-free Spark 4.1 replay. ## Prompting Intent Make GitHub issue microsoft#2010 and PR microsoft#2630 engineering-ready end to end. Preserve public signatures, serialized parameter shapes, output schema, empty-map uniform behavior, and existing observed-only semantics while covering reference-only categories, null and invalid inputs, supported category types, copy/save-load, generated Python, SQL-native scalability, and release-branch compatibility. ## Linked Sources - Bug report: microsoft#2010 - Pull request: microsoft#2630 - Related Jensen-Shannon normalization: microsoft#2631 - Automated review finding: microsoft#2630 (comment) - Spark 4.1 compatibility failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231496504 - Data balance documentation: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md ## Rationale Build a typed DataFrame from the already driver-resident reference parameter, then union and aggregate it with distributed observed statistics. This aligns support without collecting observed categories, RDD APIs, Python UDFs, or persistent caches. Normalize complex-param numeric values during Param validation so Python and persisted integral numbers are safe while preserving the Spark 4 branch's strict-map adaptation and clean patch replay. Dynamic support cardinality keeps chi-square behavior correct without changing public schema or signatures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Remove eager input and support-count actions from DistributionBalanceMeasure. Build observed and configured support in one SQL-native plan, preserve empty-input and category semantics with bounded sentinel rows, and add a regression proving transform does not launch Spark jobs or cache its input. ## Prompting Intent Continue PR microsoft#2630 through current-head automated review and address the reported double-scan performance regression without RDD APIs, driver collection, persistent caches, unbounded materialization, public signature changes, schema changes, or serialized parameter changes. Preserve reference-only, observed-only, null, integral, string, uniform, persistence, and Spark 4.1 behavior. ## Linked Sources - Bug report: microsoft#2010 - Pull request: microsoft#2630 - Current-head automated review: microsoft#2630 (review) - Performance finding: microsoft#2630 (comment) - Related Jensen-Shannon normalization: microsoft#2631 ## Rationale Avoid the suggested cache because input-size materialization and cache lifetime are unsafe for a reusable transformer. Instead, project supported string and integral sensitive columns into a long SQL representation, aggregate the input once, and perform row/support totals over the reduced category support. Small configured-reference and sentinel rows stay driver-bounded; sentinels preserve empty-input rejection lazily. This removes eager full scans while retaining all metric and compatibility contracts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Return a right-tail chi-square p-value of zero when the test statistic is positive infinity. Align the independent Scala calculator, strengthen Scala and generated-wrapper Python regressions, and clarify the documented interpretation. ## Prompting Intent Continue PR microsoft#2630 through current-head automated review, audit every new finding, and fix real metric defects without weakening tests or changing public signatures, output schema, serialized parameters, or supported category behavior. Verify the corrected metric through Scala, generated Python, Spark 3.5, and Spark 4.1 paths before pushing. ## Linked Sources - Bug report: microsoft#2010 - Pull request: microsoft#2630 - Current-head review: microsoft#2630 (review) - Chi-square p-value finding: microsoft#2630 (comment) - Data balance documentation: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md ## Rationale The implementation computes the upper-tail probability as `1 - CDF(score)`, whose limit is zero as the score approaches positive infinity. Returning one inverted the statistical meaning and reported maximum compatibility for an impossible observed category. Preserve the one-category degree-of-freedom special case at one, while making infinite statistics consistently produce zero across production, test calculators, Scala tests, Python wrappers, and documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Drain the release compatibility prerequisite manifest after the Spark 4.1 master sync incorporated every configured baseline. This prevents old master patches from being replayed over their Spark 4.1-adapted equivalents before pull request changes are validated. ## Prompting Intent Make PR microsoft#2630 fully engineering-ready by resolving the release-compatibility failure at its root. Verify each configured prerequisite against live master and Spark 4.1 history, preserve replay ordering, run the exact release replay and affected CI tests, and avoid unrelated pipeline changes. ## Linked Sources - Pull request microsoft#2630: microsoft#2630 - Spark 4.1 master sync microsoft#2645: microsoft#2645 - Azure Search auth prerequisite microsoft#2591: microsoft#2591 - Azure Search dependent change microsoft#2604: microsoft#2604 - LightGBM retry prerequisite microsoft#2612: microsoft#2612 - LightGBM IPv6 dependent change microsoft#2637: microsoft#2637 - Coverage baseline microsoft#2507: microsoft#2507 - Precision-recall AUC prerequisite microsoft#2635: microsoft#2635 - Metadata prerequisite microsoft#2632: microsoft#2632 - Failed current-head Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231542808 ## Rationale The release matrix currently validates only Spark 4.1, and sync microsoft#2645 contains every commit named by the manifest. Their source was deliberately adapted during the port, so exact reverse-patch detection cannot recognize them and sequential replay conflicts before the PR patch. An exact local replay showed the first two stale entries failing in turn; an empty manifest then applied the PR patch cleanly, compiled the complete Spark 4.1 test surface, and passed the 30-test distribution suite. Keeping the supported empty manifest is narrower and safer than weakening conflict detection or special-casing individual patches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-18T07:47:46Z

/azp run

azure-pipelines · 2026-08-18T07:47:59Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Review details

Files reviewed: 7/7 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot

Review details

Files reviewed: 7/7 changed files
Comments generated: 0 new
Review effort level: Lite

Rana Singh (ranadeepsingh) · 2026-08-18T09:37:40Z

/azp run

Rana Singh (ranadeepsingh) requested a review from Brendan Walsh (BrendanWalsh) as a code owner August 14, 2026 20:55

Copilot AI lite review requested due to automatic review settings August 14, 2026 20:55

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 14, 2026 20:56 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Comment thread .../test/scala/com/microsoft/azure/synapse/ml/exploratory/DistributionBalanceMeasureSuite.scala

Rana Singh (ranadeepsingh) force-pushed the copilot/issue-2010-reference-categories branch from 273536a to 1642707 Compare August 14, 2026 21:42

Copilot AI review requested due to automatic review settings August 14, 2026 21:42

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 14, 2026 21:42 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Rana Singh (ranadeepsingh) mentioned this pull request Aug 16, 2026

fix: normalize Jensen-Shannon distance to unit range #2631

Merged

8 tasks

Copilot AI review requested due to automatic review settings August 17, 2026 12:37

Rana Singh (ranadeepsingh) force-pushed the copilot/issue-2010-reference-categories branch from 1642707 to c1a2db4 Compare August 17, 2026 12:37

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 17, 2026 12:40 View session

Copilot AI reviewed Aug 17, 2026

View reviewed changes

Rana Singh (ranadeepsingh) force-pushed the copilot/issue-2010-reference-categories branch from c1a2db4 to 82a7ec6 Compare August 17, 2026 14:01

Copilot AI review requested due to automatic review settings August 17, 2026 14:01

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 17, 2026 14:01 View session

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 18, 2026 05:27 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

SynapseML CI and others added 4 commits August 18, 2026 00:45

Copilot AI review requested due to automatic review settings August 18, 2026 07:46

Rana Singh (ranadeepsingh) force-pushed the copilot/issue-2010-reference-categories branch from 736437a to 374286e Compare August 18, 2026 07:46

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 18, 2026 07:46 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Merge branch 'master' into copilot/issue-2010-reference-categories

6913812

Copilot AI review requested due to automatic review settings August 18, 2026 08:45

Rana Singh (ranadeepsingh) merged commit a6fd536 into microsoft:master Aug 18, 2026
9 checks passed

This was referenced Aug 18, 2026

[BUG] The setReferenceDistribution method of DistributionBalanceMeasures does not correctly handle categories not present in source dataset #2010

Closed

chore: sync spark4.1 with master #2659

Open

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Rana Singh (ranadeepsingh) mentioned this pull request Aug 18, 2026

chore: sync spark4.0 with master #2661

Open

Uh oh!

Conversation

Rana Singh (ranadeepsingh) commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Execution and compatibility

Validation

Integration, review, and CI

Uh oh!

azure-pipelines Bot commented Aug 14, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 14, 2026

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

azure-pipelines Bot commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 14, 2026

Uh oh!

azure-pipelines Bot commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 16, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 17, 2026

Uh oh!

azure-pipelines Bot commented Aug 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 18, 2026

Uh oh!

azure-pipelines Bot commented Aug 18, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 18, 2026

Uh oh!

azure-pipelines Bot commented Aug 18, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 18, 2026

Uh oh!

azure-pipelines Bot commented Aug 18, 2026

Uh oh!

Codecov Comments Bot (codecov-commenter) commented Aug 18, 2026

Codecov Report

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 18, 2026

Uh oh!

azure-pipelines Bot commented Aug 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Rana Singh (ranadeepsingh) commented Aug 14, 2026 •

edited

Loading