Skip to content

Fix #1045: seed and stratify the GBM propensity early-stopping split - #1046

Open
Hari20032005 wants to merge 1 commit into
uber:masterfrom
Hari20032005:fix/gbm-propensity-early-stop-split
Open

Fix #1045: seed and stratify the GBM propensity early-stopping split#1046
Hari20032005 wants to merge 1 commit into
uber:masterfrom
Hari20032005:fix/gbm-propensity-early-stop-split

Conversation

@Hari20032005

Copy link
Copy Markdown

Fixes #1045.

GradientBoostedPropensityModel.fit() drew its early-stopping validation set with neither a random_state nor stratify, so a model constructed with an explicit seed still produced a different fit on every call.

The change

One call, in causalml/propensity.py:

X_train, X_val, y_train, y_val = train_test_split(
    X,
    y,
    test_size=stop_val_size,
    random_state=self.model_kwargs.get("random_state", 42),
    stratify=y,
)

random_state_model resolves the XGBClassifier seed as self.model_kwargs.get("random_state", 42); reusing that expression puts the split and the classifier on the same seed. The R-learner's own early-stopping split already does this (rlearner.py:830), so this brings the last unseeded split in the library into line. random_state=None stays genuinely random, per the scikit-learn convention.

stratifyy here is the binary treatment indicator, and every other treatment split in the library stratifies on it (compute_r_residuals, whose docstring promises "every fold retains both arms"; LogisticRegressionPropensityModel's StratifiedKFold). Without it the validation set's treated count swings by an order of magnitude on imbalanced data, and at small n can reach zero, which makes the early-stopping metric meaningless.

Effect

Same seed, same data, three consecutive fits:

before after
early_stop=True not identical, max abs diff 0.5466 identical
early_stop=False identical identical

early_stop=False was already reproducible and is unchanged — it is the control that isolates the split as the source of the nondeterminism.

Tests

Two regression tests in tests/test_propensity.py:

  • ..._earlystopping_reproducible — same seed gives bit-identical scores, and a different seed still gives different scores, so the test cannot be satisfied by hard-coding a constant.
  • ..._earlystopping_stratified — on 10%-treated data, asserts the validation split preserves the treatment rate.

Both fail on master and pass with the change. Verified the fix under numpy, pandas and polars inputs, which all produce identical scores.

🤖 Generated with Claude Code

GradientBoostedPropensityModel.fit() drew its early-stopping validation
set with neither random_state nor stratify, so a model built with an
explicit seed still fit differently on every call -- propensity scores
moved by up to 0.55 between identical calls.

Pass the same random_state expression _model uses for the XGBClassifier,
matching the R-learner's own early-stopping split (rlearner.py:830), and
stratify on the treatment indicator as every other treatment split in
the library does.

Add regression tests covering both properties.
Copilot AI balanced review requested due to automatic review settings September 4, 2026 07:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@CLAassistant

CLAassistant commented Sep 4, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@Hari20032005

Copy link
Copy Markdown
Author

The Read the Docs check failed on an intersphinx network timeout, not on anything in this diff:

WARNING: failed to reach any of the inventories with the following issues:
intersphinx inventory 'https://docs.scipy.org/doc/scipy/objects.inv' not fetchable due to
<class 'requests.exceptions.ConnectTimeout'>: ... Connection to docs.scipy.org timed out. (connect timeout=30)

Since the docs build runs under -W, that single transient warning fails it. This PR touches only causalml/propensity.py and tests/test_propensity.py — no docs, no docstrings — so a re-run should be green.

The build workflows appear to be waiting on maintainer approval (first-time contributor). Locally, tests/test_propensity.py is 9 passed on Python 3.13 / numpy 2.5.2 / scikit-learn 1.9.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GradientBoostedPropensityModel early-stopping split ignores random_state and is not stratified

3 participants