Fix #1045: seed and stratify the GBM propensity early-stopping split - #1046
Open
Hari20032005 wants to merge 1 commit into
Open
Fix #1045: seed and stratify the GBM propensity early-stopping split#1046Hari20032005 wants to merge 1 commit into
Hari20032005 wants to merge 1 commit into
Conversation
GradientBoostedPropensityModel.fit() drew its early-stopping validation set with neither random_state nor stratify, so a model built with an explicit seed still fit differently on every call -- propensity scores moved by up to 0.55 between identical calls. Pass the same random_state expression _model uses for the XGBClassifier, matching the R-learner's own early-stopping split (rlearner.py:830), and stratify on the treatment indicator as every other treatment split in the library does. Add regression tests covering both properties.
Author
|
The Read the Docs check failed on an intersphinx network timeout, not on anything in this diff: Since the docs build runs under The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1045.
GradientBoostedPropensityModel.fit()drew its early-stopping validation set with neither arandom_statenorstratify, so a model constructed with an explicit seed still produced a different fit on every call.The change
One call, in
causalml/propensity.py:random_state—_modelresolves theXGBClassifierseed asself.model_kwargs.get("random_state", 42); reusing that expression puts the split and the classifier on the same seed. The R-learner's own early-stopping split already does this (rlearner.py:830), so this brings the last unseeded split in the library into line.random_state=Nonestays genuinely random, per the scikit-learn convention.stratify—yhere is the binary treatment indicator, and every other treatment split in the library stratifies on it (compute_r_residuals, whose docstring promises "every fold retains both arms";LogisticRegressionPropensityModel'sStratifiedKFold). Without it the validation set's treated count swings by an order of magnitude on imbalanced data, and at small n can reach zero, which makes the early-stopping metric meaningless.Effect
Same seed, same data, three consecutive fits:
early_stop=Trueearly_stop=Falseearly_stop=Falsewas already reproducible and is unchanged — it is the control that isolates the split as the source of the nondeterminism.Tests
Two regression tests in
tests/test_propensity.py:..._earlystopping_reproducible— same seed gives bit-identical scores, and a different seed still gives different scores, so the test cannot be satisfied by hard-coding a constant...._earlystopping_stratified— on 10%-treated data, asserts the validation split preserves the treatment rate.Both fail on master and pass with the change. Verified the fix under numpy, pandas and polars inputs, which all produce identical scores.
🤖 Generated with Claude Code