Fix degenerate default propensity model behavior ( #1027 )

[su-jin1425]

Proposed changes

Fixes #1027.

This PR fixes two stability issues in the default propensity models:

• Use neg_log_loss scoring and a wider Cs grid for LogisticRegressionCV, preventing ElasticNetPropensityModel and LogisticRegressionPropensityModel from selecting degenerate propensity scores on imbalanced data.
• Reduce the default max_depth of GradientBoostedPropensityModel from 8 to 3 to reduce excessive in-sample overfitting and propensity-score saturation.
• Add a regression test using IHDP replication 0 to cover all three default propensity models.

The changes are limited to causalml/propensity.py and the corresponding regression test.

Before

After

Test

Before

git switch --detach 35f6e734a478eea6fdf1cfaf1b85e0f04ae99b83

$env:PYTHONWARNINGS="ignore"; python -c @'
import os, numpy as np
from sklearn.metrics import roc_auc_score
from causalml.propensity import ElasticNetPropensityModel, LogisticRegressionPropensityModel, GradientBoostedPropensityModel

p=os.path.join(os.environ["TEMP"],"ihdp_npci_1-100.train.npz")
with np.load(p) as z: X,w=z["x"][:,:,0],z["t"][:,0]

e=ElasticNetPropensityModel(); es=e.fit_predict(X,w)
l=LogisticRegressionPropensityModel(); ls=l.fit_predict(X,w)
g=GradientBoostedPropensityModel(); gs=g.fit_predict(X,w)

auc=roc_auc_score(w,g.model.predict_proba(X)[:,1])
clip=np.mean((gs<=0.001001)|(gs>=0.998999))

print("=== BEFORE: 35f6e73 ===")
print(f"ElasticNet std : {es.std():.6f}")
print(f"Logistic std   : {ls.std():.6f}")
print(f"GBM max_depth  : {g.model.get_params()['max_depth']}")
print(f"GBM AUC        : {auc:.6f}")
print(f"Clipped        : {clip:.2%}")
'@

After

git switch --detach f859030b511acd37fbfb5d8bc5c8081c2c478eb0

$env:PYTHONWARNINGS="ignore"; python -c @'
import os, numpy as np
from sklearn.metrics import roc_auc_score
from causalml.propensity import ElasticNetPropensityModel, LogisticRegressionPropensityModel, GradientBoostedPropensityModel

p=os.path.join(os.environ["TEMP"],"ihdp_npci_1-100.train.npz")
with np.load(p) as z: X,w=z["x"][:,:,0],z["t"][:,0]

e=ElasticNetPropensityModel(); es=e.fit_predict(X,w)
l=LogisticRegressionPropensityModel(); ls=l.fit_predict(X,w)
g=GradientBoostedPropensityModel(); gs=g.fit_predict(X,w)

auc=roc_auc_score(w,g.model.predict_proba(X)[:,1])
clip=np.mean((gs<=0.001001)|(gs>=0.998999))

print("=== AFTER: f859030 ===")
print(f"ElasticNet std : {es.std():.6f}")
print(f"Logistic std   : {ls.std():.6f}")
print(f"GBM max_depth  : {g.model.get_params()['max_depth']}")
print(f"GBM AUC        : {auc:.6f}")
print(f"Clipped        : {clip:.2%}")
'@

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA
• Lint and unit tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• Any dependent changes have been merged and published in downstream modules

Further comments

The fix was verified on IHDP replication 0, where the previous defaults produced degenerate/saturated propensity scores and the updated defaults produce more varied propensity estimates with substantially lower GBM training AUC.

CI checks for Python 3.11, Python 3.12, and lint are passing.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[su-jin1425]

Use #1030 → #1033 → #1032 → #1031 this order to avoid merge conflicts

[jeongyoonlee]

test_propensity_models_ihdp_regression_1027 (tests/test_propensity.py:74) calls fetch_ihdp() with no network marker and no cache guard, so the ordinary suite downloads ~16MB from fredjo.com. That contradicts pyproject.toml:95-101 — "Tests that touch the network are opt-in (pytest -m network) so the ordinary suite stays offline and a source URL moving cannot redden an unrelated PR" — and the pytestmark = pytest.mark.network plus skip-if-not-cached pattern in tests/test_benchmark_loaders.py.

Please switch to a synthetic imbalanced dataset, which drops both the fetch_ihdp call and the dependency on fredjo.com staying up; test_logistic_regression_propensity_model_cs_grid (:31-44) in the same file already builds one. Marking it network would also satisfy the convention, but then the regression stops running in normal CI.

Also use RANDOM_SEED from tests/const (already imported at :11) rather than hardcoding 42 three times.

On the change itself: the Cs 4→10 bump is not load-bearing. Measured on IHDP replication 0 with the kwargs at propensity.py:105-125, scoring="neg_log_loss" alone selects the identical C_=0.0464 (pred std 0.0771, AUC 0.722) with Cs=4, while Cs=10 under the old scoring stays degenerate (C_=1e-4, std 0.0000, AUC 0.395). Since this is a defaults change with wide blast radius, please drop the Cs bump and keep the diff to the scoring fix plus max_depth.

[su-jin1425]

Done.

[jeongyoonlee]

The synthetic data doesn't reproduce the bug, so scoring="neg_log_loss" is untested. On test_propensity_models_imbalanced_1027's dataset (n=200, 5 features, 26% treated), master and this branch give identical results — C_=21.54, pred std 0.2255 — so both np.std(...) > 0.05 assertions pass on master. The test only fails on master via assert pm_gb.model.max_depth == 3, which restates the new default rather than testing behavior.

The collapse needs weak signal plus stronger imbalance in higher dimension. This reproduces on all five seeds I tried (0, 1, 7, 42, 2026):

rng = np.random.RandomState(RANDOM_SEED)
X = rng.normal(size=(400, 25))
logit = 0.3 * X[:, 0] + rng.normal(size=400)
treatment = (logit > np.quantile(logit, 0.90)).astype(int)   # 10% treated

master selects C_=1e-4 (the grid floor) with pred std ≈ 0.0001; with neg_log_loss it selects C_=0.046 with std 0.028–0.059.

Please also assert the mechanism rather than a magnitude — C_ not pinned to the smallest grid value is binary and seed-stable, whereas std > 0.05 sits inside the run-to-run spread of the fixed model. And roc_auc_score(...) < 1.0 for the depth change is knife-edge: depth 8 gives exactly 1.0 in-sample and depth 3 lands around 0.995–0.999, so a bound like < 0.99 would state the intent without depending on an exact float.

[su-jin1425]

Done fixing.

[jeongyoonlee]

build (3.12) is red — assert roc_auc_score(...) < 0.99 fails since in-sample AUC is 1.0 at depth 3 too (my < 0.99 suggestion was wrong).

Please drop the whole pm_gb block: no AUC bound separates depth 3 from depth 8 on this data, and max_depth == 3 only restates the default. The C_ > 1e-4 assertions cover the fix.

[su-jin1425]

Done Fixing.

[jeongyoonlee]

LGTM. Thanks!