Empirical coverage study for bootstrap intervals

[esaran1]

Why

Audit ranker intervals are percentile bootstraps of decision metrics under cell-wise seed resampling. That is not conformal inference.

This repo also has a separately named conformal projection-interval method (asla/analysis/conformal.py) and scripts/run_calibration_study.py for projection-interval coverage. Keep those names distinct. Ranker-metric tests are still too small to support a calibration claim for mis_selection_rate and mean_regret.

What to do

1. Keep method names honest: ranker CIs are percentile bootstrap; conformal intervals are conformal; do not mix the labels.
2. Add an offline study over repeated synthetic draws for both estimands, including:
   • noise_close_call
   • saturation_crossover
   • negative_controls_only
   • a ragged one-seed / multi-seed table
3. Report achieved vs nominal coverage for mis_selection_rate and mean_regret.
4. Record bootstrap failure rates (bootstrap_diagnostics).
5. If coverage is poor, document that; do not silently switch methods to chase 95%.
6. Reuse or cite scripts/run_calibration_study.py for projection intervals; do not pretend it already covers ranker-metric estimands unless that is shown.

Acceptance

• Coverage table exists for the preregistered estimand
• Sample size is large enough to distinguish 70% from 95% coverage
• Failures to fit are counted, not dropped from the denominator without saying so
• README/paper language matches the method that was actually used

Files

• asla/analysis/audit.py
• scripts/run_calibration_study.py
• tests/test_audit.py
• tests/test_conformal.py

Do not fabricate

• Call bootstrap intervals conformal
• Tune the synthetic generators until coverage looks good