Skip to content

Decide paired-seed resampling if seeds are shared experimental units #10

Description

@esaran1

Why

resample_runs_by_cell currently draws seeds independently inside each (intervention, compute) cell. That is correct if cells are independent replicates. It is wrong if seed 0 is one paired experimental unit reused across recipes or scales.

What to do

  1. From the recipe card (Define the actual recipes and what compute=1,2,4,8,16 means #4), state how seeds are assigned.
  2. If seeds are paired, implement a bootstrap that resamples the seed once and applies it across the paired cells.
  3. Add a test with a constructed paired table where the independent-cell bootstrap and the paired bootstrap disagree.
  4. Document which bootstrap asla audit uses in NOTES_ESTIMAND.md and the audit JSON.

Acceptance

  • Pairing policy is written down
  • Implementation matches that policy
  • A regression test locks the chosen resampling unit
  • Audit metadata names the resampling unit

Files

  • asla/analysis/audit.py
  • NOTES_ESTIMAND.md

Do not fabricate

  • Change the bootstrap after seeing the real-data intervals

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    scienceScientific design, estimands, and claim languagestatsStatistical methods, uncertainty, and tests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions