Skip to content

Build paper tables from frozen audit JSON, not from demo stdout #13

Description

@esaran1

Why

asla demo is a narrative smoke test. asla report and scripts/run_paper_experiments.py exist, but paper numbers for the real grid should come from a frozen results/audit.json plus a deterministic table generator. Synthetic experiment scripts are not a substitute for the 45-job audit.

What to do

Add something like scripts/make_tables.py that reads audit JSON and writes:

  • CSV and LaTeX for projection vs single-scale vs gate (if included); ensemble/race only if those rankers were actually evaluated on the same budget contract
  • estimand name, n_boot, seed, input SHA-256, fit form, budget roles
  • under-seeded cell list
  • crossover table with fit is blind preserved as missing, not as a dash filled in later

Also generate a one-page methods box: resampling unit, noise-band definition, gate rule, and what claims are not licensed.

Acceptance

  • Re-running the script on the same JSON is bit-identical
  • Tables cannot be produced from --fast without labeling them as smoke tests
  • None crossover budgets stay blank/None in LaTeX
  • README documents the exact command used for the paper tables

Do not fabricate

  • Copy numbers by hand from demo terminal output
  • Leave Infinity or NaN in paper artifacts
  • Treat scripts/run_paper_experiments.py --fast synthetic output as the real-grid result

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationpaperPaper tables, figures, and write-up

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions