Skip to content

[W08] Rebuild paper failure figures from real artifacts (currently unlabeled synthetic demo) #76

Description

@duckyquang

Task metadata

Field Value
Category Writing
Priority Medium
Complexity Low
Est. days 1
Related #68 (W07), PR #74

Description

The case-report PDF under paper/build/ uses failure-framework figures (taxonomy, complementarity) whose caption says "representative seed" — but the only artifacts that existed when it was built came from scripts/run_failure_analysis.py --demo, i.e. planted synthetic data (n=150). Nothing in the caption says synthetic, so the figures read as real-cohort results.

Since PR #74 the harness has a --manifest route and real artifacts exist:

python scripts/run_failure_analysis.py \
  --manifest data/processed/echo_hubert_manifest.parquet --out results/failure

Real numbers on the 245-row test frame (ridge harness, seed-fixed): echo wins per-example 245/0, LOO marginal value echo 107.3 vs ecg 6.5, drop_echo induces 188 critical failures at silent rate 1.0, drop_ecg 59 at 0.27. The real story is stronger than the synthetic one — dropping echo fails silently across the board.

The .tex source is not in this repo (only paper/build/ outputs), so this needs whoever holds the paper source.

Steps:

  1. Regenerate the taxonomy / complementarity / dropout figures from results/failure (real) and swap them into the paper.
  2. Update captions with real n and provenance; anywhere synthetic demo output remains, label it as harness validation on synthetic data.
  3. Rebuild the PDF so paper/build/ stops carrying figures over unlabeled synthetic numbers.

Acceptance criteria:

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions