[E13] Patient-level k-fold in place of the single holdout

[sebasmos]

Task metadata

Field	Value
Task ID	E13
Category	Evaluation
Priority	High
Complexity	Medium
Est. days	3
Related	#64 (bootstrap CIs, a different quantity), #63

Features / method

Patient-level k-fold cross-validation in place of the single 70/10/20 holdout.

Description

The paper states that a canonical rerun with patient-level cross-validation is in progress. Nothing in this repo tracks that work and no code path does k-fold, so the claim is currently unbacked.

Every reported number rests on one split with a 245-row test fold. The bootstrap intervals from #64 do not close this gap: they quantify resampling noise within that one split, not variance across splits. Those are different quantities and a reader can easily take the first for the second. With EF<=40% a minority class at this cohort size, split-to-split variance is plausibly the larger of the two.

Steps:

1. Add a patient-level k-fold splitter alongside scripts/make_splits.py, keeping the existing zero-leakage check per fold so no subject_id appears in two folds.
2. Retrain all four probes per fold from the same manifest, seeded, reusing the train_probes.py path so the per-fold runs are directly comparable to the canonical single-split run.
3. Report per-fold and pooled MAE and EF<=40% AUROC for all three inference conditions.
4. Report across-fold spread separately from the within-fold bootstrap, and label each in the artifact so the two intervals can never be read as the same measure.
5. Carry the manifest and per-fold checkpoint hashes in the run metadata, matching the provenance added in Real fairness, calibration, and baseline-gap artifacts from the canonical run #74.

Acceptance criteria:

• A cross-validated result set exists for full, echo_dropped and ecg_dropped.
• Zero patient-level leakage across folds, checked and recorded.
• Across-fold spread and within-fold bootstrap are reported as distinct quantities with distinct labels.
• The paper's cross-validation sentence is either backed by this result set or removed.

[slhhuang]

Hi! I’d be happy to take this issue and help with the patient-level k-fold evaluation. I’m still getting familiar with the current experiment pipeline, so I’ll start by reviewing the existing split and training scripts. Since this issue depends on the provenance changes from PR #74, should I wait for #74 to merge and then branch from main, or should I start from the current #74 branch?

[sebasmos]

Awesome thanks @slhhuang! Yes you could start and later rebase before computing the scores