You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
results/ has no provenance file, so a row cannot be tied to the versions that produced it #1129
Current gap:paper/results-frozen/ has a MANIFEST.md recording the commit, whether the tree was dirty, the library versions and the manifests included. results/ has none: it is three CSVs and figures. Yet results/ is the file everything else is compared against, it is rewritten by every rerun, and rows in it already differ from the frozen snapshot partly because of library versions (for example German Credit's S2 gap is -0.0914 in results/ and -0.0837 in the snapshot, with different test-set sizes for COMPAS). Nobody can tell from the file which run, versions or seed produced a row, and scripts/diff_results.py cannot say whether a difference comes from a code change or a library bump.
Suggested approach: have write_report also write results/PROVENANCE.json (or .md): commit, dirty flag, Python and key library versions (numpy, pandas, scikit-learn, scipy, fairlearn), the lock file hash, the resample and permutation counts, and the audits and manifests included. Make diff_results.py print both sides' provenance at the top of its output and flag a version difference. Update the README paragraph on results history.
Done when: a benchmark run writes the file, diff_results.py shows it, there are tests, and scripts/check_generated_files_current.py or a lint check fails when results/*.csv changes without it being updated.
Activity