Skip to content

results/ has no provenance file, so a row cannot be tied to the versions that produced it #1129

Open

Description

@yakew7

Where: results/, faircode/benchmark.py (write_report), scripts/freeze_paper_results.py, scripts/diff_results.py, README.md.

Current gap: paper/results-frozen/ has a MANIFEST.md recording the commit, whether the tree was dirty, the library versions and the manifests included. results/ has none: it is three CSVs and figures. Yet results/ is the file everything else is compared against, it is rewritten by every rerun, and rows in it already differ from the frozen snapshot partly because of library versions (for example German Credit's S2 gap is -0.0914 in results/ and -0.0837 in the snapshot, with different test-set sizes for COMPAS). Nobody can tell from the file which run, versions or seed produced a row, and scripts/diff_results.py cannot say whether a difference comes from a code change or a library bump.

Suggested approach: have write_report also write results/PROVENANCE.json (or .md): commit, dirty flag, Python and key library versions (numpy, pandas, scikit-learn, scipy, fairlearn), the lock file hash, the resample and permutation counts, and the audits and manifests included. Make diff_results.py print both sides' provenance at the top of its output and flag a version difference. Update the README paragraph on results history.

Done when: a benchmark run writes the file, diff_results.py shows it, there are tests, and scripts/check_generated_files_current.py or a lint check fails when results/*.csv changes without it being updated.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions