Skip to content

Explainers: Label Noise by Group, Duplicates and Near-Duplicates, and Influential Rows #1136

Closed as not planned

Description

@yakew7

Data-quality problems do not hit groups equally, and the repo has no explainer on the three that a fairness audit meets most often in practice. Add all three in one pull request.

explainers/label-noise-by-group.md - What Happens When Labels Are Noisier for One Group?

Flip a fraction of training labels for one group on a real audit and measure how the model's per-group accuracy, parity gap and calibration change, with the clean-label values next to them. Cover which kinds of noise (random against systematic) shrink a measured gap and which widen it, and how to detect group-dependent noise with confident learning.

Key citation to verify: Northcutt, Jiang, Chuang (2021), "Confident Learning: Estimating Uncertainty in Dataset Labels" (JAIR), and Jiang, Nachum (2020), "Identifying and Correcting Label Bias in Machine Learning" (AISTATS) - confirm.

explainers/duplicates-and-near-duplicates.md - How Do Duplicate Rows Distort a Fairness Audit?

The same person (or household) appears several times, so splits leak and a group's weight is inflated. Count exact and near-duplicate rows by group in one of the audits, show the effect on the train and test gap when duplicates straddle the split, and show how grouped splits fix it. Link to data leakage.

Key citation to verify: Cite a primary source on deduplication and leakage, for example Barz, Denzler (2020), "Do We Train on Test Data? Purging CIFAR of Near-Duplicates" (Journal of Imaging) - confirm.

explainers/influential-rows.md - Which Rows Move a Fairness Gap?

Leave-one-out and influence-function style checks for the rows whose removal changes a group gap the most, computed on a real audit with a logistic regression (cheap enough to refit). Report how many rows account for half of a gap's sensitivity, what they have in common, and how this differs from outlier detection.

Key citation to verify: Koh, Liang (2017), "Understanding Black-box Predictions via Influence Functions" (ICML), and Cook (1977), "Detection of Influential Observation in Linear Regression" (Technometrics) - confirm.

Related explainers: label-bias, measurement-bias, data-leakage, sampling-bias, survivorship-bias, subgroup-fairness.

See CONTRIBUTING.md: add explainers/<slug>.md plus an entry in assets/explainers-data.json (with a description of 70 to 160 characters, and a seo_title if the title plus the site name would pass 65), then run make build-explainers; the build generates the pages, sitemap, llms-full.txt and OG images, and llms.txt and the README table must list each explainer (scripts/check_explainer_count.py checks both, and scripts/check_explainer_structure.py checks the sections). Match a close sibling explainer, give every explainer a Limitations section and at least two inbound links from older ones, use real code run against one of the repo's audits (faircode.benchmark.load_audit loads an audit exactly like the benchmark harness) and real, not invented, results, and if you quote a Fair Code result say whether it comes from results/ (current) or paper/results-frozen/ (the earlier reference snapshot), per CLAUDE.md. Verify every citation yourself before using it.

Done when: the 3 explainers, their README, CONTRIBUTING, ROADMAP, llms.txt and CHANGELOG entries, the website JSON entries and the regenerated generated files are merged, make lint passes, and each explainer's code is runnable from the repo root.

Activity

  1. added
    documentationImprovements or additions to documentation
    help wantedExtra attention is needed
    on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationhelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions