Accuracy Equality is one of the six fairness metrics this repo's benchmark harness actually computes (accuracy_equality_diff in faircode/metrics.py) - not a theoretical add-on, but a real number sitting next to the other five for every audit, strategy, and model in paper/results-frozen/results_fairness.csv.
The baseline logistic regression model on race (disadvantaged: African-American defendants; advantaged: Caucasian defendants) is a case where accuracy is nearly equal but the underlying errors are not, using the frozen numbers exactly as faircode/benchmark.py computed them:
| Metric | Value | 95% CI | p-value | n (disadv. / adv.) |
|---|---|---|---|---|
| Accuracy Equality Diff | -0.035 | [-0.070, -0.003] | 0.0395 | 1,788 / 1,466 |
| Equalized Odds Diff | 0.926 | [0.909, 0.942] | 0.0 | 1,048 / 467 |
| Metric | Value | 95% CI | p-value | n (disadv. / adv.) |
|---|---|---|---|---|
| Accuracy Equality Diff | -0.035 | [-0.070, -0.003] | 0.0395 | 1,788 / 1,466 |
| Equalized Odds Diff | 0.926 | [0.909, 0.942] | 0.0 | 1,048 / 467 |
A 3.5-point accuracy gap (African-American defendants scored slightly less accurately overall) looks like a minor, borderline-significant footnote. It sits right next to a 92.6-point Equalized Odds gap on the exact same model and dataset - faircode/metrics.py's own note field for that row reads driven_by_tpr_gap, meaning the true-positive-rate gap between the two groups is enormous. Reporting only Accuracy Equality here would describe this model as nearly fair. It is not: African-American defendants who did not reoffend were flagged as high-risk at a far higher rate than Caucasian defendants who did not reoffend, a disparity Accuracy Equality's single blended number cannot see because the two error types are canceling each other out in the total.
Computes Accuracy Equality alongside Equalized Odds so a hidden gap can't cancel out unnoticed.
diff --git a/explainers/accuracy-not-enough-healthcare-ai.html b/explainers/accuracy-not-enough-healthcare-ai.html index 80e4e1a5..f738b9bd 100644 --- a/explainers/accuracy-not-enough-healthcare-ai.html +++ b/explainers/accuracy-not-enough-healthcare-ai.html @@ -198,7 +198,7 @@This is why a fairness or safety audit that stops at accuracy is not an audit at all. The questions that matter - who does the model miss, and how often, in which group - live in per-group recall, false-negative rates, and calibration, not in the headline score.
The accuracy paradox is the point where high accuracy and low usefulness meet: the rarer the event, the easier it is to score well by ignoring it.
-| Model on a 3%-readmission population | Accuracy | Readmitted patients caught |
|---|---|---|
| Predicts "no readmission" for everyone | 97% | 0 of every 100 at-risk patients |
| Flags the genuinely high-risk 3% | lower than 97% | most of them |
| Model on a 3%-readmission population | Accuracy | Readmitted patients caught |
|---|---|---|
| Predicts "no readmission" for everyone | 97% | 0 of every 100 at-risk patients |
| Flags the genuinely high-risk 3% | lower than 97% | most of them |
The "predict nothing" model wins on accuracy and loses on the only thing that matters. A useful clinical model often accepts a worse accuracy score on purpose - it takes on more false positives to stop missing true cases. Which means accuracy going down can be the sign of a better model, and a fairness gap can hide entirely inside a flat accuracy number: the same 95% can be 95% for one group and 95% for another while the false negatives are concentrated in whichever group the training data under-represented.
Audit 06 in this repo predicts 30-day hospital readmission from the Diabetes 130-US Hospitals dataset (101,766 records), with race and age as protected attributes. Readmission is a rare, imbalanced outcome - the baseline models flag only a tiny fraction of patients as high risk - so accuracy on this task is high almost by construction, and it is exactly the wrong number to trust.
diff --git a/explainers/ai-hallucinations.html b/explainers/ai-hallucinations.html index 67483f7f..8475362a 100644 --- a/explainers/ai-hallucinations.html +++ b/explainers/ai-hallucinations.html @@ -212,7 +212,7 @@The biased model (unfair.py) is trained on a dataset of patient features to predict whether an insurance claim will be high-cost (which flags it for denial). Among the features are bmi, smoker status, and diabetic status.
In this dataset, smoking is a dominant predictive signal: smokers are highly likely to have high claim costs. The model splits on this feature early. However, when we look at underweight patients who smoke and are diabetic, the training evidence is incredibly thin.
By grouping the entire dataset into bins of BMI category, smoking status, and diabetes status, we can compare the Sample Density (how many patients fit this profile in the training data) against the Model's Prediction Probability of Denial:
-| BMI Category | Smoker | Diabetic | Sample Count (Density) | Mean Predicted Probability of Denial | Mean Prediction |
|---|---|---|---|---|---|
| Underweight | Yes | No | 1 | 98.0% | Deny (1.0) |
| Underweight | Yes | Yes | 4 | 95.3% | Deny (1.0) |
| Underweight | No | Yes | 6 | 28.0% | Approve (0.0) |
| Underweight | No | No | 9 | 33.9% | Approve (0.0) |
| Normal | Yes | No | 22 | 99.1% | Deny (1.0) |
| Normal | Yes | Yes | 28 | 99.3% | Deny (1.0) |
| Overweight | Yes | Yes | 35 | 99.5% | Deny (1.0) |
| Overweight | Yes | No | 39 | 99.5% | Deny (1.0) |
| Obese | Yes | Yes | 69 | 99.6% | Deny (1.0) |
| Obese | Yes | No | 76 | 99.7% | Deny (1.0) |
| Normal | No | No | 83 | 31.6% | Approve (0.0) |
| Normal | No | Yes | 90 | 28.9% | Approve (0.0) |
| Overweight | No | Yes | 148 | 36.4% | Approve (0.0) |
| Overweight | No | No | 167 | 33.2% | Approve (0.0) |
| Obese | No | Yes | 262 | 43.6% | Approve (0.0) |
| Obese | No | No | 301 | 42.5% | Approve (0.0) |
| BMI Category | Smoker | Diabetic | Sample Count (Density) | Mean Predicted Probability of Denial | Mean Prediction |
|---|---|---|---|---|---|
| Underweight | Yes | No | 1 | 98.0% | Deny (1.0) |
| Underweight | Yes | Yes | 4 | 95.3% | Deny (1.0) |
| Underweight | No | Yes | 6 | 28.0% | Approve (0.0) |
| Underweight | No | No | 9 | 33.9% | Approve (0.0) |
| Normal | Yes | No | 22 | 99.1% | Deny (1.0) |
| Normal | Yes | Yes | 28 | 99.3% | Deny (1.0) |
| Overweight | Yes | Yes | 35 | 99.5% | Deny (1.0) |
| Overweight | Yes | No | 39 | 99.5% | Deny (1.0) |
| Obese | Yes | Yes | 69 | 99.6% | Deny (1.0) |
| Obese | Yes | No | 76 | 99.7% | Deny (1.0) |
| Normal | No | No | 83 | 31.6% | Approve (0.0) |
| Normal | No | Yes | 90 | 28.9% | Approve (0.0) |
| Overweight | No | Yes | 148 | 36.4% | Approve (0.0) |
| Overweight | No | No | 167 | 33.2% | Approve (0.0) |
| Obese | No | Yes | 262 | 43.6% | Approve (0.0) |
| Obese | No | No | 301 | 42.5% | Approve (0.0) |
Look at the first two rows. There is only one underweight smoker in the entire dataset, and only four underweight smokers who are diabetic. The training evidence for these sub-populations is virtually non-existent.
Yet, the model predicts a 98.0% and 95.3% probability of denial for these applicants.
diff --git a/explainers/automation-bias.html b/explainers/automation-bias.html index 6adb1a20..cba26cfb 100644 --- a/explainers/automation-bias.html +++ b/explainers/automation-bias.html @@ -202,7 +202,7 @@The bias in the model is a statistical problem. Automation bias is the human problem that lets it leave the lab.
| Pattern | Description | Real-World Consequence |
|---|---|---|
| Omission error | Failing to act because the system didn't flag it | High-risk patient discharged because no alert fired |
| Commission error | Acting on a system recommendation that contradicts evidence | Denying a qualified applicant because the score said "high risk" |
| Default acceptance | Treating the default output as the decision | Auto-approving or auto-denying based on threshold without review |
| Authority transfer | "The algorithm is objective" becomes a shield against accountability | No appeal path because "the math decided" |
| Pattern | Description | Real-World Consequence |
|---|---|---|
| Omission error | Failing to act because the system didn't flag it | High-risk patient discharged because no alert fired |
| Commission error | Acting on a system recommendation that contradicts evidence | Denying a qualified applicant because the score said "high risk" |
| Default acceptance | Treating the default output as the decision | Auto-approving or auto-denying based on threshold without review |
| Authority transfer | "The algorithm is objective" becomes a shield against accountability | No appeal path because "the math decided" |
The COMPAS risk assessment is the canonical example. ProPublica's 2016 analysis showed:
@@ -313,7 +313,7 @@Interpretation: In this run, the final human-decision gap (6.8 points) is actually smaller than the model's own gap (9.0 points) - not larger. That is not a counterexample to automation bias; it is a property of this specific simulation: the 20% of decisions that don't follow the AI score fall back to true_skill, which has no gender bias baked into it at all, so that fallback partially cancels the model's bias rather than compounding it. Automation bias amplifies a gap specifically when the human deviates from the model in ways that are not an unbiased correction - a recruiter's "gut feeling" override in the real world is not guaranteed to be unbiased the way true_skill is by construction here. Treat this block as a worked example of how to measure model_fairness_gap vs. human_fairness_gap with real data, not as proof that amplification is the only possible direction - the direction depends entirely on whether the human's departures from the model are themselves biased or not.
| Strategy | Mechanism | Limitation |
|---|---|---|
| Friction by design | Require explicit override justification; show confidence intervals, not just scores | Adds workflow time; may be bypassed under pressure |
| Counterfactual display | "If this applicant were male, the score would be X" | Requires causal model; contested in court |
| Blind review first | Human evaluates case before seeing AI score | Not always feasible (e.g., triage) |
| Disagreement logging | Track every human-vs-model divergence for audit | Passive; does not prevent harm in real time |
| Calibrated thresholds per group | Different decision thresholds to equalize outcomes | Legal risk (disparate treatment); hard to justify |
| Human-in-the-loop training | Simulate edge cases where model is wrong; train reviewers to catch them | Resource-intensive; decay over time |
| Strategy | Mechanism | Limitation |
|---|---|---|
| Friction by design | Require explicit override justification; show confidence intervals, not just scores | Adds workflow time; may be bypassed under pressure |
| Counterfactual display | "If this applicant were male, the score would be X" | Requires causal model; contested in court |
| Blind review first | Human evaluates case before seeing AI score | Not always feasible (e.g., triage) |
| Disagreement logging | Track every human-vs-model divergence for audit | Passive; does not prevent harm in real time |
| Calibrated thresholds per group | Different decision thresholds to equalize outcomes | Legal risk (disparate treatment); hard to justify |
| Human-in-the-loop training | Simulate edge cases where model is wrong; train reviewers to catch them | Resource-intensive; decay over time |
Automation bias is not a bug in human cognition - it is a predictable response to how AI systems are designed and presented.
diff --git a/explainers/base-rate-fallacy.html b/explainers/base-rate-fallacy.html index e3384b5e..67c7d40d 100644 --- a/explainers/base-rate-fallacy.html +++ b/explainers/base-rate-fallacy.html @@ -205,7 +205,7 @@PPV = P(Y = 1 | Ŷ = 1) = (TPR * p) / (TPR * p + FPR * (1 - p))To see how background prevalence dictates prediction reliability, consider a screening model with fixed TPR = 0.90 and FPR = 0.10 evaluated across varying base rates (p):
-| Base Rate (p) | True Positives (TPR * p) | False Positives (FPR * (1 - p)) | PPV (P(Y = 1 | Ŷ = 1)) | False Discovery Rate (1 - PPV) |
|---|---|---|---|---|
| 1% | 0.0090 | 0.0990 | 8.33% | 91.67% |
| 5% | 0.0450 | 0.0950 | 32.14% | 67.86% |
| 10% | 0.0900 | 0.0900 | 50.00% | 50.00% |
| 30% | 0.2700 | 0.0700 | 79.41% | 20.59% |
| 50% | 0.4500 | 0.0500 | 90.00% | 10.00% |
| Base Rate (p) | True Positives (TPR * p) | False Positives (FPR * (1 - p)) | PPV (P(Y = 1 | Ŷ = 1)) | False Discovery Rate (1 - PPV) |
|---|---|---|---|---|
| 1% | 0.0090 | 0.0990 | 8.33% | 91.67% |
| 5% | 0.0450 | 0.0950 | 32.14% | 67.86% |
| 10% | 0.0900 | 0.0900 | 50.00% | 50.00% |
| 30% | 0.2700 | 0.0700 | 79.41% | 20.59% |
| 50% | 0.4500 | 0.0500 | 90.00% | 10.00% |
At a 1% base rate, over 91% of flagged individuals are false alarms, despite the model having 90% sensitivity and 90% specificity.
When evaluating models across demographic groups A and B, Chouldechova (2017) demonstrated that the false positive rate (FPR), false negative rate (FNR), positive predictive value (PPV), and base rate (p) are linked by a strict identity:
diff --git a/explainers/bootstrap-confidence-intervals.html b/explainers/bootstrap-confidence-intervals.html index f1456cc3..ea530433 100644 --- a/explainers/bootstrap-confidence-intervals.html +++ b/explainers/bootstrap-confidence-intervals.html @@ -203,7 +203,7 @@Neither method assumes normality, a minimum sample size, or a particular metric - the same two functions apply identically whether the metric is a proportion, a rate difference, or a ratio.
Two rows from the exact same audit and model show what a difference in sample size does to certainty, using frozen numbers straight from paper/results-frozen/results_fairness.csv (baseline logistic regression, race):
| Metric | Value | 95% CI | p-value | n (disadv. / adv.) | Verdict |
|---|---|---|---|---|---|
| Equal Opportunity Diff | 0.926 | [0.909, 0.942] | 0.0 | 1,048 / 467 | Clearly real |
| Predictive Parity Diff (random forest) | 0.128 | [-0.218, 0.475] | 0.663 | 1,552 / 6 | Cannot tell |
| Metric | Value | 95% CI | p-value | n (disadv. / adv.) | Verdict |
|---|---|---|---|---|---|
| Equal Opportunity Diff | 0.926 | [0.909, 0.942] | 0.0 | 1,048 / 467 | Clearly real |
| Predictive Parity Diff (random forest) | 0.128 | [-0.218, 0.475] | 0.663 | 1,552 / 6 | Cannot tell |
The first row has a huge effect and a tight interval - 1.7 percentage points wide, entirely on one side of zero, p essentially 0. There is no ambiguity: African-American defendants who did not reoffend were flagged as high-risk at a dramatically higher rate than Caucasian defendants who did not reoffend, and that conclusion would survive almost any resample.
The second row, on the same audit, has a point estimate of 12.8 points - not small - but the confidence interval runs from -21.8 to +47.5, spanning zero by a wide margin, and the permutation p-value is 0.663 (not remotely significant). The reason is visible in the last column: only 6 Caucasian defendants had a positive prediction to measure predictive parity against. A gap computed from 6 people can point in almost any direction depending on which 6 people happened to be in the sample - faircode/metrics.py's own small_sample_warning flag is True on this row for exactly that reason. Reporting the 12.8-point number alone, without the interval, would misrepresent a genuinely uncertain measurement as a finding.
Actual output (n_test=200: 69 young, 131 older):
| Group | Predicted | Actual |
|---|---|---|
| Young (<30) | 0.32 | 0.29 |
| Young (<30) | 0.52 | 0.62 |
| Young (<30) | 0.71 | 0.71 |
| Young (<30) | 0.90 | 1.00 |
| Older (30+) | 0.19 | 0.00 |
| Older (30+) | 0.36 | 0.43 |
| Older (30+) | 0.53 | 0.41 |
| Older (30+) | 0.71 | 0.80 |
| Older (30+) | 0.90 | 0.91 |
| Group | Predicted | Actual |
|---|---|---|
| Young (<30) | 0.32 | 0.29 |
| Young (<30) | 0.52 | 0.62 |
| Young (<30) | 0.71 | 0.71 |
| Young (<30) | 0.90 | 1.00 |
| Older (30+) | 0.19 | 0.00 |
| Older (30+) | 0.36 | 0.43 |
| Older (30+) | 0.53 | 0.41 |
| Older (30+) | 0.71 | 0.80 |
| Older (30+) | 0.90 | 0.91 |
Mean Absolute Calibration Error: 0.0559 for young applicants, 0.0964 for older applicants - the model is better calibrated for young applicants here, the opposite of what the German Credit Lending audit's own headline gap (young applicants flagged as bad credit risks at a higher rate, see German Credit Lending/README.md) might suggest. That's the point calibration and error-rate metrics make separately: a model can be reasonably calibrated for a group (its probabilities mean roughly what they say) while still treating that group's predictions asymmetrically - the same tension Northpointe and ProPublica each measured correctly, just along different axes.
A raw gap (women are 0.109 likely to earn above $50K, men 0.306) says nothing about why. People answer that by "controlling for" other variables such as education, hours or occupation, and the choice changes the answer. Whether it should is not a statistical question. Education is plausibly caused by the attribute in part (unequal schooling), and controlling for it removes a path that may be unfair. A variable caused by both the attribute and the outcome can create a gap when you condition on it.
Counterfactual Fairness and Confounding Variables rely on a graph without showing one. Drawing it down makes the assumptions arguable, which is the point: the data cannot tell two analysts with different graphs who is right.
| Role of a variable | In the graph | Adjusting for it |
|---|---|---|
| Common cause of attribute and outcome | Z -> sex, Z -> income | Needed, or the gap is confounded |
| Mediator | sex -> M -> income | Removes the part of the gap that runs through M. Right if that path is fair, wrong if it is the discrimination |
| Collider | sex -> C <- income | Creates a gap that was not there |
| Role of a variable | In the graph | Adjusting for it |
|---|---|---|
| Common cause of attribute and outcome | Z -> sex, Z -> income | Needed, or the gap is confounded |
| Mediator | sex -> M -> income | Removes the part of the gap that runs through M. Right if that path is fair, wrong if it is the discrimination |
| Collider | sex -> C <- income | Creates a gap that was not there |
The assumed graph for the example below is sex -> education, hours, occupation, marital status -> income, sex -> income directly, and age as a common cause. It is an assumption, written down so it can be argued about.
Benefits Denial: a linear-probability regression of "income above $50K" on being a woman, with different adjustment sets. The coefficient on sex is the gap that remains after adjusting.
diff --git a/explainers/choosing-a-fairness-metric.html b/explainers/choosing-a-fairness-metric.html index 74501a04..115f9d8b 100644 --- a/explainers/choosing-a-fairness-metric.html +++ b/explainers/choosing-a-fairness-metric.html @@ -197,7 +197,7 @@That makes the choice consequential. Pick afterwards and you can find a metric that says what you hoped. The benchmark's own results show how often they diverge.
Work backwards from what goes wrong for a person:
-| The harm | Who it hits | The question | Start with |
|---|---|---|---|
| Not selected when you should have been | Qualified people in one group | Are qualified people accepted at the same rate? | Equal Opportunity |
| Wrongly flagged or denied | People who would have been fine | Are false flags equally common? | False positive rate, in Equalized Odds |
| Unequal access to a benefit | Everyone in a group | Are groups selected at the same rate? | Demographic Parity and Disparate Impact |
| A score that means different things | People relying on the score | Does a positive predict the outcome equally? | Predictive Parity and Calibration |
| One group is simply worse served | The worst-off group | How well off is the worst group? | Max-Min (Rawlsian) Fairness |
| The harm | Who it hits | The question | Start with |
|---|---|---|---|
| Not selected when you should have been | Qualified people in one group | Are qualified people accepted at the same rate? | Equal Opportunity |
| Wrongly flagged or denied | People who would have been fine | Are false flags equally common? | False positive rate, in Equalized Odds |
| Unequal access to a benefit | Everyone in a group | Are groups selected at the same rate? | Demographic Parity and Disparate Impact |
| A score that means different things | People relying on the score | Does a positive predict the outcome equally? | Predictive Parity and Calibration |
| One group is simply worse served | The worst-off group | How well off is the worst group? | Max-Min (Rawlsian) Fairness |
Then check the standard you are held to (a legal threshold such as the 80% rule, a regulator's named metric), and what the label means: if the label is itself biased (Label Bias), the error-rate metrics inherit it and demographic parity may be the safer reading.
For every audit and protected attribute, take the benchmark's S2 model (logistic regression, no protected column, no proxies) in results/ (the current results) and list which of the six metrics are significant. If the metrics agreed, each row would be all or nothing.
Benefits Denial/ audits the Adult Census Income dataset (adult.csv, 32,561 rows). The target is income == '>50K' and sex is a declared protected attribute. Using the dataset's own outcome rates (the base rate a label-level demographic parity check compares):
Unconditional:
-| Group | n | P(income > 50K) |
|---|---|---|
| Male | 21,790 | 30.6% |
| Female | 10,771 | 10.9% |
| Gap (M - F) | +19.6 pp |
| Group | n | P(income > 50K) |
|---|---|---|
| Male | 21,790 | 30.6% |
| Female | 10,771 | 10.9% |
| Gap (M - F) | +19.6 pp |
Now apply CDP with two different choices of legitimate factor L.
education.num, 16 levels)| education.num | n | Gap (M - F) |
|---|---|---|
| 9 (HS grad) | 10,501 | +13.7 pp |
| 10 (some college) | 7,291 | +19.5 pp |
| 13 (Bachelors) | 5,355 | +29.4 pp |
| 14 (Masters) | 1,723 | +32.3 pp |
| 15 (Prof-school) | 576 | +33.1 pp |
| 16 (Doctorate) | 413 | +20.1 pp |
| education.num | n | Gap (M - F) |
|---|---|---|
| 9 (HS grad) | 10,501 | +13.7 pp |
| 10 (some college) | 7,291 | +19.5 pp |
| 13 (Bachelors) | 5,355 | +29.4 pp |
| 14 (Masters) | 1,723 | +32.3 pp |
| 15 (Prof-school) | 576 | +33.1 pp |
| 16 (Doctorate) | 413 | +20.1 pp |
Sample-weighted within-stratum gap: +18.4 pp. Conditioning on education removes almost none of the aggregate gap - within every education level men are markedly more likely to be high earners, and at the top three levels the gap is larger than the unconditional +19.6. CDP conditioned on education says: the disparity is not an artifact of women being less educated in this dataset.
marital.status)| marital.status | n | Gap (M - F) |
|---|---|---|
| Married-civ-spouse | 14,976 | -0.9 pp |
| Never-married | 10,683 | +2.0 pp |
| Divorced | 4,443 | +9.3 pp |
| Widowed | 993 | +17.6 pp |
| marital.status | n | Gap (M - F) |
|---|---|---|
| Married-civ-spouse | 14,976 | -0.9 pp |
| Never-married | 10,683 | +2.0 pp |
| Divorced | 4,443 | +9.3 pp |
| Widowed | 993 | +17.6 pp |
Sample-weighted within-stratum gap: roughly +2 pp, and inside the single largest stratum the sign reverses (women 0.9 points ahead). Conditioning on marital status makes most of the aggregate gap disappear.
Same protected attribute, same dataset, same +19.6 pp aggregate. Condition on education and the gap stands. Condition on marital status and it nearly vanishes. CDP did not resolve the fairness question - it moved it to a new one: is marital status a legitimate basis for a 20-point income difference between men and women, or is it a proxy for a sex-role division of unpaid labor that the outcome variable should not be conditioned on? That is a normative and legal question, not a statistical one, and CDP's output is only as defensible as the answer to it. (This is the same aggregate-vs-stratified divergence covered from the other direction in Simpson's Paradox.)
diff --git a/explainers/confounding-variable.html b/explainers/confounding-variable.html index 645faf21..08efc318 100644 --- a/explainers/confounding-variable.html +++ b/explainers/confounding-variable.html @@ -203,7 +203,7 @@These two concepts are routinely conflated. They are not the same.
-| Proxy Variable | Confounding Variable | |
|---|---|---|
| Mechanism | Encodes the protected attribute as a feature | Independently causes both the feature and the outcome |
| Causal direction | Protected attribute → proxy → prediction | Confounder → feature AND confounder → outcome |
| Fix | Remove the proxy | Remove or statistically adjust for the confounder |
| Example in this repo | zip_code encodes race | Over-policing causes elevated CustodyStatus AND inflated recidivism labels for Black defendants |
| Proxy Variable | Confounding Variable | |
|---|---|---|
| Mechanism | Encodes the protected attribute as a feature | Independently causes both the feature and the outcome |
| Causal direction | Protected attribute → proxy → prediction | Confounder → feature AND confounder → outcome |
| Fix | Remove the proxy | Remove or statistically adjust for the confounder |
| Example in this repo | zip_code encodes race | Over-policing causes elevated CustodyStatus AND inflated recidivism labels for Black defendants |
A proxy smuggles the protected attribute into the model through a correlated feature. A confounder creates a real statistical association between a feature and an outcome - but one that is entirely or partly driven by a third variable, not by the feature itself.
In practice, the same variable often plays both roles simultaneously. CustodyStatus in COMPAS is a proxy (it correlates with race) and a confounder (over-policing independently elevated custody records and generated more surveillance-driven recidivism labels). Treating it as only one or the other misses half the problem.
Accuracy alone is a dangerous metric for fairness because it aggregates all correct predictions and all errors into a single percentage. A model that perfectly predicts outcomes for a 90% majority group but misclassifies everyone in a 10% minority group will still score 90% overall accuracy. By calculating a confusion matrix separately for each demographic group, you stop asking "is the model accurate?" and start asking "how does the model fail, and who absorbs the cost of those failures?"
A binary classification model makes a prediction (Positive or Negative), and the real world provides the actual truth (Positive or Negative). This creates a 2x2 grid of possibilities:
-| Predicted Positive (1) | Predicted Negative (0) | |
|---|---|---|
| Actual Positive (1) | True Positive (TP): Correctly flagged | False Negative (FN): Incorrectly missed |
| Actual Negative (0) | False Positive (FP): Incorrectly flagged | True Negative (TN): Correctly cleared |
| Predicted Positive (1) | Predicted Negative (0) | |
|---|---|---|
| Actual Positive (1) | True Positive (TP): Correctly flagged | False Negative (FN): Incorrectly missed |
| Actual Negative (0) | False Positive (FP): Incorrectly flagged | True Negative (TN): Correctly cleared |
From these four quadrants, we derive the rates that define most fairness metrics:
Plain English: the probability of prediction y for this individual is the same whether we set their group membership to a (the real world) or to a' (the counterfactual). The subscript notation A←a' means "intervene on A, setting it to a'", following do-calculus notation.
The critical term is U - the unobserved background variables. These represent everything about the person that is causally independent of their demographic group: their intrinsic ability, their choices given their circumstances, their effort. A counterfactually fair model makes predictions based only on U - the factors the person controls - not on A or on features that A caused.
Three categories of features emerge from the causal graph:
-| Category | Relationship to A | Allowed in fair model? |
|---|---|---|
| Resolving variables | Causally downstream of A, but a direct, legitimate part of the task | Sometimes - context-dependent |
| Proxy variables | Causally downstream of A, encoding demographic signal | No |
| Non-descendants | Causally independent of A | Yes |
| Category | Relationship to A | Allowed in fair model? |
|---|---|---|
| Resolving variables | Causally downstream of A, but a direct, legitimate part of the task | Sometimes - context-dependent |
| Proxy variables | Causally downstream of A, encoding demographic signal | No |
| Non-descendants | Causally independent of A | Yes |
The challenge is that you must know the causal graph to make these determinations. And causal graphs are not discovered - they are specified, based on domain knowledge and subject to disagreement.
The COMPAS/ audit in this repo is a direct illustration of counterfactual fairness violation - even without using the causal framework explicitly.
COMPAS assigns recidivism risk scores based on features including prior arrest count, age at first arrest, and custody status. The biased model produces:
-| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 87.16% |
| White defendants | 0.40% |
| Fairness Gap | 86.77pp |
| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 87.16% |
| White defendants | 0.40% |
| Fairness Gap | 86.77pp |
Now ask the counterfactual question. Take a Black defendant with 3 prior arrests. Ask: if this person had been white, with the same underlying behaviour and the same 3 prior arrests, would their risk score be the same?
The answer is no - and the reason is CustodyStatus, which the proxy variables explainer identifies as the key proxy. Black communities have been subject to historical over-policing: for the same underlying behaviour, Black individuals are arrested at higher rates than white individuals. Prior arrest count is therefore not causally independent of race - it is partly caused by race, via differential policing. A model trained on prior arrests is therefore using a variable that race, in part, caused.
Race ──→ Policing intensity ──→ Arrest rate ──→ Prior arrests ──→ Risk scoreThese two criteria are closely related and are often confused. The distinction is important:
-| Individual Fairness | Counterfactual Fairness | |
|---|---|---|
| Question | Are similar people treated similarly in the observed data? | Would the same person receive the same outcome in a world where they belong to a different group? |
| Framework | Observational - operates on the data as-is | Causal - requires a structural causal model |
| Unit of analysis | Pairs of individuals | The same individual across possible worlds |
| What it requires | A task-specific similarity metric d(x, y) | A causal DAG specifying which features are downstream of A |
| What it catches | Inconsistent treatment of matched pairs | Decisions whose causal ancestry runs through the protected attribute |
| What it misses | When the similarity metric is itself biased | When the causal graph is mis-specified |
| Easier to compute? | Yes - no causal model needed | No - requires DAG + counterfactual simulation |
| Easier to interpret? | Less intuitive | Highly intuitive - maps to how people reason about discrimination |
| Individual Fairness | Counterfactual Fairness | |
|---|---|---|
| Question | Are similar people treated similarly in the observed data? | Would the same person receive the same outcome in a world where they belong to a different group? |
| Framework | Observational - operates on the data as-is | Causal - requires a structural causal model |
| Unit of analysis | Pairs of individuals | The same individual across possible worlds |
| What it requires | A task-specific similarity metric d(x, y) | A causal DAG specifying which features are downstream of A |
| What it catches | Inconsistent treatment of matched pairs | Decisions whose causal ancestry runs through the protected attribute |
| What it misses | When the similarity metric is itself biased | When the causal graph is mis-specified |
| Easier to compute? | Yes - no causal model needed | No - requires DAG + counterfactual simulation |
| Easier to interpret? | Less intuitive | Highly intuitive - maps to how people reason about discrimination |
The two are complementary. Individual fairness is easier to audit empirically; counterfactual fairness provides the causal explanation for why individual fairness violations occur. A defendant who receives a different risk score than an identical defendant is an individual fairness violation. The reason it happens - because prior arrest count is causally downstream of race via differential policing - is a counterfactual fairness explanation.
Target leakage occurs when a feature used for training carries information that is only known because the outcome has already happened. The feature arrives after the label in the real world, but it is present in the training data alongside that label.
-| Domain | Leaking Feature | Why It Is Leakage |
|---|---|---|
| Credit default | Account closure flag | Account is closed because the borrower defaulted, not before |
| Healthcare readmission | Discharge prescription count | Prescribed on discharge, after the readmission decision would need to be made at admission |
| Criminal justice | Post-arrest charge count | Filed after the arrest the model is predicting |
| Insurance denial | Claim filed within 30 days | Claim exists because the denial outcome is already known |
| Domain | Leaking Feature | Why It Is Leakage |
|---|---|---|
| Credit default | Account closure flag | Account is closed because the borrower defaulted, not before |
| Healthcare readmission | Discharge prescription count | Prescribed on discharge, after the readmission decision would need to be made at admission |
| Criminal justice | Post-arrest charge count | Filed after the arrest the model is predicting |
| Insurance denial | Claim filed within 30 days | Claim exists because the denial outcome is already known |
The model learns to predict the outcome by detecting its own effects. Remove those features at deployment and performance collapses.
Train-test contamination occurs when information from the test set leaks into the training process before the model is evaluated. The most common forms:
diff --git a/explainers/demographic-parity.html b/explainers/demographic-parity.html index 2a491083..d346210e 100644 --- a/explainers/demographic-parity.html +++ b/explainers/demographic-parity.html @@ -211,7 +211,7 @@The AI Fair Recruitment audit in this repo is a direct illustration of a demographic parity violation - and its fix.
A model trained with gender and age as features assigned hire recommendations at sharply different rates:
-| Group | Hire Rate |
|---|---|
| Male applicants | 21.62% |
| Female applicants | 17.59% |
| Fairness Gap | 4.03 percentage points |
| Group | Hire Rate |
|---|---|
| Male applicants | 21.62% |
| Female applicants | 17.59% |
| Fairness Gap | 4.03 percentage points |
(The disparate-impact ratio here is 17.59 / 21.62 = 0.81, just above the 0.80 four-fifths threshold - close enough that it's worth checking directly rather than assuming a pass.)
The model was not told to discriminate. It learned to - by treating gender as a direct signal, since it was fed to the model as a feature. AI Fair Recruitment/audit.yaml also lists Age as a declared proxy feature to remove alongside gender, but in this dataset Age is not actually correlated with Gender (point-biserial r ≈ -0.002, p ≈ 0.42 - not significant, and dropping only Gender while keeping Age already closes the gap to about 0.2 percentage points, nearly as small as the fully-fixed model below). The gap here was overwhelmingly about gender being visible to the model directly, not about a career-gap-driven age proxy.
After dropping gender and age (the protected attribute and the declared proxy), the gap closes to 0.16 percentage points - a 96.1% reduction. The gap wasn't in the underlying merit of candidates - it was in which features the model was permitted to see.
diff --git a/explainers/disparate-impact.html b/explainers/disparate-impact.html index 2d00f199..24e48d8a 100644 --- a/explainers/disparate-impact.html +++ b/explainers/disparate-impact.html @@ -207,13 +207,13 @@The AI Fair Recruitment audit in this repo trains a Random Forest classifier on a recruitment dataset with gender, age, experience, and technical test scores. Apply the 80% rule directly to its outputs:
unfair.pyThe dataset's Gender column has three values, not two - Male, Female, and Other. The Four-Fifths Rule compares every group's selection rate against the highest, not just whichever two look most convenient to compare:
| Group | Hire Rate |
|---|---|
| Men | 21.62% |
| Women | 17.59% |
| Other | 13.40% |
| Group | Hire Rate |
|---|---|
| Men | 21.62% |
| Women | 17.59% |
| Other | 13.40% |
Disparate Impact Ratio = 13.40 / 21.62 = 0.620 (lowest rate ÷ highest rate, across all three groups - not just Men vs. Women)
0.620 < 0.80 → FAILS the Four-Fifths Rule.
Stopping at a Men-vs-Women comparison alone would understate the problem: that pair alone gives 17.59 / 21.62 = 0.81, just above the 0.80 line - a "pass" that hides the fact that the Other group is actually the most disadvantaged of the three, well below the threshold. A real EEOC complaint filed against this model would survive the prima facie stage. The employer would then bear the burden of proving the selection process is "job-related and consistent with business necessity" - the legal standard set in Griggs and codified in 42 U.S.C. §2000e-2(k).
fair.pyAfter dropping Gender and Age and retraining on Experience_Years and Technical_Test_Score only:
| Group | Hire Rate |
|---|---|
| Men | 11.48% |
| Women | 11.32% |
| Other | 11.62% |
| Group | Hire Rate |
|---|---|
| Men | 11.48% |
| Women | 11.32% |
| Other | 11.62% |
Disparate Impact Ratio = 11.32 / 11.62 = 0.974 (lowest ÷ highest, across all three groups)
0.974 ≥ 0.80 → PASSES the Four-Fifths Rule - for every group, not just the two that happened to fail before.
The legal exposure is gone, and the model is selecting candidates on demonstrated ability rather than inferred demographic signal.
diff --git a/explainers/disparate-treatment.html b/explainers/disparate-treatment.html index ce8884e7..b6f8283c 100644 --- a/explainers/disparate-treatment.html +++ b/explainers/disparate-treatment.html @@ -203,7 +203,7 @@These two doctrines are often confused. They are different legal claims with different burdens of proof and different fixes.
-| Disparate Treatment | Disparate Impact | |
|---|---|---|
| Definition | Protected attribute was used as a factor in the decision | Neutral process produces unequal outcomes across groups |
| Intent required? | Yes - the attribute must have been used, deliberately or by design | No - no intent required; the outcome gap is the evidence |
| Legal standard | Title VII §703(a) - treating someone differently because of race, sex, etc. | Title VII §703(k) - neutral practice with unjustified disparate effect |
| Proof | Show the attribute was an input or a determinative factor | Show the statistical outcome gap (four-fifths rule, etc.) |
| Defence | Bona Fide Occupational Qualification (BFOQ) - narrow, rarely succeeds | Business necessity + job-relatedness |
| Algorithmic form | Protected attribute or known proxy included as a model feature | Protected attribute excluded but proxies remain; outcome gap persists |
| Fix | Remove the protected attribute from the feature set | Remove the protected attribute and its proxy variables |
| Metric | Feature presence audit - is it in the model at all? | Disparate impact ratio, demographic parity gap |
| Disparate Treatment | Disparate Impact | |
|---|---|---|
| Definition | Protected attribute was used as a factor in the decision | Neutral process produces unequal outcomes across groups |
| Intent required? | Yes - the attribute must have been used, deliberately or by design | No - no intent required; the outcome gap is the evidence |
| Legal standard | Title VII §703(a) - treating someone differently because of race, sex, etc. | Title VII §703(k) - neutral practice with unjustified disparate effect |
| Proof | Show the attribute was an input or a determinative factor | Show the statistical outcome gap (four-fifths rule, etc.) |
| Defence | Bona Fide Occupational Qualification (BFOQ) - narrow, rarely succeeds | Business necessity + job-relatedness |
| Algorithmic form | Protected attribute or known proxy included as a model feature | Protected attribute excluded but proxies remain; outcome gap persists |
| Fix | Remove the protected attribute from the feature set | Remove the protected attribute and its proxy variables |
| Metric | Feature presence audit - is it in the model at all? | Disparate impact ratio, demographic parity gap |
The critical asymmetry: you can have disparate treatment without disparate impact (the attribute is used but outcomes happen to be equal), and disparate impact without disparate treatment (no protected attribute was used but proxies recreated the gap). In practice, disparate treatment usually produces disparate impact - but the legal exposure exists independently.
unfair.py in This RepoThe baseline is the benchmark's S2 strategy in results/ (the current results): accuracy 0.811, sex parity gap -0.0509. S3, the parity-constrained model, gets a gap of -0.0219 at 0.784 accuracy.
| Model | Accuracy | Worst-group log-loss | Sex parity gap |
|---|---|---|---|
| Baseline (S2) | 0.811 | 0.593 (male 55+) | -0.0509 |
| Group DRO, eta 0.5 | 0.807 | 0.549 | -0.0569 |
| Group DRO, eta 1.0 | 0.810 | 0.542 | -0.0697 |
| Group DRO, eta 2.0 | 0.809 | 0.539 | -0.0664 |
Parity constraint (S3, from results/) | 0.784 | not computed | -0.0219 |
| Model | Accuracy | Worst-group log-loss | Sex parity gap |
|---|---|---|---|
| Baseline (S2) | 0.811 | 0.593 (male 55+) | -0.0509 |
| Group DRO, eta 0.5 | 0.807 | 0.549 | -0.0569 |
| Group DRO, eta 1.0 | 0.810 | 0.542 | -0.0697 |
| Group DRO, eta 2.0 | 0.809 | 0.539 | -0.0664 |
Parity constraint (S3, from results/) | 0.784 | not computed | -0.0219 |
What it did and did not do:
eta=2.0 and nearly 0% on women, which is what over-hedging looks like.Both real audits below use equal_opportunity_diff and equalized_odds_diff exactly as computed by this repo's frozen benchmark harness (paper/results-frozen/results_fairness.csv), baseline model, no mitigation applied.
For the logistic regression baseline on race (disadvantaged: African-American defendants, n=1,048; advantaged: Caucasian defendants, n=467):
-| Metric | Value | 95% CI | p-value |
|---|---|---|---|
| Equal Opportunity Diff | 0.926 | [0.909, 0.942] | 0.0 |
| Equalized Odds Diff | 0.926 | [0.909, 0.942] | 0.0 |
| Metric | Value | 95% CI | p-value |
|---|---|---|---|
| Equal Opportunity Diff | 0.926 | [0.909, 0.942] | 0.0 |
| Equalized Odds Diff | 0.926 | [0.909, 0.942] | 0.0 |
The two numbers are identical here, and the frozen results record exactly why: equalized_odds_diff's own definition is max(|TPR gap|, |FPR gap|), and its note field for this row reads driven_by_tpr_gap - meaning the true-positive-rate gap (what Equal Opportunity measures) was the larger of the two, so Equalized Odds simply inherited it. Nothing was hidden in the false-positive-rate gap this time.
For the same baseline model on race (disadvantaged: Black applicants; advantaged: White applicants):
-| Metric | Value | 95% CI | p-value | Driven by |
|---|---|---|---|---|
| Equal Opportunity Diff | 0.047 | [0.018, 0.077] | 0.0025 | - |
| Equalized Odds Diff | 0.068 | [0.023, 0.111] | 0.0025 | false positive rate |
| Metric | Value | 95% CI | p-value | Driven by |
|---|---|---|---|---|
| Equal Opportunity Diff | 0.047 | [0.018, 0.077] | 0.0025 | - |
| Equalized Odds Diff | 0.068 | [0.023, 0.111] | 0.0025 | false positive rate |
Here they disagree. A screening process relying only on Equal Opportunity would report a modest 4.7-point gap and move on. The real, larger disparity - a 6.8-point false-positive-rate gap, meaning qualified-to-reject Black applicants are screened out at a meaningfully higher excess rate than qualified-to-reject White applicants - only shows up once Equalized Odds is checked. This is the exact failure mode Equal Opportunity's relaxation accepts: it was never designed to catch it.
Computes both metrics side by side so a gap between them is visible immediately, rather than computing Equal Opportunity alone and never finding out what it left unchecked.
diff --git a/explainers/equalized-odds.html b/explainers/equalized-odds.html index 5f0a1974..e8ec752d 100644 --- a/explainers/equalized-odds.html +++ b/explainers/equalized-odds.html @@ -209,7 +209,7 @@COMPAS is a risk assessment tool deployed across 46 US states. Judges use its scores to make bail, sentencing, and parole decisions. It doesn't just need to be accurate - it needs to be equally accurate for every defendant, regardless of race.
Training a RandomForestClassifier(random_state=42) on Sex_Code_Text, CustodyStatus, and MaritalStatus (race already removed, matching this repo's own "remove race only" recipe - see proxy-variables.md), we measured true positive rate (TPR) and false positive rate (FPR) against ScoreText-derived is_high_risk, broken down by race:
| Group | True Positive Rate | False Positive Rate |
|---|---|---|
| Black Defendants | 0.73 | 0.61 |
| White Defendants | 0.64 | 0.43 |
| Gap | 0.08 | 0.18 |
| Group | True Positive Rate | False Positive Rate |
|---|---|---|
| Black Defendants | 0.73 | 0.61 |
| White Defendants | 0.64 | 0.43 |
| Gap | 0.08 | 0.18 |
Interpretation:
The model isn't just wrong more often for Black defendants. It's wrong in a specific direction - over-flagging them as high-risk. Equalized Odds catches this because it checks both error types, not just overall accuracy.
diff --git a/explainers/fairness-accuracy-tradeoff.html b/explainers/fairness-accuracy-tradeoff.html index fa84f3ca..6c7ca1df 100644 --- a/explainers/fairness-accuracy-tradeoff.html +++ b/explainers/fairness-accuracy-tradeoff.html @@ -197,7 +197,7 @@This isn't a reason to avoid fairness mitigation. It's a reason to report both numbers side by side, every time, so the trade being made is visible and can be judged deliberately rather than discovered later.
Both tables below use the baseline logistic regression model on race, across all five strategies, using the frozen numbers exactly as faircode/benchmark.py computed them (paper/results-frozen/results_fairness.csv and paper/results-frozen/results_performance.csv).
| Strategy | Demographic Parity Diff | Accuracy | Accuracy vs. S0 |
|---|---|---|---|
| S0 baseline | 0.855 (sig.) | 65.3% | - |
| S1 unawareness | 0.172 (sig.) | 57.1% | -8.2 pts |
| S2 unawareness_proxy_removal | 0.115 (sig.) | 57.2% | -8.1 pts |
| S3 in_processing | -0.014 (n.s.) | 53.9% | -11.4 pts |
| S4 post_processing | 0.023 (n.s.) | 55.7% | -9.7 pts |
| Strategy | Demographic Parity Diff | Accuracy | Accuracy vs. S0 |
|---|---|---|---|
| S0 baseline | 0.855 (sig.) | 65.3% | - |
| S1 unawareness | 0.172 (sig.) | 57.1% | -8.2 pts |
| S2 unawareness_proxy_removal | 0.115 (sig.) | 57.2% | -8.1 pts |
| S3 in_processing | -0.014 (n.s.) | 53.9% | -11.4 pts |
| S4 post_processing | 0.023 (n.s.) | 55.7% | -9.7 pts |
The pattern Mitigation Strategies already shows on the fairness side - a huge S0→S1 improvement, a smaller S1→S2 gain, and S3/S4 converging near zero - has a cost sitting directly underneath it that a fairness-only table hides completely:
The single largest accuracy drop (S0 → S1, -8.2 points) coincides with the single largest fairness gain (85.5 → 17.2 points closed). Simply dropping the protected attribute buys most of the fairness improvement this ladder achieves, and most of the accuracy cost too, in the same step.
S3 in_processing reaches the best fairness number (-1.4 points, not statistically significant) at the worst accuracy (53.9%, an 11.4-point drop from baseline). This is the sharpest illustration of the trade-off on this audit: the strategy that gets closest to eliminating the demographic parity gap is also the one that costs the model the most predictive accuracy - not a coincidence, since ExponentiatedGradient is directly trading classification accuracy against the fairness constraint during training, by design.
A seeded RandomForestClassifier's output can shift slightly across CPU/BLAS backends (see issue #521); the White group's PPV in particular rests on only 6 predicted-positive rows and should be read as illustrative, not a precise estimate.
Now look at what each party claimed, and why both were technically correct:
-| Metric | ProPublica's Claim | Northpointe's Claim |
|---|---|---|
| False Positive Rate | Black FPR 78.1% vs White 0.3% - unfair | (didn't dispute this) |
| Predictive Parity | (didn't dispute this) | PPV 62.8% vs 50.0% - closer than the error-rate gap, if noisy on 6 White predictions |
| Who bears the cost | Black defendants wrongly flagged high-risk | - |
| Metric | ProPublica's Claim | Northpointe's Claim |
|---|---|---|
| False Positive Rate | Black FPR 78.1% vs White 0.3% - unfair | (didn't dispute this) |
| Predictive Parity | (didn't dispute this) | PPV 62.8% vs 50.0% - closer than the error-rate gap, if noisy on 6 White predictions |
| Who bears the cost | Black defendants wrongly flagged high-risk | - |
Both claims were mathematically accurate. They were measuring different things.
ProPublica asked: do false accusations fall equally? Northpointe asked: when we flag someone, are we equally right? Because Black defendants reoffend at a higher base rate in this dataset - itself a product of over-policing - satisfying one criterion mathematically forces the other to fail.
There is no neutral choice. Each metric prioritises a different kind of fairness and imposes a different kind of cost:
-| Metric | What it prioritises | Who bears the cost when it fails |
|---|---|---|
| Demographic Parity | Equal access to positive outcomes | Groups with higher true positive rates may be under-predicted |
| Equalized Odds | Equal error rates in both directions | Predictive accuracy per group may be sacrificed |
| Predictive Parity | Equal reliability of predictions | Groups with lower base rates face higher false positive rates |
| Metric | What it prioritises | Who bears the cost when it fails |
|---|---|---|
| Demographic Parity | Equal access to positive outcomes | Groups with higher true positive rates may be under-predicted |
| Equalized Odds | Equal error rates in both directions | Predictive accuracy per group may be sacrificed |
| Predictive Parity | Equal reliability of predictions | Groups with lower base rates face higher false positive rates |
Document your choice in the same place you document your model card. A model without a declared fairness metric has not been audited - it has been deployed.
This repo's own benchmark harness names this exact strategy unawareness (S1 in the mitigation strategies ladder) and runs it on every audit specifically to measure how much it actually helps - which, as the real numbers below show, is not a fixed or guaranteed amount.
The demographic parity gap for the baseline logistic regression model on race (disadvantaged: Black applicants, n=2,943; advantaged: White applicants, n=2,224), using the frozen numbers exactly as faircode/benchmark.py computed them:
| Strategy | Demographic Parity Diff | 95% CI | p-value |
|---|---|---|---|
| S0 baseline (race included) | 0.060 | [0.032, 0.086] | 0.0 |
| S1 unawareness (race dropped) | 0.105 | [0.077, 0.131] | 0.0 |
| Strategy | Demographic Parity Diff | 95% CI | p-value |
|---|---|---|---|
| S0 baseline (race included) | 0.060 | [0.032, 0.086] | 0.0 |
| S1 unawareness (race dropped) | 0.105 | [0.077, 0.131] | 0.0 |
Dropping race from the model's inputs did not shrink the gap - it grew, from 6.0 points to 10.5 points, and both numbers are tightly estimated and clearly significant (large n, narrow CIs, p essentially 0). This is the opposite of what "fairness through unawareness" predicts.
It's tempting to explain this by saying the core features still available to the model - Supervision_Risk_Score_First, prior-arrest-episode counts, employment percentage - carry strong correlation with race, and the model just leaned on those correlates once race itself was gone. Running this file's own unawareness_gap_check() (below) against exactly that feature set on the real data doesn't support that story: reconstruction accuracy comes out to 0.602 (0.617 with a fuller feature list), barely above the 0.573 majority-class base rate and well under the function's own 0.75 "reconstructs well" threshold. By this check, these features do not cleanly reconstruct race - so a clean "the proxies stood in for race" explanation for the widened gap does not hold up against this repo's own tool.
What more plausibly moved the gap instead is the model's fitted coefficients, not a proxy reconstructing race: with race unavailable, LogisticRegression refits every remaining coefficient from scratch, and even a modest, imperfect correlate can get reweighted differently once it's the only remaining signal touching group membership - without that correlate ever reconstructing race well enough to clear a 0.75 bar. faircode/strategies.py's next strategy, unawareness_proxy_removal (S2), goes further and drops the explicitly-listed proxy features too - see Mitigation Strategies for how that step, and the constraint-based strategies after it, actually move this specific gap.
In medicine, the two error types are not interchangeable. A false positive on a sepsis alert means an unnecessary blood draw and a false alarm. A false negative means a patient who needed early antibiotics does not get them. A false positive on a hospital readmission model means a wasted phone call from a care manager. A false negative means a preventable readmission, a penalty for the hospital, and a patient who relapses without support. The same asymmetry shows up in cancer screening, suicide-risk scoring, and insurance triage: missing a true case is almost always more expensive, in health and in dollars, than a false alarm.
This is why a model's overall accuracy or AUC can look excellent while quietly failing one group. Two models can hit the same 90% accuracy on the same population and still make completely different trades: one spreads its false negatives evenly, the other concentrates them in a single demographic group. Demographic parity, the metric most audits start with, only checks how often each group gets a positive prediction. It says nothing about which predictions were wrong, or in which direction. A model can pass a demographic parity check and still be silently under-flagging sicker patients in one group - which is exactly what happened in the most consequential case of this kind on record.
| False Positive | False Negative | |
|---|---|---|
| What happens | Model flags high risk; the event does not occur | Model says low risk; the event occurs anyway |
| Immediate cost | Extra test, outreach call, or short-term treatment | Missed intervention, delayed treatment, missed enrollment |
| Downstream cost | Alert fatigue, wasted clinical capacity, patient anxiety | Disease progression, preventable readmission, malpractice exposure |
| Who absorbs it | Health system budget, clinician time | Patient's health, sometimes the patient's life |
| False Positive | False Negative | |
|---|---|---|
| What happens | Model flags high risk; the event does not occur | Model says low risk; the event occurs anyway |
| Immediate cost | Extra test, outreach call, or short-term treatment | Missed intervention, delayed treatment, missed enrollment |
| Downstream cost | Alert fatigue, wasted clinical capacity, patient anxiety | Disease progression, preventable readmission, malpractice exposure |
| Who absorbs it | Health system budget, clinician time | Patient's health, sometimes the patient's life |
A model's decision threshold controls the trade-off directly. Lower the threshold and the model catches more true cases (fewer false negatives) at the cost of more false alarms (more false positives). Raise it and the reverse happens. Because a missed diagnosis is usually costlier than a false alarm, clinical risk models are often tuned deliberately toward higher sensitivity, accepting a worse false positive rate on purpose.
The fairness problem sits one layer beneath this. A single global threshold, chosen once for the whole population using an aggregate cost function, assumes the model's risk scores mean the same thing for every group. If the training data under-represents one group's true risk (through label bias, unequal access to care, or a proxy that behaves differently by group), the same threshold can produce a low false negative rate for one group and a high one for another - even while the overall accuracy number stays flat.
The mechanism is self-sealing: the model predicts high risk for Group A, so Group A receives more scrutiny, so more violations are detected in Group A, so the next model sees even more Group A violations in its training data, so it predicts even higher risk for Group A. At no point does anyone add discrimination. The feedback loop does it automatically.
| Domain | How the Loop Forms |
|---|---|
| Predictive policing | Model flags high-crime zip codes → police are deployed there → more arrests are recorded there → next model treats those zip codes as even higher risk |
| Credit scoring | Model denies loans to certain groups → those groups can't build credit history → next model sees thin credit files and rates them as high risk |
| Hiring AI | Model deprioritizes candidates from certain schools → those schools produce fewer hires → next model has no positive signal from those schools and ranks them lower still |
| Content recommendation | Model shows certain content to a demographic → that demographic's engagement data is dominated by that content type → next model assumes that's all they want |
| Healthcare resource allocation | Model predicts Group A has lower healthcare costs → Group A receives fewer resources → Group A has lower recorded care utilisation → next model interprets low utilisation as lower need |
| Domain | How the Loop Forms |
|---|---|
| Predictive policing | Model flags high-crime zip codes → police are deployed there → more arrests are recorded there → next model treats those zip codes as even higher risk |
| Credit scoring | Model denies loans to certain groups → those groups can't build credit history → next model sees thin credit files and rates them as high risk |
| Hiring AI | Model deprioritizes candidates from certain schools → those schools produce fewer hires → next model has no positive signal from those schools and ranks them lower still |
| Content recommendation | Model shows certain content to a demographic → that demographic's engagement data is dominated by that content type → next model assumes that's all they want |
| Healthcare resource allocation | Model predicts Group A has lower healthcare costs → Group A receives fewer resources → Group A has lower recorded care utilisation → next model interprets low utilisation as lower need |
The most documented case of feedback loop bias in production is predictive policing.
@@ -316,7 +316,7 @@The critical difference between feedback loop bias and ordinary historical bias:
-| Historical Bias | Feedback Loop Bias | |
|---|---|---|
| Source | Biased past data | Model predictions shaping future data |
| Trajectory | Stable (fixed at training time) | Compounding (worsens each cycle) |
| Fix | Audit features once before training | Ongoing monitoring across retraining cycles |
| Detectability | Single-point audit | Requires longitudinal tracking |
| Self-sealing? | No | Yes - model confirms its own prior beliefs |
| Historical Bias | Feedback Loop Bias | |
|---|---|---|
| Source | Biased past data | Model predictions shaping future data |
| Trajectory | Stable (fixed at training time) | Compounding (worsens each cycle) |
| Fix | Audit features once before training | Ongoing monitoring across retraining cycles |
| Detectability | Single-point audit | Requires longitudinal tracking |
| Self-sealing? | No | Yes - model confirms its own prior beliefs |
Feedback loop bias requires interventions at the deployment level, not just the training level.
diff --git a/explainers/individual-fairness.html b/explainers/individual-fairness.html index d4013125..d9c07476 100644 --- a/explainers/individual-fairness.html +++ b/explainers/individual-fairness.html @@ -215,7 +215,7 @@The AI Fair Recruitment audit in this repo is a direct worked example of an individual fairness violation that group metrics partially miss.
Two candidates matched on the merit features the fair model retains:
-| Feature | Candidate A (Female) | Candidate B (Male) |
|---|---|---|
| Experience Years | 8 | 8 |
| Technical Test Score | 84 | 84 |
| Biased model prediction | 0 - rejected | 1 - hired |
| Feature | Candidate A (Female) | Candidate B (Male) |
|---|---|---|
| Experience Years | 8 | 8 |
| Technical Test Score | 84 | 84 |
| Biased model prediction | 0 - rejected | 1 - hired |
d(A, B) on task-relevant features: zero - they are identical. D(f(A), f(B)) on model output: 1 - opposite decisions.
This violates individual fairness: equal inputs, unequal outputs. Group-level metrics catch the aggregate pattern (a 4.03pp gender hire-rate gap). Individual fairness makes explicit why: these two specific people, who are the same on every dimension that should matter, received opposite verdicts.
import pandas as pd
@@ -418,7 +418,7 @@ 3. Consistency metric (Zemel et al.)
Individual Fairness vs. Group Fairness
The two families of fairness criteria are not alternatives - they are different questions with different failure modes.
-Group Fairness Individual Fairness Unit of analysis Demographic population Specific individual Question Are outcomes equal across groups? Are similar people treated similarly? Can be satisfied without the other? Yes Yes Legal grounding Disparate Impact (Title VII) Intentional discrimination, due process What it catches Systematic demographic gaps Within-group and cross-group inconsistency What it misses Harm to specific individuals inside a passing aggregate Whether the task-relevant similarity metric is itself fair Typical detection method Fairness ratio, MetricFrame Matched-pair tests, consistency score, counterfactual probing Repair strategy Resampling, reweighting, post-processing constraints Regularisation, fairness-aware representation learning
+Group Fairness Individual Fairness Unit of analysis Demographic population Specific individual Question Are outcomes equal across groups? Are similar people treated similarly? Can be satisfied without the other? Yes Yes Legal grounding Disparate Impact (Title VII) Intentional discrimination, due process What it catches Systematic demographic gaps Within-group and cross-group inconsistency What it misses Harm to specific individuals inside a passing aggregate Whether the task-relevant similarity metric is itself fair Typical detection method Fairness ratio, MetricFrame Matched-pair tests, consistency score, counterfactual probing Repair strategy Resampling, reweighting, post-processing constraints Regularisation, fairness-aware representation learning
A model can pass Demographic Parity - equal hire rates across genders - while still systematically giving inconsistent decisions to matched pairs across groups. Equally, a model can achieve high individual consistency while producing aggregate outcome gaps if one group contains fewer high-merit individuals in the dataset (which may itself reflect historical bias, not ability).
The two are complementary, not competing. A complete fairness audit runs both.
diff --git a/explainers/label-bias.html b/explainers/label-bias.html
index 78b7d34d..ddd86e42 100644
--- a/explainers/label-bias.html
+++ b/explainers/label-bias.html
@@ -355,7 +355,7 @@ 3. Counterfactual label audit
The Difference Between Label Bias and Model Bias
This distinction matters when deciding how to fix the problem.
-Model Bias Label Bias Source Features or training process Ground-truth labels Location In the feature matrix or model weights In the target column Detection Standard fairness metrics (DI ratio, Equalized Odds) Conditional label distribution audits Fix Drop proxy variables, re-train, apply constraints Relabel data, collect new labels, apply label-noise correction Detectable without external ground truth? Yes Often no
+Model Bias Label Bias Source Features or training process Ground-truth labels Location In the feature matrix or model weights In the target column Detection Standard fairness metrics (DI ratio, Equalized Odds) Conditional label distribution audits Fix Drop proxy variables, re-train, apply constraints Relabel data, collect new labels, apply label-noise correction Detectable without external ground truth? Yes Often no
A model can be perfectly calibrated and still perpetuate label bias. It learned exactly what it was taught.
Mitigation Strategies
diff --git a/explainers/maxmin-fairness.html b/explainers/maxmin-fairness.html
index d40f2092..c99e50e2 100644
--- a/explainers/maxmin-fairness.html
+++ b/explainers/maxmin-fairness.html
@@ -208,7 +208,7 @@ Concrete Example: Benefits Denial
max-min overall 0.811 | male, White/Asian 0.770 | male, other race 0.852 | female, White/Asian 0.874 | female, other race 0.914 | worst group 0.770 | sex parity gap -0.0563
group weights [11.39 1. 1. 1. ]Read it against the benchmark's own rows in results/ (the current results, not the frozen snapshot). The baseline here is the benchmark's S2 strategy (unawareness_proxy_removal): 0.811 accuracy and a sex parity gap of -0.0509. The in-processing strategy S3 (Fairlearn's ExponentiatedGradient, a parity constraint) reaches a gap of -0.0219 at 0.784 accuracy.
| Model | Overall accuracy | Worst group accuracy | Sex parity gap |
|---|---|---|---|
| Baseline (S2) | 0.811 | 0.757 | -0.0509 |
| Max-min reweighting | 0.811 | 0.770 | -0.0563 |
Parity constraint (S3, from results/) | 0.784 | not computed | -0.0219 |
| Model | Overall accuracy | Worst group accuracy | Sex parity gap |
|---|---|---|---|
| Baseline (S2) | 0.811 | 0.757 | -0.0509 |
| Max-min reweighting | 0.811 | 0.770 | -0.0563 |
Parity constraint (S3, from results/) | 0.784 | not computed | -0.0219 |
Three things to notice:
This repo's own benchmark harness is built entirely around this taxonomy - it applies the same five-strategy ladder to every audit, so "does this fix work" always means "compared to which stage of the pipeline."
faircode/strategies.py implements five strategies in strictly increasing order of intervention:
| Strategy | Stage | What it does |
|---|---|---|
S0 baseline | none | Every feature, including the protected attribute, used as-is. |
S1 unawareness | pre-processing | Drop the protected attribute column only. |
S2 unawareness_proxy_removal | pre-processing | Drop the protected attribute and its known proxies. |
S3 in_processing | in-processing | Fairlearn ExponentiatedGradient (Agarwal et al. 2018) - trains under a Demographic Parity constraint, on the same reduced feature set as S2. |
S4 post_processing | post-processing | Fairlearn ThresholdOptimizer (Hardt, Price & Srebro 2016 style) - fits per-group decision thresholds after training, on the same reduced feature set as S2. |
| Strategy | Stage | What it does |
|---|---|---|
S0 baseline | none | Every feature, including the protected attribute, used as-is. |
S1 unawareness | pre-processing | Drop the protected attribute column only. |
S2 unawareness_proxy_removal | pre-processing | Drop the protected attribute and its known proxies. |
S3 in_processing | in-processing | Fairlearn ExponentiatedGradient (Agarwal et al. 2018) - trains under a Demographic Parity constraint, on the same reduced feature set as S2. |
S4 post_processing | post-processing | Fairlearn ThresholdOptimizer (Hardt, Price & Srebro 2016 style) - fits per-group decision thresholds after training, on the same reduced feature set as S2. |
S1 alone is fairness through unawareness - the intuition that removing the protected attribute is sufficient - and this repo's own results are the direct rebuttal to that intuition, shown below. S3 and S4 deliberately train on the same feature set as S2, not the full S0 feature set, so any further improvement they show over S2 can be attributed to the constraint or the threshold adjustment itself, not to seeing different data.
The demographic parity gap for the baseline logistic regression model on race, across all five strategies, using the frozen numbers exactly as faircode/benchmark.py computed them (paper/results-frozen/results_fairness.csv):
| Strategy | Demographic Parity Diff | 95% CI | p-value |
|---|---|---|---|
| S0 baseline | 0.855 | [0.839, 0.871] | 0.0 |
| S1 unawareness | 0.172 | [0.139, 0.206] | 0.0 |
| S2 unawareness_proxy_removal | 0.115 | [0.088, 0.144] | 0.0 |
| S3 in_processing | -0.014 | [-0.040, 0.011] | 0.3305 |
| S4 post_processing | 0.023 | [-0.009, 0.056] | 0.1485 |
| Strategy | Demographic Parity Diff | 95% CI | p-value |
|---|---|---|---|
| S0 baseline | 0.855 | [0.839, 0.871] | 0.0 |
| S1 unawareness | 0.172 | [0.139, 0.206] | 0.0 |
| S2 unawareness_proxy_removal | 0.115 | [0.088, 0.144] | 0.0 |
| S3 in_processing | -0.014 | [-0.040, 0.011] | 0.3305 |
| S4 post_processing | 0.023 | [-0.009, 0.056] | 0.1485 |
Three things this progression shows that a single before/after comparison would miss:
Dropping the protected attribute alone (S0 → S1) does most of the work, but leaves a real gap. The 85.5-point baseline gap falls to 17.2 points - a huge improvement - but 17.2 points is still a large, clearly significant disparity. This is exactly the failure mode Fairness Through Unawareness warns about: "we removed race" does not mean "we removed race's effect."
Removing proxies too (S1 → S2) helps further, but not proportionally as much. Another 5.7 points close, to 11.5 - a real but much smaller gain than the first step, since not every remaining correlate of race was captured by the defined proxy list.
diff --git a/explainers/ml-bias.html b/explainers/ml-bias.html index 58ab1375..2f04314b 100644 --- a/explainers/ml-bias.html +++ b/explainers/ml-bias.html @@ -327,7 +327,7 @@Detecting a gap requires choosing a metric. Fair Code uses demographic parity as the default: equal positive prediction rates across groups. But demographic parity is not the only option, and it is not always the right one. There are at least three fairness metrics that practitioners commonly reach for, and it has been mathematically proven that you cannot satisfy all three simultaneously except in degenerate cases.
-| Metric | What it requires |
|---|---|
| Demographic Parity | Equal positive prediction rates across groups |
| Equalized Odds | Equal true positive and false positive rates across groups |
| Predictive Parity | Equal precision (positive predictive value) across groups |
| Metric | What it requires |
|---|---|
| Demographic Parity | Equal positive prediction rates across groups |
| Equalized Odds | Equal true positive and false positive rates across groups |
| Predictive Parity | Equal precision (positive predictive value) across groups |
Choosing the wrong metric for a domain produces a model that looks fair on the metric you chose and is biased on the metrics you ignored. See Why Fairness Metrics Conflict for the full mathematical proof and guidance on which metric fits which domain.
The German Credit Lending dataset has no timestamp column, it is a static, single-release academic dataset of 1,000 applicants, so there is no genuinely chronological split available. What follows treats row order as a stand-in for arrival order, which is a real limitation named explicitly below, not a claim that this specific file exhibits true calendar drift.
Splitting the 1,000 rows into five sequential windows of 200 and re-measuring the age-based fairness gap (age < 30 vs 30+, good-credit rate) in each window shows the gap is not a fixed number:
| Window | Good-credit rate | Share under 30 | Fairness gap (older - younger) |
|---|---|---|---|
| 1 | 71.5% | 33.5% | 4.3% |
| 2 | 74.5% | 40.0% | 11.7% |
| 3 | 65.5% | 38.5% | 13.6% |
| 4 | 69.0% | 38.5% | 15.1% |
| 5 | 69.5% | 35.0% | 10.2% |
| Window | Good-credit rate | Share under 30 | Fairness gap (older - younger) |
|---|---|---|---|
| 1 | 71.5% | 33.5% | 4.3% |
| 2 | 74.5% | 40.0% | 11.7% |
| 3 | 65.5% | 38.5% | 13.6% |
| 4 | 69.0% | 38.5% | 15.1% |
| 5 | 69.5% | 35.0% | 10.2% |
The original unfair.py audit reports a single 7.16% gap from one random 80/20 split, not statistically significant at n=1,000 (p=0.2564). That one number sits inside a range that swings from 4.3% to 15.1% depending on which 200-row slice happens to get measured. A single audit run can land near either end of that range purely by chance of which rows fall in the test set, which is exactly the instability a rolling-window view is built to catch and a one-shot snapshot cannot.
Running PSI between the first and last windows on three features gives a concrete read on which ones actually moved:
-| Feature | PSI (window 1 vs window 5) | Read |
|---|---|---|
age | 0.096 | borderline, just under the conventional 0.10 "no significant shift" cutoff |
credit_amount | 0.119 | moderate shift |
duration | 0.045 | no meaningful shift |
| Feature | PSI (window 1 vs window 5) | Read |
|---|---|---|
age | 0.096 | borderline, just under the conventional 0.10 "no significant shift" cutoff |
credit_amount | 0.119 | moderate shift |
duration | 0.045 | no meaningful shift |
credit_amount is the feature that moved the most between the two windows, ahead of age itself, which is the kind of result a rolling PSI check surfaces and a single covariate-shift test run only once would not necessarily have been pointed at.
import numpy as np
diff --git a/explainers/multiple-comparisons.html b/explainers/multiple-comparisons.html
index 7428644f..f5b451d2 100644
--- a/explainers/multiple-comparisons.html
+++ b/explainers/multiple-comparisons.html
@@ -203,9 +203,9 @@ Core Concept
- Benjamini-Hochberg sorts the
m p-values ascending (rank 1..m) and finds the largest rank i where p_(i) <= (i / m) * alpha. Every p-value at or below that rank is called significant - not just the ones that individually clear their own step, unlike Holm's sequential stop. Because later, larger ranks get a looser threshold that scales up toward alpha itself, BH-FDR keeps far more discoveries than Bonferroni or Holm at the same nominal level, at the cost of tolerating more false positives among them (bounded, on average, at the chosen FDR rate).
Concrete Example: COMPAS - Audit 01
COMPAS declares one protected attribute (race), so its full grid is 5 strategies x 3 models x 6 metrics = 90 p-values - real numbers straight from paper/results-frozen/results_fairness.csv. At the raw 0.05 threshold, 67 of the 90 read "significant." After correction:
-Correction Significant (of 90) Threshold used Raw (p < 0.05) 67 0.05 for every test Bonferroni 53 0.05 / 90 = 0.00056 for every test Holm 55 sequential, from 0.00056 up to 0.05 Benjamini-Hochberg FDR 64 sequential, scaling with rank
+Correction Significant (of 90) Threshold used Raw (p < 0.05) 67 0.05 for every test Bonferroni 53 0.05 / 90 = 0.00056 for every test Holm 55 sequential, from 0.00056 up to 0.05 Benjamini-Hochberg FDR 64 sequential, scaling with rank
A sample of individual rows across that boundary shows what each correction actually does to a specific finding:
-Strategy Model Metric p-value Raw Bonferroni Holm BH-FDR baseline logistic_regression demographic_parity_diff 0.0000 significant significant significant significant unawareness_proxy_removal random_forest equal_opportunity_diff 0.0010 significant not significant significant unawareness_proxy_removal logistic_regression equalized_odds_diff 0.0015 significant not not significant baseline random_forest accuracy_equality_diff 0.0065 significant not not significant in_processing random_forest demographic_parity_diff 0.0225 significant not not significant in_processing logistic_regression equal_opportunity_diff 0.0385 significant not not not baseline logistic_regression accuracy_equality_diff 0.0395 significant not not not unawareness logistic_regression accuracy_equality_diff 0.9455 not not not not
+Strategy Model Metric p-value Raw Bonferroni Holm BH-FDR baseline logistic_regression demographic_parity_diff 0.0000 significant significant significant significant unawareness_proxy_removal random_forest equal_opportunity_diff 0.0010 significant not significant significant unawareness_proxy_removal logistic_regression equalized_odds_diff 0.0015 significant not not significant baseline random_forest accuracy_equality_diff 0.0065 significant not not significant in_processing random_forest demographic_parity_diff 0.0225 significant not not significant in_processing logistic_regression equal_opportunity_diff 0.0385 significant not not not baseline logistic_regression accuracy_equality_diff 0.0395 significant not not not unawareness logistic_regression accuracy_equality_diff 0.9455 not not not not
The first row (p essentially 0) survives every correction - nothing about running 90 tests changes the conclusion that COMPAS's baseline demographic parity gap by race is real. The last row was never significant to begin with. The six rows in between are exactly the point: identical raw p-values read as five different verdicts depending on which question you ask ("no false positives anywhere" vs. "a controlled fraction of my discoveries are false"), and two of them - both around p = 0.038-0.040 - are "significant" under the repo's current uncorrected reporting but wouldn't survive even the most permissive correction here.
Detection Code
A from-scratch implementation of all three corrections, verified against statsmodels.stats.multitest.multipletests on the exact 90-row COMPAS/race slice above (identical significant-count results: 53 Bonferroni, 55 Holm, 64 BH-FDR):
diff --git a/explainers/neural-networks.html b/explainers/neural-networks.html
index a2764190..22b996c9 100644
--- a/explainers/neural-networks.html
+++ b/explainers/neural-networks.html
@@ -284,7 +284,7 @@ What We Did
'Technical_Test_Score'
]Results:
-| Group | Hire Rate |
|---|---|
| Male candidates | 21.62% |
| Female candidates | 17.59% |
| Fairness gap | 4.03 percentage points |
| Group | Hire Rate |
|---|---|
| Male candidates | 21.62% |
| Female candidates | 17.59% |
| Fairness gap | 4.03 percentage points |
The network didn't contain a rule that said "prefer men." It learned from historical hiring data in which men were hired more. The weights encoded that pattern. The bias was invisible - buried in floating-point numbers across hidden layers.
Step 2 - Remove gender + proxy (our fix):
@@ -296,7 +296,7 @@Result: the fairness gap closes to 0.16 percentage points.
| Approach | Fairness Gap | Reduction |
|---|---|---|
| Biased model | 4.03% | - |
| Remove gender only (keep Age) | 0.23% | 94% |
| Remove gender + proxy | 0.16% | 96.1% |
| Approach | Fairness Gap | Reduction |
|---|---|---|
| Biased model | 4.03% | - |
| Remove gender only (keep Age) | 0.23% | 94% |
| Remove gender + proxy | 0.16% | 96.1% |
The network's architecture didn't change. The training procedure didn't change. Only the inputs changed - and the bias disappeared.
In a fair system without structural barriers, healthcare spending would be directly proportional to health need (Y_cost proportional to Y*) across all demographic groups. In reality, historical healthcare expenditures exhibit severe racial disparity at equal levels of illness:
Expected_Cost(Race = Black, Illness = k) < Expected_Cost(Race = White, Illness = k)Because less money is spent caring for Black patients at any given illness level k, an algorithm trained to predict Y_cost learns a biased spending score. When the algorithm ranks patients by predicted risk to enroll the top 3% (or 5%) into specialized care programs:
| Metric at Shared Enrollment Score Threshold | White Patients | Black Patients | Structural Disparity |
|---|---|---|---|
| Predicted Healthcare Cost | Equal | Equal | Algorithm appears calibrated on cost |
| Actual Chronic Conditions Count | Baseline | ~28% Higher | Black patients are substantially sicker |
| Biomedical Biomarkers (e.g., HbA1c, BP) | Baseline | Significantly Worse | Black patients have worse physiological health |
| Care Program Auto-Enrollment Rate | Baseline | Substantially Reduced | Sicker Black patients are systematically bypassed |
| Metric at Shared Enrollment Score Threshold | White Patients | Black Patients | Structural Disparity |
|---|---|---|---|
| Predicted Healthcare Cost | Equal | Equal | Algorithm appears calibrated on cost |
| Actual Chronic Conditions Count | Baseline | ~28% Higher | Black patients are substantially sicker |
| Biomedical Biomarkers (e.g., HbA1c, BP) | Baseline | Significantly Worse | Black patients have worse physiological health |
| Care Program Auto-Enrollment Rate | Baseline | Substantially Reduced | Sicker Black patients are systematically bypassed |
The algorithm is not broken in a mathematical sense - it predicts future spending with high accuracy for both groups. The failure lies in the semantic gap between the proxy label (Y_cost) and the human goal (Y*).
In 2019, Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan published their landmark study in Science, auditing a commercial risk-prediction algorithm applied to over 200 million patients annually across major US health systems.
diff --git a/explainers/precision-recall-curve.html b/explainers/precision-recall-curve.html index 2209a834..e28bf248 100644 --- a/explainers/precision-recall-curve.html +++ b/explainers/precision-recall-curve.html @@ -199,8 +199,8 @@Every point on a PR curve is one threshold, same as ROC - it just plots different axes:
| What AP/PR tells you | What AP/PR does not tell you |
|---|---|
| How well precision holds up as you demand more recall | Whether a "good" AP is good relative to the base rate - always compare against it |
| The trade-off you actually face when tuning a threshold under class imbalance | Anything about true negatives - it never rewards you for correctly ignoring the majority class |
| A picture that gets harder to look good on as positives get rarer | Group-level gaps - like AUC, it needs to be computed per group to see them |
| Base rate (rare positives) | ROC/AUC | PR/AP |
|---|---|---|
| What moves | False positive rate barely moves - true negatives dominate the denominator | Precision moves a lot - false positives are a large share of a small "flagged positive" pool |
| Random baseline | Always 0.5, regardless of base rate | Equal to the base rate - drops toward 0 as positives get rarer |
| What a "good" score hides | A ranker that's mediocre specifically among the flagged cases | Nothing extra beyond precision/recall themselves - which is the point |
| What AP/PR tells you | What AP/PR does not tell you |
|---|---|
| How well precision holds up as you demand more recall | Whether a "good" AP is good relative to the base rate - always compare against it |
| The trade-off you actually face when tuning a threshold under class imbalance | Anything about true negatives - it never rewards you for correctly ignoring the majority class |
| A picture that gets harder to look good on as positives get rarer | Group-level gaps - like AUC, it needs to be computed per group to see them |
| Base rate (rare positives) | ROC/AUC | PR/AP |
|---|---|---|
| What moves | False positive rate barely moves - true negatives dominate the denominator | Precision moves a lot - false positives are a large share of a small "flagged positive" pool |
| Random baseline | Always 0.5, regardless of base rate | Equal to the base rate - drops toward 0 as positives get rarer |
| What a "good" score hides | A ranker that's mediocre specifically among the flagged cases | Nothing extra beyond precision/recall themselves - which is the point |
Audit 06 predicts 30-day hospital readmission from the Diabetes 130-US Hospitals dataset (101,766 records). Readmission is rare: only 11.2% of patients in the dataset were readmitted within 30 days - the other 88.8% were not.
The baseline logistic regression model's frozen performance numbers (paper/results-frozen/results_performance.csv) look reasonable at a glance: 88.7% accuracy and an AUC of 0.62. Its F1 score - the harmonic mean of precision and recall - is 0.039. F1 stays low because AUC never had to reckon with the base rate: sweeping every threshold shows the model can rank cases somewhat sensibly (AUC 0.62, better than a coin flip), but at the threshold that actually gets deployed, the number of false positives it generates is large relative to the tiny pool of true positives, so precision collapses. The random forest and gradient boosting baselines show the same pattern (AUC 0.65 and 0.67, F1 of 0.021 and 0.024) - this isn't one unlucky model family, it's the base rate.
The non-obvious part is that "trustworthy for everyone" is not the same fairness property as "fair to everyone." A score can satisfy predictive parity perfectly and still send one group to jail on false pretenses far more often than another. Which group bears that cost depends on the false positive rate, a completely different metric, and the two cannot both be balanced unless the underlying base rates already match.
Predictive parity is a sufficiency metric: it conditions on the prediction and asks whether that prediction means the same thing across groups. Equalized Odds (see explainers/equalized-odds.md) is a separation metric: it conditions on the true outcome and asks whether error rates match.
-| Metric | Conditions on | Question it asks |
|---|---|---|
| Demographic Parity | nothing (prior) | Are positive predictions handed out at the same rate? |
| Equalized Odds | true label | Given the actual outcome, are error rates equal? |
| Predictive Parity | predicted label | Given a positive prediction, is it equally trustworthy? |
| Metric | Conditions on | Question it asks |
|---|---|---|
| Demographic Parity | nothing (prior) | Are positive predictions handed out at the same rate? |
| Equalized Odds | true label | Given the actual outcome, are error rates equal? |
| Predictive Parity | predicted label | Given a positive prediction, is it equally trustworthy? |
PPV(group) = TP(group) / (TP(group) + FP(group))
Predictive Parity holds when: PPV(group a) ≈ PPV(group b)The sharpest illustration of proxy entanglement in this repository is the Healthcare Readmission audit - the most structurally complex of all seven audits.
The dataset is diabetic_data.csv, drawn from 130 US hospitals between 1999 and 2008 (101,766 records). The model predicts whether a diabetic patient will be readmitted within 30 days and flags them as high clinical risk - a decision that directly affects discharge planning, post-acute care allocation, and follow-up resource assignment.
The protected attributes are race, gender, and age. The biased model (unfair.py) trained directly on all three alongside the full feature set, producing these gaps:
| Group | High-Risk Flag Rate |
|---|---|
| Caucasian / Asian | 0.25% |
| Other minorities | 0.17% |
| Fairness Gap (Race) | 0.08% |
| Group | High-Risk Flag Rate |
|---|---|
| Under 70 | 0.36% |
| 70+ (elderly) | 0.08% |
| Fairness Gap (Age) | 0.28% |
| Group | High-Risk Flag Rate |
|---|---|
| Caucasian / Asian | 0.25% |
| Other minorities | 0.17% |
| Fairness Gap (Race) | 0.08% |
| Group | High-Risk Flag Rate |
|---|---|
| Under 70 | 0.36% |
| 70+ (elderly) | 0.08% |
| Fairness Gap (Age) | 0.28% |
Removing race and age alone is not enough. Four proxy variables - payer_code, discharge_disposition_id, medical_specialty, and number_inpatient - form an entangled cluster, each encoding race and income through a different administrative channel, all tracing back to the same structural cause.
# Each proxy correlates with race through a different administrative pathway
@@ -254,7 +254,7 @@ The Fix: Remove the Entire Cluster
| Gap | Before | After | Reduction |
|---|---|---|---|
| Race | 0.08% | 0.06% | 25% |
| Age | 0.28% | 0.09% | 68% |
| Gap | Before | After | Reduction |
|---|---|---|---|
| Race | 0.08% | 0.06% | 25% |
| Age | 0.28% | 0.09% | 68% |
Removing the entire entangled cluster - rather than any single variable - is what produces the reduction. Dropping only payer_code and leaving discharge_disposition_id, medical_specialty, and number_inpatient in place would have left most of the race signal intact, because those three remaining variables still encode the same structural inequality through different administrative columns.
diff --git a/explainers/proxy-variables.html b/explainers/proxy-variables.html index 6e1b69fd..c82a716e 100644 --- a/explainers/proxy-variables.html +++ b/explainers/proxy-variables.html @@ -201,7 +201,7 @@Key insight:
payer_code,discharge_disposition_id,medical_specialty, andnumber_inpatientare not four independent proxies. They are four administrative measurements of the same underlying structural gap in US healthcare access. The causal direction matters: lower insurance coverage creates both restricted discharge destinations and fragmented inpatient histories. The patient does not bring the disparity - the system creates it, and then measures it four times. Removing only one measurement while leaving the others is not mitigation; it is relabelling.
This is called proxy discrimination, and it's one of the hardest problems in algorithmic fairness.
| Feature | Protected Attribute It Encodes | Why |
|---|---|---|
| Zip code | Race | Historical redlining segregated neighborhoods by race. Zip codes still reflect this. |
| Custody status | Race | Over-policing of Black communities leads to disproportionate pretrial detention. |
| Name | Gender / Ethnicity | Names encode gender and ethnicity with high accuracy. |
| College attended | Socioeconomic background | Elite colleges remain heavily stratified by class and race. |
| Credit score | Race / Class | Credit histories reflect historical exclusion from banking. |
| Prior arrests | Race | Arrest records reflect over-policing, not actual crime rates. |
| Feature | Protected Attribute It Encodes | Why |
|---|---|---|
| Zip code | Race | Historical redlining segregated neighborhoods by race. Zip codes still reflect this. |
| Custody status | Race | Over-policing of Black communities leads to disproportionate pretrial detention. |
| Name | Gender / Ethnicity | Names encode gender and ethnicity with high accuracy. |
| College attended | Socioeconomic background | Elite colleges remain heavily stratified by class and race. |
| Credit score | Race / Class | Credit histories reflect historical exclusion from banking. |
| Prior arrests | Race | Arrest records reflect over-policing, not actual crime rates. |
We tested this directly using the ProPublica COMPAS dataset.
@@ -217,7 +217,7 @@Results:
-| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 87.16% |
| White defendants | 0.40% |
| Fairness gap | 86.77% |
| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 87.16% |
| White defendants | 0.40% |
| Fairness gap | 86.77% |
Step 2 - Remove race only (naive approach):
Many developers stop here. They drop race_binary and assume the model is now fair. Dropping race alone already takes the gap from 86.77% down to 18.38% - most of the signal was riding on race directly. But it isn't fully fixed: CustodyStatus is still acting as a racial proxy, and its own marginal contribution beyond race is the remaining ~2.7 points down to 15.69% once it's removed too (see Step 3).
Results:
-| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 84.71% |
| White defendants | 69.02% |
| Fairness gap | 15.69% |
| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 84.71% |
| White defendants | 69.02% |
| Fairness gap | 15.69% |
| Approach | Fairness Gap | Reduction |
|---|---|---|
| Biased model | 86.77% | - |
| Remove race only | 18.38% | 79% |
| Remove race + proxy | 15.69% | 82% |
| Approach | Fairness Gap | Reduction |
|---|---|---|
| Biased model | 86.77% | - |
| Remove race only | 18.38% | 79% |
| Remove race + proxy | 15.69% | 82% |
Removing the protected attribute alone is not enough. You must audit every feature for correlation with protected attributes.
To understand how a race multiplier shifts patients across clinical thresholds, consider a sample of 1,000 Black patients presenting with elevated serum creatinine (1.3 to 1.8 mg/dL).
When evaluated using the 2009 CKD-EPI equation, applying the 1.212 Black race multiplier inflates eGFR scores across the diagnostic boundary (60 mL/min/1.73m²):
-| Metric | Without Race Multiplier (Unadjusted) | With 1.212 Race Multiplier (Race-Corrected) | Impact of Race Correction |
|---|---|---|---|
| Average eGFR Score | 53.4 mL/min/1.73m² | 64.7 mL/min/1.73m² | Inflated by +11.3 mL/min/1.73m² |
| Classified as CKD (eGFR < 60) | 640 patients (64.0%) | 410 patients (41.0%) | 230 patients (23.0%) denied CKD diagnosis |
| Eligible for Transplant List (eGFR < 20) | 85 patients (8.5%) | 42 patients (4.2%) | 43 patients (4.3%) delayed from transplant list |
| Metric | Without Race Multiplier (Unadjusted) | With 1.212 Race Multiplier (Race-Corrected) | Impact of Race Correction |
|---|---|---|---|
| Average eGFR Score | 53.4 mL/min/1.73m² | 64.7 mL/min/1.73m² | Inflated by +11.3 mL/min/1.73m² |
| Classified as CKD (eGFR < 60) | 640 patients (64.0%) | 410 patients (41.0%) | 230 patients (23.0%) denied CKD diagnosis |
| Eligible for Transplant List (eGFR < 20) | 85 patients (8.5%) | 42 patients (4.2%) | 43 patients (4.3%) delayed from transplant list |
The race multiplier hides real kidney impairment in 23% of patients, treating them as healthy on paper while their kidney function declines.
Below are two modular Python functions:
diff --git a/explainers/reinforcement-learning.html b/explainers/reinforcement-learning.html index 4fe47aac..93d74c65 100644 --- a/explainers/reinforcement-learning.html +++ b/explainers/reinforcement-learning.html @@ -351,7 +351,7 @@Results - biased policy (race + custody_status included):
-| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 87.16% |
| White defendants | 0.40% |
| Fairness gap | 86.77% |
| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 87.16% |
| White defendants | 0.40% |
| Fairness gap | 86.77% |
The policy did not contain a rule that said "flag Black defendants." It learned from historical reoffense records in which policing patterns, pretrial detention rates, and charge severity were all racially stratified. The reward signal encoded that stratification. The policy reproduced it.
Results - mitigated policy (race + proxy removed):
-| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 84.71% |
| White defendants | 69.02% |
| New fairness gap | 15.69% |
| Approach | Fairness Gap | Reduction |
|---|---|---|
| Biased policy | 86.77% | - |
| Remove race only (keep custody_status) | 18.38% | 79% |
| Remove race + custody_status | 15.69% | 82% |
| Group | High-Risk Flag Rate |
|---|---|
| Black defendants | 84.71% |
| White defendants | 69.02% |
| New fairness gap | 15.69% |
| Approach | Fairness Gap | Reduction |
|---|---|---|
| Biased policy | 86.77% | - |
| Remove race only (keep custody_status) | 18.38% | 79% |
| Remove race + custody_status | 15.69% | 82% |
The policy architecture did not change. The training procedure did not change. Only the state representation changed - and most of the discriminatory behaviour disappeared.
If a bank historically required younger applicants to meet a higher credit bar than older applicants, then the younger applicants present in the approved dataset (S = 1) represent an artificially selected, ultra-qualified subset of all young applicants. A model trained on this sample will overestimate the credit standards required for young borrowers to succeed.
Practitioners use four main statistical approaches to correct for reject inference:
-| Method | Core Mechanism | Strengths | Key Vulnerability |
|---|---|---|---|
| Hard Parceling (Pseudo-Labeling) | Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows. | Simple to implement in standard ML pipelines. | Propagates initial model errors and thresholding artifacts into retraining. |
| Soft Parceling / Fuzzy Augmentation | Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases. | Avoids hard threshold cutoffs; preserves prediction uncertainty. | Dilutes training signal if initial model probability estimates are miscalibrated. |
| Inverse Probability Weighting (IPW) | Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance. |
| Heckman Two-Stage Model | Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε). | Explicitly models unobserved selection correlation ρ. | Relies heavily on bivariate normality and valid exclusion restrictions (Z). |
| Method | Core Mechanism | Strengths | Key Vulnerability |
|---|---|---|---|
| Hard Parceling (Pseudo-Labeling) | Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows. | Simple to implement in standard ML pipelines. | Propagates initial model errors and thresholding artifacts into retraining. |
| Soft Parceling / Fuzzy Augmentation | Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases. | Avoids hard threshold cutoffs; preserves prediction uncertainty. | Dilutes training signal if initial model probability estimates are miscalibrated. |
| Inverse Probability Weighting (IPW) | Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance. |
| Heckman Two-Stage Model | Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε). | Explicitly models unobserved selection correlation ρ. | Relies heavily on bivariate normality and valid exclusion restrictions (Z). |
German Credit Lending/credit_customers.csv is the dataset behind Audit 03 in this repository. Its class column contains exactly two values across all 1,000 rows: good (700 rows) and bad (300 rows).
Not the same as Reject Inference. That explainer's "reject" is about rejected loan applicants who never entered the training data - a selection-bias problem in credit scoring. This explainer's "reject" is the classifier's reject option: the band of inputs a probabilistic model is too uncertain about to commit to. Same word, unrelated ideas. (The same name-collision warning Counterfactual Explanation carries for its overlap with counterfactual fairness.)
This repo's benchmark harness already includes a post-processing mitigation as strategy S4 - Fairlearn's ThresholdOptimizer, which searches for a per-group decision threshold that makes the whole model satisfy a fairness constraint. ROC is a different post-processing mechanism with the same access level (model outputs only, no retraining):
ThresholdOptimizer (this repo's S4) | Reject Option Classification | |
|---|---|---|
| What it changes | one decision threshold per group, applied to every prediction | the label of individual predictions, only inside a confidence band |
| Where it acts | globally, across the whole score range | locally, near p = 0.5 |
| Free parameter | the fairness constraint to satisfy | the width of the critical region |
| Rationale | pick thresholds that equalize a chosen rate | flip where the model had the least evidence, so accuracy loss is smallest |
ThresholdOptimizer (this repo's S4) | Reject Option Classification | |
|---|---|---|
| What it changes | one decision threshold per group, applied to every prediction | the label of individual predictions, only inside a confidence band |
| Where it acts | globally, across the whole score range | locally, near p = 0.5 |
| Free parameter | the fairness constraint to satisfy | the width of the critical region |
| Rationale | pick thresholds that equalize a chosen rate | flip where the model had the least evidence, so accuracy loss is smallest |
ROC's appeal is the "least evidence" argument: a prediction with p = 0.52 is barely a prediction at all, so overriding it toward the disadvantaged group is a small, defensible correction. Its weakness is that "how wide is the band" has no principled answer - and, as the worked example shows, the answer completely determines whether you fix the gap, do nothing, or overcorrect into a mirror-image disparity.
Let p_i = P(Y = 1 | X_i) be the model's predicted probability of the unfavorable outcome (here: "high risk"), and let Y = 1 be unfavorable, Y = 0 favorable.
COMPAS/ audits the ProPublica COMPAS raw file, filtered (per audit.yaml) to African-American and Caucasian defendants scored for "Risk of Recidivism" (~16.3k rows). Following the audit's setup, a logistic-regression baseline was trained on Sex_Code_Text, race, CustodyStatus, MaritalStatus (an 80/20 split, random_state=42), and ROC was applied to its test-set probabilities. Favorable outcome = "not high risk"; disadvantaged group = African-American defendants.
Baseline model (no ROC):
-| Value | |
|---|---|
| Black defendants flagged high-risk | 86.1% |
| White defendants flagged high-risk | 0.1% |
| Demographic parity gap | +86.0 pp |
| Test accuracy | 65.9% |
| Value | |
|---|---|
| Black defendants flagged high-risk | 86.1% |
| White defendants flagged high-risk | 0.1% |
| Demographic parity gap | +86.0 pp |
| Test accuracy | 65.9% |
The baseline is close to a step function - race dominates the model, so almost every predicted probability sits near 0 or near 1 and very few land near the boundary. That is the setting ROC handles worst, and it shows:
Critical region theta | Predictions in region (of 3,254) | Predictions flipped | New parity gap | Test accuracy |
|---|---|---|---|---|
| baseline | - | - | +86.0 pp | 65.9% |
| ±0.05 | 17 | 17 | +85.0 pp | 65.9% |
| ±0.10 | 671 | 651 | +43.9 pp | 61.0% |
| ±0.15 | 2,609 | 2,566 | -70.2 pp | 45.5% |
Critical region theta | Predictions in region (of 3,254) | Predictions flipped | New parity gap | Test accuracy |
|---|---|---|---|---|
| baseline | - | - | +86.0 pp | 65.9% |
| ±0.05 | 17 | 17 | +85.0 pp | 65.9% |
| ±0.10 | 671 | 651 | +43.9 pp | 61.0% |
| ±0.15 | 2,609 | 2,566 | -70.2 pp | 45.5% |
No value of theta was "correct"; the three rows above are three different fairness outcomes produced by three arbitrary choices of one number. That is the central caveat, made concrete.
import numpy as np
diff --git a/explainers/roc-curve-auc.html b/explainers/roc-curve-auc.html
index f749937a..df3fa23d 100644
--- a/explainers/roc-curve-auc.html
+++ b/explainers/roc-curve-auc.html
@@ -202,8 +202,8 @@ Reading a ROC Curve
Every point on a ROC curve is one threshold. As you sweep the threshold from strict to lenient, you trace the curve from the bottom-left to the top-right:
- The y-axis is the true positive rate (sensitivity, recall) - of the people who really are positive, what fraction did the model catch.
- The x-axis is the false positive rate (1 - specificity) - of the people who really are negative, what fraction did the model wrongly flag.
- The diagonal line from corner to corner is a coin flip: AUC 0.5, no ranking ability at all.
- The top-left corner is a perfect classifier: every positive caught, no negatives flagged, AUC 1.0.
A curve that bows toward the top-left is a better ranker. The area underneath it is the AUC.
-AUC Rough interpretation 0.5 No better than random guessing 0.6 - 0.7 Weak - only slightly better than a coin flip 0.7 - 0.8 Acceptable for many applied settings 0.8 - 0.9 Strong ranking ability > 0.9 Excellent - or a sign of data leakage, worth checking
-What AUC tells you What AUC does not tell you How well the model orders cases by risk Where you should set the decision threshold A threshold-independent summary of ranking How the model behaves at the threshold you deploy One number to compare models at a glance Whether ranking quality is equal across groups Robustness to class imbalance Whether the scores are calibrated or comparable between groups
+AUC Rough interpretation 0.5 No better than random guessing 0.6 - 0.7 Weak - only slightly better than a coin flip 0.7 - 0.8 Acceptable for many applied settings 0.8 - 0.9 Strong ranking ability > 0.9 Excellent - or a sign of data leakage, worth checking
+What AUC tells you What AUC does not tell you How well the model orders cases by risk Where you should set the decision threshold A threshold-independent summary of ranking How the model behaves at the threshold you deploy One number to compare models at a glance Whether ranking quality is equal across groups Robustness to class imbalance Whether the scores are calibrated or comparable between groups
Concrete Example: COMPAS - Audit 01
Audit 01 in this repo models COMPAS's own high/medium risk label from the ProPublica dataset (3,254 test records), with race as the protected attribute; it does not predict an independent two-year recidivism outcome. Its baseline models are ordinary, unremarkable rankers: the logistic-regression baseline lands at an AUC of 0.68, and the random forest at 0.69 (frozen figures from paper/results-frozen/summary.csv and results_performance.csv).
That number sits squarely in the "weak" band - the model barely out-ranks a coin flip. But the documented harm in COMPAS was never really about overall ranking quality. It was about what happened at the operating threshold: Black defendants who did not go on to reoffend were labeled high-risk at nearly twice the rate of comparable white defendants. That is a false-positive-rate gap - a single point on the ROC curve, read separately for each group - and the aggregate AUC of 0.68 averages straight over it.
diff --git a/explainers/sampling-bias.html b/explainers/sampling-bias.html
index 9ceee549..8b1a5863 100644
--- a/explainers/sampling-bias.html
+++ b/explainers/sampling-bias.html
@@ -201,12 +201,12 @@ Why This Matters
This is called sampling bias, and it shows up in facial recognition systems that fail on darker skin tones, clinical tools that miss diagnoses in women, and NLP models that perform worse on dialects outside Standard American English. In each case, the data collection process systematically excluded certain groups - and the model inherited that exclusion.
Common Forms of Sampling Bias
-Form What It Means Real Example Underrepresentation A demographic group appears far less in training data than in the real world Facial recognition trained mostly on light-skinned faces Selection bias Data is collected from a non-random subset of the population Clinical AI trained on hospital records from affluent ZIP codes Historical bias Past data reflects past inequities; the model learns the inequity, not the truth Hiring models trained on 10 years of male-dominated engineering hires Survivorship bias Only "successful" cases make it into the dataset; failures are invisible Loan models trained only on approved applicants, never denied ones Temporal bias Training data is from a time period that no longer reflects current reality A model trained on 2005 internet text used today Measurement bias The same underlying reality is measured differently for different groups Over-policing means more arrest records for Black communities - not higher crime rates
+Form What It Means Real Example Underrepresentation A demographic group appears far less in training data than in the real world Facial recognition trained mostly on light-skinned faces Selection bias Data is collected from a non-random subset of the population Clinical AI trained on hospital records from affluent ZIP codes Historical bias Past data reflects past inequities; the model learns the inequity, not the truth Hiring models trained on 10 years of male-dominated engineering hires Survivorship bias Only "successful" cases make it into the dataset; failures are invisible Loan models trained only on approved applicants, never denied ones Temporal bias Training data is from a time period that no longer reflects current reality A model trained on 2005 internet text used today Measurement bias The same underlying reality is measured differently for different groups Over-policing means more arrest records for Black communities - not higher crime rates
Real-World Proof: MIT Gender Shades
The clearest documented case of sampling bias in AI is the Gender Shades study (Joy Buolamwini & Timnit Gebru, 2018).
Three commercial facial recognition systems were tested on a dataset specifically balanced across gender and skin tone. The results showed that every system performed dramatically worse on darker-skinned women - not because the algorithms were explicitly programmed to fail them, but because the training datasets were overwhelmingly composed of lighter-skinned faces.
-Group Error Rate (best system) Error Rate (worst system) Lighter-skinned males ~0.8% ~3.4% Lighter-skinned females ~1.7% ~7.1% Darker-skinned males ~10.8% ~16.0% Darker-skinned females ~20.8% ~34.7%
+Group Error Rate (best system) Error Rate (worst system) Lighter-skinned males ~0.8% ~3.4% Lighter-skinned females ~1.7% ~7.1% Darker-skinned males ~10.8% ~16.0% Darker-skinned females ~20.8% ~34.7%
The gap between best-case (lighter males) and worst-case (darker females) was up to 34 percentage points - from the same model, on the same task, because of who was and wasn't in the training data.
The fix was not a new algorithm. It was a more representative dataset.
diff --git a/explainers/shap-values.html b/explainers/shap-values.html
index 8b0d6896..1a524a12 100644
--- a/explainers/shap-values.html
+++ b/explainers/shap-values.html
@@ -336,7 +336,7 @@ Bias Audit Using SHAP
Sex_Code_Text_Male 0.006635| Method | Scope | Consistency | Model-Agnostic | Speed |
|---|---|---|---|---|
| SHAP | Local + Global | ✓ Mathematically guaranteed | ✓ Yes | Medium–Fast |
| LIME | Local only | ✗ Approximate | ✓ Yes | Slow |
| Feature Importance (Gini) | Global only | ✗ Biased toward high-cardinality | ✗ Tree models only | Fast |
| Permutation Importance | Global only | Reasonable | ✓ Yes | Medium |
| Method | Scope | Consistency | Model-Agnostic | Speed |
|---|---|---|---|---|
| SHAP | Local + Global | ✓ Mathematically guaranteed | ✓ Yes | Medium–Fast |
| LIME | Local only | ✗ Approximate | ✓ Yes | Slow |
| Feature Importance (Gini) | Global only | ✗ Biased toward high-cardinality | ✗ Tree models only | Fast |
| Permutation Importance | Global only | Reasonable | ✓ Yes | Medium |
SHAP is the only method that is locally accurate (values sum exactly to the prediction), consistent (if a feature matters more, its SHAP value will be higher), and unified across model types. For fairness auditing specifically, local accuracy is essential - you need to know what drove this specific decision, not just what matters on average.
Two things vary between groups in that sum: the within-stratum rates P(Y = 1 | G = g, S = s), and the stratum weights P(S = s | G = g). The aggregate gap between two groups blends both. If the weights differ enough, the weight term can dominate and the aggregate gap stops reflecting the within-stratum rates at all.
Consider a hiring screen with two departments. Within each department the callback rate for Group A is higher than for Group B:
-| Department | Group A callbacks | Group B callbacks |
|---|---|---|
| Engineering (easy to pass) | 70 / 80 = 87.5% | 18 / 20 = 90.0% |
| Design (hard to pass) | 2 / 20 = 10.0% | 8 / 80 = 10.0% |
| Pooled | 72 / 100 = 72.0% | 26 / 100 = 26.0% |
| Department | Group A callbacks | Group B callbacks |
|---|---|---|
| Engineering (easy to pass) | 70 / 80 = 87.5% | 18 / 20 = 90.0% |
| Design (hard to pass) | 2 / 20 = 10.0% | 8 / 80 = 10.0% |
| Pooled | 72 / 100 = 72.0% | 26 / 100 = 26.0% |
Within Engineering, Group B does slightly better (90.0% vs 87.5%); within Design the two are tied (10.0% vs 10.0%). Neither department favors Group A. Yet the pooled rate shows Group A ahead by 46 points, because 80% of Group A's applicants are in the high-callback department and 80% of Group B's are in the low-callback one. The pooled 46-point "gap" is entirely a fact about where each group applied, not about how either group was treated inside a department.
Benefits Denial/ audits the Adult Census Income dataset (adult.csv, 32,561 rows), where the target is income == '>50K' and sex is a declared protected attribute. These are the dataset's own outcome rates (not a model's predictions), which is the base rate a demographic-parity check on the labels compares.
Aggregate:
-| Group | n | P(income > 50K) |
|---|---|---|
| Male | 21,790 | 30.6% |
| Female | 10,771 | 10.9% |
| Gap (M - F) | +19.6 pp |
| Group | n | P(income > 50K) |
|---|---|---|
| Male | 21,790 | 30.6% |
| Female | 10,771 | 10.9% |
| Gap (M - F) | +19.6 pp |
Now stratify by marital.status. The single largest stratum, Married-civ-spouse, holds 14,976 rows - 46% of the entire dataset:
| Stratum | Male n | Male rate | Female n | Female rate | Gap (M - F) |
|---|---|---|---|---|---|
| Married-civ-spouse | 13,319 | 44.6% | 1,657 | 45.5% | -0.9 pp |
| Never-married | 5,916 | 5.5% | 4,767 | 3.5% | +2.0 pp |
| Divorced | 1,771 | 16.0% | 2,672 | 6.7% | +9.3 pp |
| Widowed | 168 | 23.2% | 825 | 5.6% | +17.6 pp |
| Separated | 394 | 12.4% | 631 | 2.7% | +9.7 pp |
| Stratum | Male n | Male rate | Female n | Female rate | Gap (M - F) |
|---|---|---|---|---|---|
| Married-civ-spouse | 13,319 | 44.6% | 1,657 | 45.5% | -0.9 pp |
| Never-married | 5,916 | 5.5% | 4,767 | 3.5% | +2.0 pp |
| Divorced | 1,771 | 16.0% | 2,672 | 6.7% | +9.3 pp |
| Widowed | 168 | 23.2% | 825 | 5.6% | +17.6 pp |
| Separated | 394 | 12.4% | 631 | 2.7% | +9.7 pp |
Collapsed to just married vs not-married, the gap shrinks from +19.6 pp to +3.0 pp (married) and +3.9 pp (not married). And inside Married-civ-spouse - nearly half the data - the sign reverses: women are 0.9 points ahead.
The mechanism is the stratum weights. Women are 52% of the "not married" pool, which has an overall high-income rate of 6.5%, but only 12% of the "married" pool, which has an overall rate of 43.7%. The aggregate comparison places most men in the high-earning stratum and splits women toward the low-earning one, so roughly 16 of the 19.6 aggregate points come from where each group sits, not from a within-stratum difference in outcome rates.
Whether marital.status is a legitimate thing to condition on here is a separate, normative question - see Conditional Demographic Parity and the Limitations section below. The point Simpson's Paradox makes is narrower: the +19.6 pp aggregate figure, reported on its own, describes the dataset's composition at least as much as it describes any disparity in outcomes.
Healthcare Readmission/ audits the UCI diabetes 130-hospitals dataset (diabetic_data.csv, 101,766 encounters). The target is readmitted == '<30' (readmission within 30 days), and gender, race, and age are declared protected attributes. Using the dataset's own readmission rates (the base rate a label-level demographic parity check compares):
Marginal check on gender - passes:
| Group | n | Readmit < 30 |
|---|---|---|
| Female | 54,708 | 11.25% |
| Male | 47,055 | 11.06% |
| Gap | 0.18 pp (ratio 0.984) |
| Group | n | Readmit < 30 |
|---|---|---|
| Female | 54,708 | 11.25% |
| Male | 47,055 | 11.06% |
| Gap | 0.18 pp (ratio 0.984) |
A 0.18-point gap clears the EEOC four-fifths rule (ratio 0.984, well above 0.80) and any reasonable demographic parity tolerance. On the gender axis alone, this dataset looks clean.
Now gender x race - the gender gap inside each race stratum:
| Race | Female rate | Male rate | Gender gap (F - M) | Subgroup n |
|---|---|---|---|---|
| Caucasian | 11.49% | 11.07% | +0.42 pp | 76,099 |
| AfricanAmerican | 11.08% | 11.43% | -0.34 pp | 19,210 |
| Hispanic | 9.16% | 11.85% | -2.69 pp | 2,037 |
| Asian | 7.55% | 12.69% | -5.15 pp | 641 |
| Race | Female rate | Male rate | Gender gap (F - M) | Subgroup n |
|---|---|---|---|---|
| Caucasian | 11.49% | 11.07% | +0.42 pp | 76,099 |
| AfricanAmerican | 11.08% | 11.43% | -0.34 pp | 19,210 |
| Hispanic | 9.16% | 11.85% | -2.69 pp | 2,037 |
| Asian | 7.55% | 12.69% | -5.15 pp | 641 |
Among Asian patients, women are readmitted at 7.55% and men at 12.69% - a 5.15-point gap and a four-fifths ratio of 0.595, a clear violation, pointing the opposite direction from the (negligible) marginal gender gap. Among Hispanic patients the gender gap is -2.69 pp, also a reversal and also larger than the marginal. The gender guarantee that held at the population level does not hold once you condition on race, and no single-attribute check would have surfaced that.
Two honest caveats, both of which the Limitations section expands:
min_group_size of 100, but small enough that a 5-point rate is not a precise estimate - exactly the case this repo's small-sample warnings exist for.The German Credit Lending audit is a useful second example because its label is a different kind of "ground truth." Where Hiring_Decision in the recruitment dataset is a direct historical human decision, the credit dataset's class (good/bad credit) label is closer to an observed outcome. Both are still supervised learning in exactly the same structural sense - features in, label out, mapping learned - and both still let a protected attribute back in through a correlated feature. In lending it is employment tenure standing in for age; in hiring, historical hiring patterns stand in for gender. The mechanism does not care what kind of label it is given. It treats a human decision and an observed outcome identically: as ground truth to be reproduced.
The AI Fair Recruitment dataset pairs applicant features (gender, age, years of experience, technical test score) with a historical Hiring_Decision label. Trained on all four features, the model reproduced a hiring gap that closely tracked gender rather than the merit-based inputs alone.
| Group | Hire Rate |
|---|---|
| Men | 21.62% |
| Women | 17.59% |
| Fairness Gap | 4.03% |
| Group | Hire Rate |
|---|---|
| Men | 21.62% |
| Women | 17.59% |
| Fairness Gap | 4.03% |
The model was never given a rule about gender. It inferred the gap because a gap already existed in the training labels, and inferring existing patterns is exactly what model.fit() does.
# THE FIX: Merit only
X = df[['Experience_Years', 'Technical_Test_Score']]
# gender removed ✓
# age removed ✓Retraining on experience and test score alone - the two features that plausibly measure job-relevant merit - collapsed the gap:
-| Group | Hire Rate |
|---|---|
| Men | 11.48% |
| Women | 11.32% |
| New Fairness Gap | 0.16% |
| Group | Hire Rate |
|---|---|
| Men | 11.48% |
| Women | 11.32% |
| New Fairness Gap | 0.16% |
A 96.1% reduction, achieved without touching the learning algorithm at all. Same Random Forest, same fit() call, same random_state=42 - the only thing that changed was which columns the mapping was allowed to use.
The functions below train a supervised classifier on labeled data, then check whether the gap present in the original labels reappears in the model's own predictions on held-out data.
diff --git a/explainers/treatment-equality.html b/explainers/treatment-equality.html index 8335ce51..77e86a41 100644 --- a/explainers/treatment-equality.html +++ b/explainers/treatment-equality.html @@ -198,7 +198,7 @@None of faircode/benchmark.py's six computed metrics is a treatment-equality ratio, so this example is a from-scratch confusion-matrix computation on the same COMPAS baseline logistic regression model used throughout this repo's other explainers - the same precedent Individual Fairness uses, not a number pulled from paper/results-frozen/results_fairness.csv.
For the baseline model on race (disadvantaged: African-American defendants, n=1,788; advantaged: Caucasian defendants, n=1,466), the confusion matrix per group gives:
-| Group | False Negatives | False Positives | FN:FP Ratio |
|---|---|---|---|
| African-American (disadvantaged) | 73 | 575 | 0.13 |
| Caucasian (advantaged) | 465 | 15 | 31.00 |
| Group | False Negatives | False Positives | FN:FP Ratio |
|---|---|---|---|
| African-American (disadvantaged) | 73 | 575 | 0.13 |
| Caucasian (advantaged) | 465 | 15 | 31.00 |
(Positive-rate check against the frozen demographic_parity_diff for this exact model/split: (975+575)/1,788 = 86.7% vs. (2+15)/1,466 = 1.2%, an 85.5-point gap - matching paper/results-frozen/results_fairness.csv exactly, confirming this is the same baseline run.)
The two groups' individual false-negative and false-positive counts are the same kind of numbers Equalized Odds already compares - what Treatment Equality adds is the ratio between them, and here it is extreme in both directions at once. This baseline model predicts "high risk" for the overwhelming majority of African-American defendants and almost none of the Caucasian defendants (the 85.5-point demographic parity gap above), so its errors on each group are lopsided in opposite ways: African-American defendants' errors are almost entirely false positives (FN:FP of 0.13 - the model rarely clears someone it should have flagged, but very often flags someone it shouldn't), while Caucasian defendants' errors are almost entirely false negatives (FN:FP of 31.0 - the model almost never wrongly flags anyone, but very often wrongly clears someone). A 0.13-vs-31.0 ratio gap is not a subtle statistical artifact; it is the direct arithmetic consequence of a model that essentially never predicts "high risk" for one group and reports it as two completely different kinds of unreliable, one per group.
sex, race, and native.country never appear in feature_cols. The clustering step has no way to "see" them. Whatever split it produces has to come entirely from the other nine columns.
Running the code above on the UCI Adult Census Income dataset (32,561 records, the same file the Benefits Denial audit trains on) produces two clusters of size 17,680 and 14,881. Neither sex, race, nor native.country was part of the clustering input, so the composition of each cluster along those attributes was checked after the fact, not fed in beforehand:
| Cluster 0 (n=17,680) | Cluster 1 (n=14,881) | |
|---|---|---|
| Female | 51.9% | 10.7% |
| Male | 48.1% | 89.3% |
| Black | 13.0% | 5.6% |
| White | 81.9% | 89.6% |
| Foreign-born | 10.2% | 10.7% |
| Cluster 0 (n=17,680) | Cluster 1 (n=14,881) | |
|---|---|---|
| Female | 51.9% | 10.7% |
| Male | 48.1% | 89.3% |
| Black | 13.0% | 5.6% |
| White | 81.9% | 89.6% |
| Foreign-born | 10.2% | 10.7% |
Cluster 1 is 89.3% male against cluster 0's near-even split, and Black applicants appear at more than double the rate in cluster 0 versus cluster 1. Nobody asked the algorithm to separate applicants by sex or race. It separated them by relationship, marital.status, occupation, and hours.per.week, and a sex and race split came out the other side as a side effect, exactly the mechanism What Is a Proxy Variable? describes for supervised models, now showing up with no label in sight.
National origin is the counterexample worth keeping: foreign-born applicants land at essentially the same rate in both clusters (10.2% vs 10.7%). Not every protected attribute reappears in every clustering. Whether one does depends on how strongly it correlates with the specific features chosen and the number of clusters requested, which is precisely the fragility this explainer's limitations section covers next.