Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion assets/explainers-ui.js
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@
const headers = splitRow(rows[0]);
const bodyRows = rows.slice(2).map(splitRow);

const headerHtml = headers.map(cell => `<th>${inlineMarkdown(cell)}</th>`).join('');
const headerHtml = headers.map(cell => `<th scope="col">${inlineMarkdown(cell)}</th>`).join('');
const bodyHtml = bodyRows.map(row => `<tr>${row.map(cell => `<td>${inlineMarkdown(cell)}</td>`).join('')}</tr>`).join('');

return {
Expand Down
2 changes: 1 addition & 1 deletion explainers/accuracy-equality.html
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ <h3 id="why-it-matters">Why It Matters</h3>
<p>Accuracy Equality is one of the six fairness metrics this repo&#x27;s benchmark harness actually computes (<code>accuracy_equality_diff</code> in <code>faircode/metrics.py</code>) - not a theoretical add-on, but a real number sitting next to the other five for every audit, strategy, and model in <code>paper/results-frozen/results_fairness.csv</code>.</p>
<h3 id="concrete-example-compas-audit-01">Concrete Example: COMPAS - Audit 01</h3>
<p>The baseline logistic regression model on race (disadvantaged: African-American defendants; advantaged: Caucasian defendants) is a case where accuracy is nearly equal but the underlying errors are not, using the frozen numbers exactly as <code>faircode/benchmark.py</code> computed them:</p>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th>Metric</th><th>Value</th><th>95% CI</th><th>p-value</th><th>n (disadv. / adv.)</th></tr></thead><tbody><tr><td>Accuracy Equality Diff</td><td>-0.035</td><td>[-0.070, -0.003]</td><td>0.0395</td><td>1,788 / 1,466</td></tr><tr><td>Equalized Odds Diff</td><td>0.926</td><td>[0.909, 0.942]</td><td>0.0</td><td>1,048 / 467</td></tr></tbody></table></div>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th scope="col">Metric</th><th scope="col">Value</th><th scope="col">95% CI</th><th scope="col">p-value</th><th scope="col">n (disadv. / adv.)</th></tr></thead><tbody><tr><td>Accuracy Equality Diff</td><td>-0.035</td><td>[-0.070, -0.003]</td><td>0.0395</td><td>1,788 / 1,466</td></tr><tr><td>Equalized Odds Diff</td><td>0.926</td><td>[0.909, 0.942]</td><td>0.0</td><td>1,048 / 467</td></tr></tbody></table></div>
<p>A 3.5-point accuracy gap (African-American defendants scored slightly <em>less</em> accurately overall) looks like a minor, borderline-significant footnote. It sits right next to a 92.6-point Equalized Odds gap on the exact same model and dataset - <code>faircode/metrics.py</code>&#x27;s own <code>note</code> field for that row reads <code>driven_by_tpr_gap</code>, meaning the true-positive-rate gap between the two groups is enormous. Reporting only Accuracy Equality here would describe this model as nearly fair. It is not: African-American defendants who did not reoffend were flagged as high-risk at a far higher rate than Caucasian defendants who did not reoffend, a disparity Accuracy Equality&#x27;s single blended number cannot see because the two error types are canceling each other out in the total.</p>
<h3 id="detection-code">Detection Code</h3>
<p>Computes Accuracy Equality alongside Equalized Odds so a hidden gap can&#x27;t cancel out unnoticed.</p>
Expand Down
2 changes: 1 addition & 1 deletion explainers/accuracy-not-enough-healthcare-ai.html
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ <h3 id="why-it-matters">Why It Matters</h3>
<p>This is why a fairness or safety audit that stops at accuracy is not an audit at all. The questions that matter - <em>who does the model miss, and how often, in which group</em> - live in per-group recall, false-negative rates, and calibration, not in the headline score.</p>
<h3 id="the-accuracy-paradox">The Accuracy Paradox</h3>
<p>The accuracy paradox is the point where high accuracy and low usefulness meet: the rarer the event, the easier it is to score well by ignoring it.</p>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th>Model on a 3%-readmission population</th><th>Accuracy</th><th>Readmitted patients caught</th></tr></thead><tbody><tr><td>Predicts &quot;no readmission&quot; for everyone</td><td>97%</td><td>0 of every 100 at-risk patients</td></tr><tr><td>Flags the genuinely high-risk 3%</td><td>lower than 97%</td><td>most of them</td></tr></tbody></table></div>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th scope="col">Model on a 3%-readmission population</th><th scope="col">Accuracy</th><th scope="col">Readmitted patients caught</th></tr></thead><tbody><tr><td>Predicts &quot;no readmission&quot; for everyone</td><td>97%</td><td>0 of every 100 at-risk patients</td></tr><tr><td>Flags the genuinely high-risk 3%</td><td>lower than 97%</td><td>most of them</td></tr></tbody></table></div>
<p>The &quot;predict nothing&quot; model wins on accuracy and loses on the only thing that matters. A useful clinical model often <em>accepts</em> a worse accuracy score on purpose - it takes on more false positives to stop missing true cases. Which means accuracy going <strong>down</strong> can be the sign of a <strong>better</strong> model, and a fairness gap can hide entirely inside a flat accuracy number: the same 95% can be 95% for one group and 95% for another while the false negatives are concentrated in whichever group the training data under-represented.</p>
<h3 id="concrete-example-healthcare-readmission-audit-06">Concrete Example: Healthcare Readmission - Audit 06</h3>
<p>Audit 06 in this repo predicts 30-day hospital readmission from the Diabetes 130-US Hospitals dataset (101,766 records), with race and age as protected attributes. Readmission is a <strong>rare, imbalanced</strong> outcome - the baseline models flag only a tiny fraction of patients as high risk - so accuracy on this task is high almost by construction, and it is exactly the wrong number to trust.</p>
Expand Down
2 changes: 1 addition & 1 deletion explainers/ai-hallucinations.html
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ <h3 id="tabular-proof-insurance-denial-model">Tabular Proof: Insurance Denial Mo
<p>The biased model (<code>unfair.py</code>) is trained on a dataset of patient features to predict whether an insurance claim will be high-cost (which flags it for denial). Among the features are <code>bmi</code>, <code>smoker</code> status, and <code>diabetic</code> status.</p>
<p>In this dataset, smoking is a dominant predictive signal: smokers are highly likely to have high claim costs. The model splits on this feature early. However, when we look at underweight patients who smoke and are diabetic, the training evidence is incredibly thin.</p>
<p>By grouping the entire dataset into bins of BMI category, smoking status, and diabetes status, we can compare the <strong>Sample Density</strong> (how many patients fit this profile in the training data) against the <strong>Model&#x27;s Prediction Probability of Denial</strong>:</p>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th>BMI Category</th><th>Smoker</th><th>Diabetic</th><th>Sample Count (Density)</th><th>Mean Predicted Probability of Denial</th><th>Mean Prediction</th></tr></thead><tbody><tr><td><strong>Underweight</strong></td><td><strong>Yes</strong></td><td><strong>No</strong></td><td><strong>1</strong></td><td><strong>98.0%</strong></td><td><strong>Deny (1.0)</strong></td></tr><tr><td><strong>Underweight</strong></td><td><strong>Yes</strong></td><td><strong>Yes</strong></td><td><strong>4</strong></td><td><strong>95.3%</strong></td><td><strong>Deny (1.0)</strong></td></tr><tr><td>Underweight</td><td>No</td><td>Yes</td><td>6</td><td>28.0%</td><td>Approve (0.0)</td></tr><tr><td>Underweight</td><td>No</td><td>No</td><td>9</td><td>33.9%</td><td>Approve (0.0)</td></tr><tr><td>Normal</td><td>Yes</td><td>No</td><td>22</td><td>99.1%</td><td>Deny (1.0)</td></tr><tr><td>Normal</td><td>Yes</td><td>Yes</td><td>28</td><td>99.3%</td><td>Deny (1.0)</td></tr><tr><td>Overweight</td><td>Yes</td><td>Yes</td><td>35</td><td>99.5%</td><td>Deny (1.0)</td></tr><tr><td>Overweight</td><td>Yes</td><td>No</td><td>39</td><td>99.5%</td><td>Deny (1.0)</td></tr><tr><td>Obese</td><td>Yes</td><td>Yes</td><td>69</td><td>99.6%</td><td>Deny (1.0)</td></tr><tr><td>Obese</td><td>Yes</td><td>No</td><td>76</td><td>99.7%</td><td>Deny (1.0)</td></tr><tr><td>Normal</td><td>No</td><td>No</td><td>83</td><td>31.6%</td><td>Approve (0.0)</td></tr><tr><td>Normal</td><td>No</td><td>Yes</td><td>90</td><td>28.9%</td><td>Approve (0.0)</td></tr><tr><td>Overweight</td><td>No</td><td>Yes</td><td>148</td><td>36.4%</td><td>Approve (0.0)</td></tr><tr><td>Overweight</td><td>No</td><td>No</td><td>167</td><td>33.2%</td><td>Approve (0.0)</td></tr><tr><td>Obese</td><td>No</td><td>Yes</td><td>262</td><td>43.6%</td><td>Approve (0.0)</td></tr><tr><td>Obese</td><td>No</td><td>No</td><td>301</td><td>42.5%</td><td>Approve (0.0)</td></tr></tbody></table></div>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th scope="col">BMI Category</th><th scope="col">Smoker</th><th scope="col">Diabetic</th><th scope="col">Sample Count (Density)</th><th scope="col">Mean Predicted Probability of Denial</th><th scope="col">Mean Prediction</th></tr></thead><tbody><tr><td><strong>Underweight</strong></td><td><strong>Yes</strong></td><td><strong>No</strong></td><td><strong>1</strong></td><td><strong>98.0%</strong></td><td><strong>Deny (1.0)</strong></td></tr><tr><td><strong>Underweight</strong></td><td><strong>Yes</strong></td><td><strong>Yes</strong></td><td><strong>4</strong></td><td><strong>95.3%</strong></td><td><strong>Deny (1.0)</strong></td></tr><tr><td>Underweight</td><td>No</td><td>Yes</td><td>6</td><td>28.0%</td><td>Approve (0.0)</td></tr><tr><td>Underweight</td><td>No</td><td>No</td><td>9</td><td>33.9%</td><td>Approve (0.0)</td></tr><tr><td>Normal</td><td>Yes</td><td>No</td><td>22</td><td>99.1%</td><td>Deny (1.0)</td></tr><tr><td>Normal</td><td>Yes</td><td>Yes</td><td>28</td><td>99.3%</td><td>Deny (1.0)</td></tr><tr><td>Overweight</td><td>Yes</td><td>Yes</td><td>35</td><td>99.5%</td><td>Deny (1.0)</td></tr><tr><td>Overweight</td><td>Yes</td><td>No</td><td>39</td><td>99.5%</td><td>Deny (1.0)</td></tr><tr><td>Obese</td><td>Yes</td><td>Yes</td><td>69</td><td>99.6%</td><td>Deny (1.0)</td></tr><tr><td>Obese</td><td>Yes</td><td>No</td><td>76</td><td>99.7%</td><td>Deny (1.0)</td></tr><tr><td>Normal</td><td>No</td><td>No</td><td>83</td><td>31.6%</td><td>Approve (0.0)</td></tr><tr><td>Normal</td><td>No</td><td>Yes</td><td>90</td><td>28.9%</td><td>Approve (0.0)</td></tr><tr><td>Overweight</td><td>No</td><td>Yes</td><td>148</td><td>36.4%</td><td>Approve (0.0)</td></tr><tr><td>Overweight</td><td>No</td><td>No</td><td>167</td><td>33.2%</td><td>Approve (0.0)</td></tr><tr><td>Obese</td><td>No</td><td>Yes</td><td>262</td><td>43.6%</td><td>Approve (0.0)</td></tr><tr><td>Obese</td><td>No</td><td>No</td><td>301</td><td>42.5%</td><td>Approve (0.0)</td></tr></tbody></table></div>
<h4 id="the-hallucination-mechanism">The Hallucination Mechanism</h4>
<p>Look at the first two rows. There is only <strong>one</strong> underweight smoker in the entire dataset, and only <strong>four</strong> underweight smokers who are diabetic. The training evidence for these sub-populations is virtually non-existent.</p>
<p>Yet, the model predicts a <strong>98.0%</strong> and <strong>95.3% probability of denial</strong> for these applicants.</p>
Expand Down
4 changes: 2 additions & 2 deletions explainers/automation-bias.html
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ <h3 id="why-this-matters">Why This Matters</h3>
<p>The bias in the model is a <em>statistical</em> problem. Automation bias is the <em>human</em> problem that lets it leave the lab.</p>
<hr>
<h3 id="common-manifestations">Common Manifestations</h3>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th>Pattern</th><th>Description</th><th>Real-World Consequence</th></tr></thead><tbody><tr><td><strong>Omission error</strong></td><td>Failing to act because the system didn&#x27;t flag it</td><td>High-risk patient discharged because no alert fired</td></tr><tr><td><strong>Commission error</strong></td><td>Acting on a system recommendation that contradicts evidence</td><td>Denying a qualified applicant because the score said &quot;high risk&quot;</td></tr><tr><td><strong>Default acceptance</strong></td><td>Treating the default output as the decision</td><td>Auto-approving or auto-denying based on threshold without review</td></tr><tr><td><strong>Authority transfer</strong></td><td>&quot;The algorithm is objective&quot; becomes a shield against accountability</td><td>No appeal path because &quot;the math decided&quot;</td></tr></tbody></table></div>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th scope="col">Pattern</th><th scope="col">Description</th><th scope="col">Real-World Consequence</th></tr></thead><tbody><tr><td><strong>Omission error</strong></td><td>Failing to act because the system didn&#x27;t flag it</td><td>High-risk patient discharged because no alert fired</td></tr><tr><td><strong>Commission error</strong></td><td>Acting on a system recommendation that contradicts evidence</td><td>Denying a qualified applicant because the score said &quot;high risk&quot;</td></tr><tr><td><strong>Default acceptance</strong></td><td>Treating the default output as the decision</td><td>Auto-approving or auto-denying based on threshold without review</td></tr><tr><td><strong>Authority transfer</strong></td><td>&quot;The algorithm is objective&quot; becomes a shield against accountability</td><td>No appeal path because &quot;the math decided&quot;</td></tr></tbody></table></div>
<hr>
<h3 id="real-world-proof-compas-in-the-courtroom">Real-World Proof: COMPAS in the Courtroom</h3>
<p>The COMPAS risk assessment is the canonical example. ProPublica&#x27;s 2016 analysis showed:</p>
Expand Down Expand Up @@ -313,7 +313,7 @@ <h3 id="detection-code-measuring-automation-bias-in-practice">Detection Code: Me
<p><strong>Interpretation</strong>: In this run, the final human-decision gap (6.8 points) is actually <em>smaller</em> than the model&#x27;s own gap (9.0 points) - not larger. That is not a counterexample to automation bias; it is a property of this specific simulation: the 20% of decisions that don&#x27;t follow the AI score fall back to <code>true_skill</code>, which has no gender bias baked into it at all, so that fallback partially cancels the model&#x27;s bias rather than compounding it. Automation bias amplifies a gap specifically when the human deviates from the model in ways that are <em>not</em> an unbiased correction - a recruiter&#x27;s &quot;gut feeling&quot; override in the real world is not guaranteed to be unbiased the way <code>true_skill</code> is by construction here. Treat this block as a worked example of how to <em>measure</em> <code>model_fairness_gap</code> vs. <code>human_fairness_gap</code> with real data, not as proof that amplification is the only possible direction - the direction depends entirely on whether the human&#x27;s departures from the model are themselves biased or not.</p>
<hr>
<h3 id="mitigation-strategies">Mitigation Strategies</h3>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th>Strategy</th><th>Mechanism</th><th>Limitation</th></tr></thead><tbody><tr><td><strong>Friction by design</strong></td><td>Require explicit override justification; show confidence intervals, not just scores</td><td>Adds workflow time; may be bypassed under pressure</td></tr><tr><td><strong>Counterfactual display</strong></td><td>&quot;If this applicant were male, the score would be X&quot;</td><td>Requires causal model; contested in court</td></tr><tr><td><strong>Blind review first</strong></td><td>Human evaluates case <em>before</em> seeing AI score</td><td>Not always feasible (e.g., triage)</td></tr><tr><td><strong>Disagreement logging</strong></td><td>Track every human-vs-model divergence for audit</td><td>Passive; does not prevent harm in real time</td></tr><tr><td><strong>Calibrated thresholds per group</strong></td><td>Different decision thresholds to equalize outcomes</td><td>Legal risk (disparate treatment); hard to justify</td></tr><tr><td><strong>Human-in-the-loop training</strong></td><td>Simulate edge cases where model is wrong; train reviewers to catch them</td><td>Resource-intensive; decay over time</td></tr></tbody></table></div>
<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th scope="col">Strategy</th><th scope="col">Mechanism</th><th scope="col">Limitation</th></tr></thead><tbody><tr><td><strong>Friction by design</strong></td><td>Require explicit override justification; show confidence intervals, not just scores</td><td>Adds workflow time; may be bypassed under pressure</td></tr><tr><td><strong>Counterfactual display</strong></td><td>&quot;If this applicant were male, the score would be X&quot;</td><td>Requires causal model; contested in court</td></tr><tr><td><strong>Blind review first</strong></td><td>Human evaluates case <em>before</em> seeing AI score</td><td>Not always feasible (e.g., triage)</td></tr><tr><td><strong>Disagreement logging</strong></td><td>Track every human-vs-model divergence for audit</td><td>Passive; does not prevent harm in real time</td></tr><tr><td><strong>Calibrated thresholds per group</strong></td><td>Different decision thresholds to equalize outcomes</td><td>Legal risk (disparate treatment); hard to justify</td></tr><tr><td><strong>Human-in-the-loop training</strong></td><td>Simulate edge cases where model is wrong; train reviewers to catch them</td><td>Resource-intensive; decay over time</td></tr></tbody></table></div>
<hr>
<h3 id="the-deeper-problem-automation-bias-as-a-design-choice">The Deeper Problem: Automation Bias as a Design Choice</h3>
<p>Automation bias is not a bug in human cognition - it is a predictable response to how AI systems are <em>designed and presented</em>.</p>
Expand Down
Loading
Loading