Bias and Fairness in AI

AI systems now help decide who gets a job interview, a loan, extra medical attention or a second look from security. If a system works noticeably worse for some groups of people, those people pay the cost, often without knowing a machine was involved.

What "bias" means here

In statistics, bias simply means a systematic error: results that are consistently off in one direction. Algorithmic bias is when an automated system produces systematically worse or less favorable outcomes for certain groups, such as people of a particular sex, age, skin tone, language or region. Fairness is the goal of avoiding unjustified differences in how a system treats or performs for those groups. As you will see, people can reasonably disagree about which definition of fairness fits a given situation.

A useful analogy: a model is like a student who learned everything from one library. If the library's books are old, one-sided or missing whole subjects, the student will repeat those gaps with complete confidence, and sometimes exaggerate them.

Where bias comes from

• Historical data. Models learn patterns from past decisions. If those decisions were skewed, the model learns the skew.
• Unrepresentative samples. If some groups are rare in the training data, the model gets less practice on them and makes more mistakes.
• The wrong target (proxy). Builders often predict something easy to measure that stands in for what they really care about. If the stand-in tracks some groups differently, errors follow.
• Measurement and labels. The people or tools that label training examples can be inconsistent across groups.
• Deployment and feedback loops. A system used in a different setting from the one it was tested in, or whose decisions shape its future training data, can drift.
• For chatbots specifically. Large language models learn from huge amounts of internet text, which overrepresents some languages, regions and viewpoints, and then from human raters' preferences during fine-tuning. Both can shape which assumptions and associations show up in answers.

Documented examples

Two lessons stand out. First, bias rarely comes from anyone intending it; it comes from data and design choices. Second, it can be fixed or reduced once it is measured: the NIST study found the most equitable systems were among the most accurate, and the hospital study showed a better target removes most of the gap.

How bias is measured

You cannot manage what you do not measure. The basic method: split test results by group and compare.

Worked example. A resume screener reviews 1,000 applicants, 500 from group A and 500 from group B.

• Group A: 150 advance, a selection rate of 30%.
• Group B: 90 advance, a selection rate of 18%.
• Ratio: 18 ÷ 30 = 0.60.

US employment regulators have long used a "four-fifths" rule of thumb: if one group's selection rate is below 80% of the highest group's, that is a signal to investigate. Here 0.60 is well below 0.80, so the tool needs a closer look. It is a screening signal, not proof of discrimination.

Common fairness measures include:

• Selection-rate parity: groups are chosen at similar rates.
• Equal error rates: groups face similar rates of false alarms and misses.
• Calibration: a given score means the same real-world likelihood in every group.

When groups have different underlying rates of the outcome, researchers have shown that a tool generally cannot satisfy equal error rates and calibration at the same time. That is exactly what happened in the COMPAS debate. Choosing a fairness measure is therefore partly a values decision about which errors matter most, not only a technical one.

How bias is reduced

• Better data: collect more examples from underrepresented groups, check labels and remove proxies that stand in for protected traits.
• Better targets: make sure the model predicts what you actually care about.
• Training adjustments: reweight examples or add constraints that penalize unequal error rates.
• Threshold and output checks: test results by group before and after launch, and adjust decision thresholds where appropriate and lawful.
• Process safeguards: impact assessments before deployment, human review for consequential decisions, a way for people to contest outcomes, and ongoing monitoring.
• Documentation: record what data was used, what was tested and where the system should not be used.

None of these is a one-time fix. Populations and uses change, so testing has to continue.

Going deeper

The confusion matrix, per group

Most fairness measures come from one table. For a yes/no decision (advance, approve, flag), compare each person's prediction with what actually turned out to be true:

From these four counts, computed separately for each group:

• Selection rate $= \dfrac{TP + FP}{N}$, the share of the group that gets a "yes";
• True positive rate (TPR, also called recall) $= \dfrac{TP}{TP + FN}$, the share of genuinely qualified people who are caught;
• False positive rate (FPR) $= \dfrac{FP}{FP + TN}$, the share of genuinely unqualified people wrongly flagged;
• Positive predictive value (PPV, also called precision) $= \dfrac{TP}{TP + FP}$, the share of "yes" decisions that were right;
• False negative rate $\text{FNR} = 1 - \text{TPR}$.

Selection-rate parity compares the first line across groups; "equal error rates" compares TPR and FPR; equal PPV (sometimes called predictive parity) is a close cousin of calibration: it means a "yes" is equally trustworthy in every group.

Why the criteria collide

Write $p$ for a group's base rate (the share who are actually positive). Chouldechova (2017) pointed out that the four quantities are tied together by an identity that follows directly from the definitions:

$\text{FPR} = \frac{p}{1-p} \cdot \frac{1 - \text{PPV}}{\text{PPV}} \cdot (1 - \text{FNR})$

If two groups have the same PPV and the same FNR but different base rates $p$, their false positive rates must differ.

Worked example. A risk tool is applied to 1,000 people in group A and 1,000 in group B. In A, 50% are actually positive ($p = 0.5$); in B, 20% are ($p = 0.2$). The tool achieves the same precision (PPV = 0.8) and the same true positive rate (0.8, so FNR = 0.2) in both groups.

• Group A: $\text{FPR} = \frac{0.5}{0.5} \times \frac{0.2}{0.8} \times 0.8 = 1 \times 0.25 \times 0.8 = 0.20$.
• Group B: $\text{FPR} = \frac{0.2}{0.8} \times 0.25 \times 0.8 = 0.25 \times 0.25 \times 0.8 = 0.05$.

Check with counts. A has 500 positives: TP = 400, FN = 100; PPV = 0.8 means FP = 100; with 500 negatives, FPR = 100/500 = 0.20. B has 200 positives: TP = 160, FN = 40; FP = 40; with 800 negatives, FPR = 40/800 = 0.05. Selection rates are $(400+100)/1000 = 0.50$ for A and $(160+40)/1000 = 0.20$ for B, a ratio of 0.40, far below four-fifths.

So the same tool is "fair" by precision and true positive rate, "unfair" by false positive rate (an innocent member of A is four times as likely to be flagged), and "unfair" by selection rate. This is the COMPAS debate in miniature. Kleinberg, Mullainathan and Raghavan (2016) proved the general version: apart from special cases (perfect prediction, or equal base rates), a risk score cannot be calibrated within groups and also balance error rates across them. Someone has to decide which errors matter most in the setting, and write that decision down.

Measuring it in code

import numpy as np

def make_group(name, tp, fn, fp, tn):
    y_true = np.array([1] * (tp + fn) + [0] * (fp + tn))
    y_pred = np.array([1] * tp + [0] * fn + [1] * fp + [0] * tn)
    return y_true, y_pred, np.array([name] * len(y_true))

def group_report(y_true, y_pred, group):
    rates = {}
    for g in np.unique(group):
        m = group == g
        t, p = y_true[m], y_pred[m]
        tp = np.sum((t == 1) & (p == 1)); fp = np.sum((t == 0) & (p == 1))
        fn = np.sum((t == 1) & (p == 0)); tn = np.sum((t == 0) & (p == 0))
        rates[g] = (tp + fp) / m.sum()
        print(f"{g}: n={m.sum()} selected={rates[g]:.2f} TPR={tp/(tp+fn):.2f} "
              f"FPR={fp/(fp+tn):.2f} PPV={tp/(tp+fp):.2f}")
    return rates

a = make_group("A", tp=400, fn=100, fp=100, tn=400)
b = make_group("B", tp=160, fn=40, fp=40, tn=760)
y_true, y_pred, group = (np.concatenate(x) for x in zip(a, b))
rates = group_report(y_true, y_pred, group)
print("selection-rate ratio:", round(min(rates.values()) / max(rates.values()), 2))

This reproduces the worked example: both groups show TPR 0.80 and PPV 0.80, while FPR is 0.20 versus 0.05 and the selection-rate ratio is 0.4. In real projects, y_true and y_pred come from a held-out test set, and libraries such as Fairlearn compute the same per-group breakdowns for any metric.

Small groups are noisy

A rate measured on few people is uncertain. Its standard error is $\sqrt{r(1-r)/n}$ for a rate $r$ measured on $n$ people. For a group of 40 with a 25% selection rate, that is $\sqrt{0.25 \times 0.75 / 40} \approx 0.068$, so the true rate could plausibly be anywhere from about 12% to 38%. Report group sizes alongside rates, pool data over time where you can, and never conclude "no bias found" from a handful of cases.

Where mitigation happens

Mitigations are usually grouped by stage: pre-processing (fix the data: rebalance, relabel, remove proxies), in-processing (train with a fairness constraint or penalty), and post-processing (adjust decision thresholds). Each has costs, and some are legally restricted. In US hiring, for example, federal law (42 U.S.C. § 2000e-2(l), added by the Civil Rights Act of 1991) makes it unlawful to adjust scores, use different cutoff scores or otherwise alter the results of employment-related tests on the basis of race, colour, religion, sex or national origin, so group-specific thresholds can themselves be unlawful. Involve legal advice before using them.

Common pitfalls

• Reporting only overall accuracy. A system can be 95% accurate overall and much worse for a small group.
• "Fairness through unawareness." Deleting the protected attribute does not help if postcode, school or name still stands in for it, and it stops you measuring the problem.
• Picking the metric after seeing the results. Decide in advance which measures matter and why.
• Ignoring intersections. Gender Shades found the worst errors for darker-skinned women, a gap that averages by sex or by skin tone alone would hide.
• Trusting small samples. Wide uncertainty can hide a real gap or invent one.
• Treating the four-fifths rule as a safe harbour. Passing it does not prove a tool is fair or lawful; failing it does not prove discrimination.
• Testing once. Populations, inputs and uses drift; schedule re-testing.

In practice

If you use AI tools:

• For decisions about people (hiring, grading, lending, housing), treat AI output as one input, not the verdict.
• Try the same request with details changed, such as a different name or region, and see whether the answer shifts.
• Ask vendors what fairness testing they did, on which groups, and how you can appeal a result.

If you build with AI:

• Decide in advance which groups and which fairness measures matter for your use case, and write that down.
• Report accuracy and error rates per group, not just one overall number.
• Keep a human in the loop and an appeal route for consequential decisions, and keep monitoring after launch.