Module 2: Bias and Fairness — Detection, Measurement, Mitigation

5. Impossibility Theorems and Trade-offs

Capsule description

This is the mathematically densest capsule in the module. And the most important one for not falling into false confidence.

Formally proven facts (Chouldechova 2017, Kleinberg-Mullainathan-Raghavan 2017):

You cannot simultaneously satisfy demographic parity, equalized odds, and calibration except in trivial cases (when the groups have identical distributions).

This is not a philosophical opinion. It's a mathematical theorem. And it has direct implications for your work:

When a PM says "let's satisfy all the fairness metrics," the right answer is "mathematically impossible. We have to choose."
That choice isn't purely technical — it's socio-technical. It requires a decision about what kind of fairness we prioritize, based on values and context.
Documenting the choice and the trade-off is part of your ethical responsibility. Don't hide it.

This capsule gives you the intellectual tool to navigate this trade-off honestly.

The two main theorems

Theorem 1: Chouldechova (2017)

Setup: a classifier predicting a binary outcome with a continuous score.

Result: in general, you cannot simultaneously have:

Calibration: P(Y=1 | S=s, A=a) = s ∀ s, a.
Equal Predictive Value: P(Y=1 | Ŷ=1, A=a) = P(Y=1 | Ŷ=1, A=b).

Trivial exceptions: when the groups have identical base rates (P(Y=1 | A=a) = P(Y=1 | A=b)).

In practical language: if the groups genuinely have different outcome rates, choosing between calibration and equal predictive value is unavoidable.

Theorem 2: Kleinberg-Mullainathan-Raghavan (2017)

Setup: a classifier with a score, threshold-based decisions.

Result: in general, you cannot simultaneously have:

Calibration across groups.
Balance for the negative class (equal FPR across groups).
Balance for the positive class (equal TPR across groups).

Trivial exceptions: when the base rates are identical, or when the model is perfect (zero error).

Implication: if you want equalized odds (equal TPR + equal FPR), you have to sacrifice calibration. If you want calibration, you have to sacrifice equalized odds.

Why it matters

For real systems:

Loans: defaulters and non-defaulters have different base rates across demographic groups in real data.
Hiring: the qualified rate can differ.
Medical diagnosis: the prevalence of conditions differs.

The trivial exception almost never applies in real systems. The trade-off is inescapable.

The COMPAS example again

To make the conflict concrete:

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) predicts recidivism for defendants in US criminal courts.

ProPublica (2016) measured equalized odds:

Group	TPR (recidivism predicted correctly)	FPR (non-recidivists labeled as risky)
Black defendants	65%	45%
White defendants	60%	23%

A false positive rate 2x higher for Black defendants. That means: Black defendants labeled "high risk" were in fact non-recidivists at a significantly higher rate than white defendants.

ProPublica's conclusion: discriminatory.

Northpointe (the vendor) responded by measuring calibration:

Score	Recidivism rate (Black)	Recidivism rate (White)
Score 1-3	22%	21%
Score 4-6	41%	39%
Score 7-10	68%	65%

Score = X means a similar recidivism rate for both groups. Calibration ✅.

Northpointe's conclusion: well calibrated, not discriminatory.

Who was right?

Both. By Chouldechova/Kleinberg, calibration and equalized odds cannot coexist when groups have different base rates.

Black defendants have a higher recidivism rate in the dataset (that isn't controversial — it's a fact about the dataset). Which is why satisfying calibration forces a higher FPR for them.

The right conclusion isn't "ProPublica vs. Northpointe":

ProPublica argued that in criminal justice, equalized odds should be prioritized: false positives → incarceration or a harsher sentence for people who aren't recidivists.
Northpointe argued that calibration is basic transparency: a consistent score across groups.

Both positions are mathematically legitimate. The decision is a policy choice, not a technical one.

What was missing

What Northpointe should have documented explicitly:

"This model prioritizes calibration. As a mathematical consequence (the Chouldechova theorem), it has a higher FPR for groups with higher base rates. In the available dataset, that primarily affects Black defendants. Stakeholders must assess whether this trade-off is acceptable for use in criminal sentencing."

Intellectual honesty requires acknowledging the trade-off, not defending the model as "completely fair."

How to choose which metric to prioritize

The decision framework

Question	If YES → prioritize
Is the score used directly (not just as a binary)?	Calibration
Do errors have a severe impact on people?	Equalized odds
Is there no reliable ground truth?	Demographic parity
Does the legal framework dictate it?	Whatever the regulation requires
Do stakeholders need a consistent, interpretable score?	Calibration
Is equity of outcomes the explicit goal?	Demographic parity or equalized odds

By industry

Hiring:

Equalized odds is the priority: false positives (rejecting a qualified candidate) and false negatives (accepting a non-qualified one) should be equitable.
Demographic parity too, if it's a policy requirement.
Calibration is less critical (the score is rarely communicated to applicants).

Lending:

Equalized odds: equity in errors has a direct financial impact.
Calibration matters for risk-based pricing.
Demographic parity can be problematic (forcing approval of high-risk cases = harm to the borrower).

Healthcare:

Calibration is critical (scores are used for risk stratification, treatment decisions).
Equalized odds for diagnostic systems (false negatives are critical).
Demographic parity is rarely appropriate (base rates vary biologically).

Criminal Justice:

Equalized odds (false positives → lost liberty).
Calibration is secondary.

Advertising / Recommendation:

Demographic parity is often appropriate (no clear ground truth).
Calibration matters for CTR predictions.
Equalized odds is less relevant.

Documenting the choice

Your Bias Audit must include:

## Fairness Metric Decision

We have chosen to prioritize **[METRIC NAME]** for this system.

**Rationale**:
- [Reason 1, e.g., "Errors have direct impact on persons (loans denied)"]
- [Reason 2, e.g., "Ground truth (default) is verifiable"]
- [Reason 3, e.g., "Equity of errors aligns with our stakeholder values"]

**Trade-offs accepted**:
- We will likely **not** satisfy [OTHER METRIC] perfectly.
- Concretely: [describe the gap, e.g., "demographic parity ratio expected ~0.85"]

**Documented by**: [author], [date]
**Reviewed by**: [stakeholders]
**Re-evaluation**: [date 6-12 months out]

That documentation is what protects you under regulator/auditor scrutiny. "We chose priority X for reason Y" is defensible. "We didn't look at the metrics" is not.

When there's no technical solution

Sometimes, the result of the impossibility theorems is that no viable fair system exists. Examples:

Case: criminal recidivism with systemic inequality

If society has systemic inequalities (economic opportunity, education, differential policing), the historical recidivism dataset is going to reflect those inequalities. Any model based on that dataset will reproduce the structural bias.

There is no technical fix that isn't cosmetic. The solution is at the policy/system level: investing in opportunity, reforming policing, etc.

Applying AI to this case is simply automating historical bias with a veneer of objectivity.

A valid ethical decision: don't build the system. Or build it strictly as advisory, never authoritative.

Case: hiring with highly skewed historical data

If your historical dataset is 90% men and 10% women in role X, any model is going to inherit that skew. Applying fairness constraints can mitigate but rarely eliminate it.

A valid alternative: do NOT use historical data for training. Instead, define explicit criteria (skills, experience requirements) and use AI only for structured matching, not for learning from outcomes.

Conclusion

Intellectual honesty: sometimes, the technical tools are insufficient. Acknowledging that is part of the responsibility. The choice can be:

Apply the best techniques and document the trade-offs (common).
Redesign the system to avoid the problem (better).
Don't build the system (valid in some cases).

Trade-offs between fairness and accuracy

Another important trade-off: fairness constraints often reduce accuracy.

Why

The "most accurate" model on historical data may be accurate precisely because it captures the bias. If the bias is predictive (in the dataset), removing it reduces predictive power.

How much?

It depends on the case. Usually:

Light mitigations (re-balancing): 0-2% accuracy loss.
Medium mitigations (post-processing thresholds): 2-5%.
Aggressive mitigations (adversarial debiasing): 5-15%.

How to decide how much to sacrifice

The framework:

Establish a minimum fairness threshold that is non-negotiable.
Establish a minimum accuracy threshold required for the system to be useful.
Find the Pareto frontier between the two.
Choose a point on the frontier that satisfies both thresholds.

If no point satisfies both, the system must not be deployed as designed. Go back to Phase 5 of the Impact Analysis.

Documentation

## Fairness vs Accuracy Trade-off

**Without fairness constraints**:
- Accuracy: 92%
- Demographic parity ratio: 0.55 (fails 4/5 rule)

**With fairness mitigations**:
- Accuracy: 88% (4 points lower)
- Demographic parity ratio: 0.85 (passes 4/5 rule)

**Decision**: deploy with mitigations. Accuracy of 88% is above the minimum useful threshold (85%). Fairness is above the legal threshold.

**Monitoring**: re-evaluate every 6 months. If accuracy drops below 85% with drift, retrain.

Mitigation techniques previewed

To resolve the trade-offs, there are three categories of mitigation (we cover the details in capsules 06-07):

Pre-processing: modify the training data before training.

Re-balancing groups.
Data augmentation for under-represented groups.
Re-weighting samples.

In-processing: modify the training algorithm.

Fairness constraints in the loss function.
Adversarial debiasing.
Multi-task learning with fairness as a task.

Post-processing: modify the trained model's predictions.

Threshold tuning per group.
Calibration adjustment.
A reject option in marginal zones.

Each category has its pros/cons. Capsules 06-07 cover them.

Traps and common mistakes

1. "Let's satisfy every metric"

Mathematically impossible. If someone asks for it, educate them: "By the Chouldechova theorem, we can't. We have to choose which one to prioritize."

2. "If we choose metric X, we're fair"

No. You're fair with respect to metric X. You're going to fail on Y and Z. Document it honestly.

3. "No trade-off here"

When someone says "this solution has no trade-off," be suspicious. If you don't see the trade-off, you're probably not measuring everything.

4. Denying the accuracy/fairness trade-off

Sometimes it exists, sometimes it doesn't. Pretending it never exists is dishonest. Pretending it always exists (and therefore not applying fairness) is too.

5. Applying mitigations without measuring the effect

You applied re-balancing, so now "it's fair." Did you verify? Did you measure all three metrics post-mitigation? If not, you don't know.

Self-check

1. When do the impossibility theorems NOT apply?

Trivial cases where a unique solution exists:

Identical base rates across groups: P(Y=1 | A=a) = P(Y=1 | A=b) ∀ a, b.
A perfect model (zero error): if you're going to have 100% accuracy, there are no errors to distribute unequally.

In real systems, practically never does either hold:

Base rates differ across demographic groups for historical, social, and biological reasons.
Models aren't perfect.

Therefore, in practice, the trade-offs are inescapable.

Implication: when someone says "let's satisfy every metric," the answer is "mathematically impossible in this context. We choose which one to prioritize."

2. Who was right in COMPAS, ProPublica or Northpointe?

Both were right about what they measured. ProPublica measured equalized odds (a 22-point FPR difference for Black defendants). Northpointe measured calibration (similar for both groups). Both results are mathematically correct.

The real conflict is policy: what kind of fairness do we prioritize in criminal justice?

ProPublica's argument: equalized odds (false positives → lost liberty).
Northpointe's argument: calibration (transparency in scores).

Both positions are legitimate. The decision is socio-technical, not mathematical:

What do we value as a society?
What does the law require?
What do stakeholders consider acceptable?

The general lesson: if a model drives high-stakes decisions, the stakeholders (not just the engineers) must consciously choose which metric to prioritize, with a full understanding of the trade-off.

3. How would you choose a metric for a hiring system?

The framework:

Is there ground truth? Yes (hires who performed well/badly post-hire).
Do errors have a severe impact? Yes (false negatives = a qualified candidate rejected; false positives = hiring a poor fit).
Is the score used directly? No (a binary "interview" or "no").
Legal framework? The US EEOC framework (4/5 rule = demographic-parity-adjacent).

Decision: prioritize equalized odds (equitable errors) and monitor demographic parity (legal compliance).

Trade-off accepted: calibration probably won't be satisfied (irrelevant, because the score isn't used directly).

Documentation:

"We prioritize equalized odds: TPR difference < 5%, FPR difference < 5%. Demographic parity ratio monitored to ensure 4/5 rule compliance. Calibration not explicitly enforced because hiring decisions are binary, not score-driven."

Documenting it is defensible. Not documenting it creates exposure.

4. When is it ethically appropriate NOT to build the system?

When:

There's no viable technical solution: the trade-off between fairness metrics allows no acceptable point + the accuracy threshold is unreachable.
Structural bias in the data makes any model unavoidably biased: e.g., recidivism with datasets that reflect historically differential policing.
The system replicates systemic inequality: applying AI doesn't solve the underlying social problem, it just automates it with a veneer of objectivity.
The cost of prevention exceeds the system's benefit: if it needs 5 years of bias mitigation work for a responsible deploy, the business case may not support it.
The use case is inherently problematic: facial recognition for policing, in a context where regulation is trending toward a ban, may not be viable.

In these cases, "don't build it" or "build it as advisory only, never authoritative" is a valid and sometimes required decision.

This requires the engineer to speak up. PMs/stakeholders aren't going to suggest it — they want to ship. The engineer has the technical information to argue (with analysis and numbers) that some systems shouldn't exist as designed.

Summary and next step

The impossibility theorems (Chouldechova, KMR) prove you can't satisfy all fairness metrics simultaneously when groups have different base rates (which is always, in practice).
The COMPAS example: ProPublica and Northpointe are both correct about what they measured — the conflict is policy, not math.
Choosing a metric is a socio-technical decision. By industry:
- Hiring → equalized odds.
- Lending → equalized odds + calibration.
- Healthcare → calibration is critical.
- Criminal justice → equalized odds (false positives = liberty).
- Advertising → demographic parity.
Documenting the choice + trade-offs + monitoring is your defense against regulator scrutiny.
Sometimes the right answer is "don't build the system as designed."

Checkpoint: you should be able to articulate why no system can satisfy "all the fairness metrics" simultaneously, and choose a primary metric based on context.

Bridge to the next capsule: capsules 02-05 covered measurement and detection. Capsules 06-07 cover mitigation: what to do when you detect bias. Capsule 06 covers pre-processing mitigations (modifying the data before training): re-balancing, re-sampling, augmentation, re-weighting. They're the simplest techniques and often effective.

Resources

Chouldechova (2017) — the original paper.
Kleinberg-Mullainathan-Raghavan (2017) — the original paper.
The Mythos of Model Interpretability (Lipton, 2016) — related context.
Fair ML Book — Chapter 4 — a pedagogical explanation.
Northpointe response to ProPublica — for their perspective.

Next: 06-mitigation-in-pre-processing.md — Pre-processing mitigation: modifying the data before training.

Capsule 05 of 08 — Module 2 — AI Ethics & Compliance Guide

← Previous4. Bias Detection: Slicing, A/B Testing and Counterfactual Testing Next →6. Mitigation: Pre-processing Techniques