3. Fairness Metrics: Equalized Odds and Calibration

Capsule description

Demographic parity (capsule 02) compares output rates. But that metric alone has gaps: it can be perfect while the model's errors differ by group, or while the model's confidence means different things for each group.

This capsule covers two metrics that fill those gaps:

1. Equalized Odds — equal error rates (false positives + false negatives) across groups.
2. Calibration — the predicted score must correspond to the real probability of the outcome, equally across groups.

Each measures something different. Together with demographic parity, they give you a much more complete picture. But (spoiler for capsule 05) you can't satisfy all three simultaneously except in trivial cases — that's what the impossibility theorems are about.

By the end of the capsule you'll know: when each metric is the right one, how to implement it, and how to communicate the trade-offs.

---

Equalized Odds

The formal definition

Equalized odds requires that the model's errors be equally distributed across groups:

P(Ŷ = 1 | Y = y, A = a) = P(Ŷ = 1 | Y = y, A = b)  ∀ a, b ∈ A, ∀ y ∈ {0, 1}

Where:

• Ŷ = the model's prediction.
• Y = the ground truth (real label).
• A = the protected attribute.

In words: conditional on the ground truth, the probability of a positive prediction must be equal across all groups.

Equivalently, it requires equality in:

• True Positive Rate (TPR) = sensitivity = recall = P(Ŷ=1 | Y=1).
• False Positive Rate (FPR) = P(Ŷ=1 | Y=0).

Both have to be equal across groups.

The difference from demographic parity

Demographic parity does NOT use Y (ground truth). It compares predictions only.

Equalized odds DOES use Y. It compares error rates given the real ground truth.

A simple example:

Loan approvals for 100 applicants per group:

Group A: 50 who would default (Y=0), 50 who wouldn't (Y=1)
Group B: 80 who would default (Y=0), 20 who wouldn't (Y=1)

The model approves:
Group A: 40 (of the 50 Y=1) + 10 (of the 50 Y=0) = 50 approved
Group B: 18 (of the 20 Y=1) + 30 (of the 80 Y=0) = 48 approved

Demographic parity: 50/100 = 50%, 48/100 = 48%. Ratio ~ 0.96. Nearly perfect parity.

TPR (recall):
Group A: 40/50 = 0.80
Group B: 18/20 = 0.90
Difference: 10 points.

FPR:
Group A: 10/50 = 0.20
Group B: 30/80 = 0.375
Difference: 17 points.

Demographic parity comes out fine, but the model is noticeably worse on group B (more false positives). Equalized odds detects this.

Implementation

import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix

def equalized_odds(predictions, ground_truth, groups, favorable_outcome=1):
    """
    Compute equalized odds metrics per group.
    
    Args:
        predictions: the model's predictions.
        ground_truth: the true labels.
        groups: the protected attribute.
        favorable_outcome: the value considered favorable.
    
    Returns:
        dict with TPR, FPR per group and the differences.
    """
    df = pd.DataFrame({
        'pred': predictions,
        'true': ground_truth,
        'group': groups,
    })
    
    metrics_per_group = {}
    
    for group_name, group_df in df.groupby('group'):
        y_true = (group_df['true'] == favorable_outcome).astype(int)
        y_pred = (group_df['pred'] == favorable_outcome).astype(int)
        
        tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
        
        tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
        fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
        
        metrics_per_group[group_name] = {
            'TPR': tpr,
            'FPR': fpr,
            'n': len(group_df),
            'positives': int((y_true == 1).sum()),
            'negatives': int((y_true == 0).sum()),
        }
    
    # Difference between worst/best
    tprs = [m['TPR'] for m in metrics_per_group.values()]
    fprs = [m['FPR'] for m in metrics_per_group.values()]
    
    return {
        'metrics_per_group': metrics_per_group,
        'tpr_difference': max(tprs) - min(tprs),
        'fpr_difference': max(fprs) - min(fprs),
        'meets_threshold_5pct': (
            max(tprs) - min(tprs) < 0.05 and
            max(fprs) - min(fprs) < 0.05
        ),
    }

The typical threshold

For a production deploy:

• TPR difference < 5% across groups.
• FPR difference < 5% across groups.

These are conservative thresholds. Some contexts tolerate more (10-15%), others less (<2% for healthcare). It's an informed decision based on context.

---

Calibration

The formal definition

Calibration requires that the predicted score correspond to the real probability of the outcome, equally across groups:

P(Y = 1 | S = s, A = a) = s = P(Y = 1 | S = s, A = b)  ∀ a, b, ∀ s ∈ [0, 1]

Where:

• S = the score predicted by the model (a probability).
• Y = the ground truth.
• A = the protected attribute.

In words: if the model predicts "score = 0.7," then the real outcome must occur 70% of the time, regardless of the group.

Why it matters

Without calibration, the same score means different things depending on the group:

• "Score = 0.7" for group A → the outcome actually occurs 70% of the time.
• "Score = 0.7" for group B → the outcome actually occurs 50% of the time.

If you use the score as the basis for a decision (e.g., approve if score > 0.6), you're applying a stricter standard to group B.

It's a subtle but real problem. Common in credit scoring, healthcare risk prediction, and any system where a "score" drives a decision.

Implementation

Calibration testing requires bucketing predictions into score bins, and comparing the "actual rate" against the score:

def calibration_by_group(scores, ground_truth, groups, n_bins=10):
    """
    Measure calibration per group, bucketing scores into bins.
    """
    df = pd.DataFrame({
        'score': scores,
        'true': ground_truth,
        'group': groups,
    })
    
    df['bin'] = pd.cut(df['score'], bins=n_bins)
    
    calibration_per_group = {}
    
    for group_name, group_df in df.groupby('group'):
        bin_stats = group_df.groupby('bin').agg(
            mean_score=('score', 'mean'),
            actual_rate=('true', 'mean'),
            n=('score', 'count'),
        )
        
        # Calibration error: difference between mean_score and actual_rate per bin
        bin_stats['error'] = (bin_stats['actual_rate'] - bin_stats['mean_score']).abs()
        
        # Expected Calibration Error (ECE): weighted average
        ece = (bin_stats['error'] * bin_stats['n']).sum() / bin_stats['n'].sum()
        
        calibration_per_group[group_name] = {
            'ECE': ece,
            'bins': bin_stats.reset_index().to_dict('records'),
        }
    
    eces = [m['ECE'] for m in calibration_per_group.values()]
    
    return {
        'calibration_per_group': calibration_per_group,
        'ece_difference': max(eces) - min(eces),
        'meets_threshold_2pct': max(eces) - min(eces) < 0.02,
    }

Visualization: the calibration plot

A calibration plot helps you see it visually:

import matplotlib.pyplot as plt

def plot_calibration(scores, ground_truth, groups, n_bins=10):
    """Calibration plot with one line per group."""
    df = pd.DataFrame({
        'score': scores,
        'true': ground_truth,
        'group': groups,
    })
    df['bin'] = pd.cut(df['score'], bins=n_bins)
    
    fig, ax = plt.subplots(figsize=(8, 6))
    
    # Diagonal line = perfect calibration
    ax.plot([0, 1], [0, 1], 'k--', label='Perfect calibration')
    
    for group_name, group_df in df.groupby('group'):
        stats = group_df.groupby('bin').agg(
            mean_score=('score', 'mean'),
            actual_rate=('true', 'mean'),
        )
        ax.plot(stats['mean_score'], stats['actual_rate'],
                marker='o', label=f'Group {group_name}')
    
    ax.set_xlabel('Predicted score')
    ax.set_ylabel('Actual rate')
    ax.set_title('Calibration by group')
    ax.legend()
    ax.grid(True)
    
    return fig

If each group's line diverges significantly from the diagonal, there's miscalibration. If one line is consistently above or below another, the score means different things per group.

---

When to use equalized odds vs. calibration vs. demographic parity

When equalized odds

Appropriate when:

1. There's reliable ground truth: you can verify who "deserved" the outcome.
2. Errors carry significant impact and should be distributed equitably.
3. Base rates differ across groups and you want error parity, not rate parity.

Examples: loans (default is verifiable ground truth), medical diagnosis, criminal recidivism.

When calibration

Appropriate when:

1. The score is used directly in decisions (not just binary classification).
2. Multiple downstream decisions depend on the score.
3. Stakeholders interpret the score as a probability.

Examples: credit scoring (interest rate scaled by score), insurance pricing (premium based on risk score), healthcare risk stratification.

When demographic parity

Appropriate when:

1. There's no reliable ground truth for merit.
2. Policy/law requires parity explicitly.
3. Marketing/exposure decisions.

Examples: ad serving, content recommendation, affirmative action programs.

Summary table

Metric	Compares	Requires ground truth	When to apply
Demographic parity	Output rates	No	No ground truth, policy parity
Equalized odds	Error rates (TPR, FPR)	Yes	Errors must be equitable
Calibration	Score-to-actual mapping	Yes	The score is used directly in decisions

---

The conflict between metrics

You'll learn about the impossibility theorems next (capsule 05), but we can already see the conflict:

Example: COMPAS (criminal recidivism prediction)

ProPublica (2016) analyzed COMPAS, used in US courts to predict recidivism:

• A 1-10 score. A prediction of re-arrest.
• ProPublica measured equalized odds: they found Black defendants had a much higher FPR than white defendants. Conclusion: discrimination.
• Northpointe (the vendor) responded by measuring calibration: the model was well calibrated for both groups. Score = 0.7 → 70% recidivism for Black defendants, 70% for white. Conclusion: no discrimination.

Who was right? Both. The model satisfied calibration but violated equalized odds. Because of the impossibility theorems, you can't have both.

The right conclusion isn't "it was fine" or "it was broken" — it's "the model has this trade-off; the legal system has to consciously decide which one to prioritize." ProPublica argued that in criminal justice, equalized odds matters more (false positives cause unjust incarceration). Northpointe argued that calibration matters more (a consistent score).

Decision-making requires choosing explicitly. There's no universal "fairness" metric.

---

Traps and common mistakes

1. Reporting only average accuracy

Demographic parity, equalized odds, calibration — all of them require a breakdown by subgroup. Global accuracy hides the gaps.

2. Confusing TPR difference with FPR difference

Equalized odds requires BOTH to be similar across groups. If you only equalize TPR but FPR differs, that's equal opportunity (a weaker variant), not full equalized odds.

3. Ignoring sample size in the calibration bins

If a bin has 5 samples, its "actual_rate" is very noisy. Only report ECE if the bins have at least 30 samples.

4. Pretending the metrics complement each other with no trade-off

When a PM says "let's satisfy every metric," the right answer is "mathematically impossible (see Chouldechova). We have to prioritize."

5. Applying the same threshold to different contexts

A 5% TPR difference may be acceptable in advertising and unacceptable in healthcare. The threshold must be context-specific.

---

Self-check

1. What's the conceptual difference between demographic parity and equalized odds?

Demographic parity measures output rates with no ground truth: P(Ŷ=1 | A=a) = P(Ŷ=1 | A=b).

Equalized odds measures error rates given the ground truth: P(Ŷ=1 | Y=y, A=a) = P(Ŷ=1 | Y=y, A=b) ∀ y.

Implication:

• Demographic parity can ignore who "deserved" the outcome.
• Equalized odds incorporates "merit" (ground truth) into the metric.

For systems with reliable ground truth and different base rates across groups, equalized odds is typically more appropriate.

2. Why can calibration be violated even when equalized odds holds?

Equalized odds compares TPR and FPR at a specific threshold (typically 0.5).

Calibration looks at the full distribution of scores: whether "score = 0.7" means the same thing for every group across all scores.

You can have:

• TPR and FPR equalized at a threshold of 0.5 (equalized odds ✅).
• But scores in the 0.3-0.4 range distributed differently per group (calibration ❌).

The two metrics measure different things. And because of the impossibility theorems (capsule 05), you can't have both in general.

3. When would you choose equalized odds over demographic parity?

When:

1. You have reliable ground truth that can be verified (defaults, medical outcomes, recidivism).

2. Base rates differ across groups and forcing demographic parity would require approving marginal cases (causing harm to both the individual and the system).

3. Equity of errors matters more than equity of rates: you want the model to be equally good (or bad) for every group, not to approve the same percentage.

The typical case: loan approvals where groups have verifiably different default rates. Demographic parity would force you to give loans to high-risk cases, which harms the borrower. Equalized odds ensures that given the real risk, the model treats everyone the same.

4. Why does COMPAS expose the conflict between metrics?

COMPAS satisfied calibration (a consistent score across groups) but violated equalized odds (higher false positives for Black defendants).

Because of the impossibility theorems, you cannot have both when the groups have different base rates.

ProPublica vs. Northpointe represent two legitimate positions:

• ProPublica: "Equalized odds matters more in criminal justice — false positives = unjust incarceration."
• Northpointe: "Calibration matters more — a consistent score is basic transparency."

Both are mathematically valid. The decision is a policy choice, not math.

The lesson: when a model drives high-stakes decisions, the stakeholders (not the engineers alone) have to decide which metric to prioritize, based on values and impact analysis. Engineering provides the options; the decision is socio-technical.

---

Summary and next step

• Equalized odds measures error rates (TPR, FPR) per group. Appropriate when there's ground truth.
• Calibration measures whether scores correspond to real probabilities per group. Critical when scores are used directly.
• Each metric captures something different: demographic parity (rates), equalized odds (errors), calibration (scores).
• You can't satisfy all three simultaneously (the impossibility theorems, capsule 05).
• The COMPAS example: calibration passed, equalized odds failed — that's a policy decision, not math.

Checkpoint: you should be able to compute all three metrics in code and choose which one is appropriate for a given context.

Bridge to the next capsule: capsule 04 covers bias detection in practice — how to apply these metrics to your real system. You'll see techniques like subgroup slicing, demographic A/B testing, counterfactual testing, and how to wire it all into CI/monitoring.

---

Resources

1. Equality of Opportunity (Hardt et al., 2016) — the canonical equalized odds paper.
2. Inherent Trade-offs (Kleinberg et al., 2017) — the impossibility theorems.
3. ProPublica COMPAS analysis — the case study.
4. Reliability Diagrams (DeGroot & Fienberg, 1983) — the original calibration paper.

---

Next: 04-bias-detection-slicing.md — Bias detection in practice: slicing, A/B, counterfactual.

Capsule 03 of 08 — Module 2 — AI Ethics & Compliance Guide