Module 2: Bias and Fairness — Detection, Measurement, Mitigation

1. Module introduction: Bias and Fairness

Capsule description

In Module 1 you established that ethical failures in AI carry a quantifiable cost: legal, reputational, technical, human. You identified bias risks in your Ethics Impact Analysis. You know that Amazon Hiring discriminated by gender, that facial recognition fails 30+ points worse for Black women, that Apple Card produced disparity through indirect proxies.

What you still don't know is: how do you measure whether your system has those problems?

"My system might be biased" is a useless statement. "My system has a demographic parity of 0.72, which indicates significant disparity requiring mitigation" is an actionable one.

This module is the transition from ethical discourse to ethical engineering. Bias stops being an abstract topic and becomes something you:

Measure with industry-standard metrics (demographic parity, equalized odds, calibration, disparate impact).
Detect with specific tests wired into CI (slicing, demographic A/B, adversarial testing).
Mitigate with documented techniques in three categories (pre-processing, in-processing, post-processing).

It's the most technical module in Phase 1. It'll have formulas, code, and uncomfortable trade-offs. It's exactly what your ethical workflow needs to stop being theory.

Why bias first?

Bias is the most frequent, most measurable, and most actionable ethical risk in AI systems. That's why it's the guide's first technical module.

Most frequent

Any system trained on real-world data probably has bias. The default isn't "no bias" — the default is "biased until proven otherwise." Why:

Real-world data reflects historical biases.
Models learn patterns, including the biased ones.
Without explicit testing, the bias is invisible in global metrics.

Most measurable

Unlike "transparency" or "accountability" (concepts that require judgment), bias is quantified with explicit mathematical formulas. You can compute a number, compare it against a threshold, and decide.

That concreteness is valuable: it turns ethics from "a topic of discussion" into "a QA gate decision."

Most actionable

There are dozens of documented techniques for mitigating detected bias:

Pre-processing: rebalance the training data.
In-processing: add fairness constraints to the loss function.
Post-processing: adjust thresholds per group.

You aren't inventing anything. You're applying a mature body of knowledge, with software libraries (AI Fairness 360, Fairlearn, Fairness Indicators) ready to use.

What you'll learn

8 capsules, in order:

Introduction (this capsule) — what bias is, why it's measurable, the module map.
Demographic parity — the most common metric. Definition, formula, code, when to apply it.
Equalized odds and calibration — more sophisticated metrics. When they matter more.
Bias detection: slicing and demographic A/B testing — how to detect it in practice.
Impossibility theorems — why you can't satisfy every metric. Real trade-offs.
Mitigation: pre-processing — rebalancing, re-sampling, data augmentation.
Mitigation: in-processing and post-processing — adversarial debiasing, threshold tuning.
Mini-project: Bias Audit Toolkit — a reusable implementation.

By the end:

You compute 4 fairness metrics from scratch, in code.
You design bias tests specific to your system.
You apply the right mitigations for the problem you detected.
You articulate trade-offs to make informed decisions (not follow recipes).
You produce a reusable toolkit you apply to future systems.

The mindset shift in this module

Before:

"Bias is important. I'm going to try to be careful with the data."

After:

"Bias is measurable with formula X. My system currently has metric X = 0.72. The legal threshold is 0.80. I'm going to apply mitigation technique Y to get it to 0.85, and add test X.test_fairness() to CI so it doesn't regress."

Concreteness. Metrics. Code. Informed decisions instead of good intentions.

The honest trade-off: fairness vs. accuracy

Before we get into the module, an uncomfortable truth:

Fairness and accuracy sometimes trade off. Not always, but sometimes.

A simplified example: if your model is very accurate at predicting "good candidate" based on patterns from the past, and the past was biased, then the model accurately predicts biased patterns. Applying fairness constraints can reduce accuracy by a few points.

This is not an excuse for not applying fairness. It's context for making informed decisions:

How much accuracy are you willing to lose for how much fairness gained?
What's your minimum threshold for each?
How do you justify the decision to stakeholders?

The module teaches you to make and document these decisions, not to hide them.

The impossibility theorems

Deeper still: mathematically, you cannot satisfy all definitions of fairness simultaneously. Chouldechova (2017) and Kleinberg-Mullainathan-Raghavan (2017) proved it formally:

Demographic parity (equal rate per group) +
Equalized odds (equal error per group) +
Calibration (equal confidence means equal probability per group)

The three are mutually incompatible except in trivial cases (when the groups have identical distributions, which never happens in practice).

Implication: you have to choose which one to prioritize based on context. Capsule 05 covers how.

Defining a "protected group": it isn't trivial

Before you measure bias, you have to define with respect to what attribute:

Gender (M/F/non-binary/not disclosed)?
Race/ethnicity (categorized how)?
Age (in what brackets)?
Disability?
Religion?
Sexual orientation?
National origin?
Immigration status?
Veteran status?
Combinations (intersectionality)?

Every choice has complications:

In practice: you often don't have that data. Some jurisdictions (Europe) prohibit collecting certain attributes. How do you do bias testing without the data?

Partial solution: inference (ZIP → race), demographic sampling (surveying a sub-sample), proxy testing (counterfactuals with swapped markers).

Intersectionality: a Black woman can have a different experience than "women" in aggregate or "Black people" in aggregate. Gender bias alone does NOT capture intersectional discrimination. Capturing it requires disaggregating by gender × race, which increases the data requirement.

The module tackles these difficulties. We're not going to pretend it's trivial — because it isn't.

What we do NOT cover in this module

This module covers bias as a technical topic. There are related topics the guide covers in other modules:

Privacy of demographic data → Module 3.
Specific legal compliance (GDPR, EU AI Act) → Modules 4-6.
The integrating framework → Module 7.
A complete audit → Module 8.

Here we focus on: how to measure, detect, and mitigate bias. The legal consequences and organizational processes are covered elsewhere.

Module prerequisites

To get the most out of it:

You completed M1 and have your Ethics Impact Analysis ready.
You know Python and understand basic numpy, pandas, and scikit-learn.
Familiarity with ML concepts: classifier, training, predictions, confusion matrix.
Some basic statistics: distributions, ratios, proportions.

If you're missing some scikit-learn or numpy, guide #3 (Python + REST APIs) or the path's ML guides have the fundamentals.

Mindset: bias audit as a continuous process

An important idea for the whole module: bias isn't something you "fix once."

Reasons:

Models drift: your model's behavior in production changes over time.
Data drifts: the distribution of inputs changes (demographic shifts, new contexts).
Concepts drift: what "good candidate" or "loan default" means changes with economic/social conditions.

Implication: bias testing has to be continuous, wired into CI, monitored in production.

The mini-project's "Bias Audit Toolkit" (capsule 8) is designed to be run periodically, not as a one-time pre-deploy check.

Traps and common mistakes on this topic

1. "My system doesn't have bias because it doesn't use gender/race"

False. As Apple Card and Amazon Hiring demonstrated, proxies make the bias indirect. Without testing the output, you can't know.

2. "Applying fairness constraints guarantees fairness"

False. Applying one definition of fairness can violate others. Because of the impossibility theorems, you can't satisfy everything. The choice requires informed judgment.

3. "The tools (AI Fairness 360, Fairlearn) solve it"

The tools implement metrics and mitigations, yes. But you have to choose which ones to apply and how. The tools don't make the decisions for you.

4. "More data solves bias"

Partly. More balanced data helps. More imbalanced data just amplifies it. Balanced quality matters more than quantity.

5. "Fairness and accuracy are aligned"

Sometimes yes, sometimes no. Honestly: there's often a trade-off. The module teaches you to navigate it, not to deny it.

Self-check

1. Why is bias the most measurable ethical risk?

For three reasons:

Concrete mathematical formulas: demographic parity = P(positive | group A) / P(positive | group B). It's a number, not an opinion.
Automatable tests: they run in CI with no human intervention. Pass or fail.
A mature body of knowledge: decades of research on fairness in ML, statistics, and legal frameworks. You have literature and tools ready to go.

Compared to "transparency" or "accountability," which require qualitative judgment, bias is quantifiable. That's why it's the technical module to start with.

2. What's the problem with defining a "protected group"?

Several:

Complex categorization: is gender binary or a spectrum? is race by self-identification, by census categories, by inference?
Data not available: you often don't collect those attributes.
Legally prohibited to collect: Europe's GDPR limits it.
Intersectionality: aggregates hide intersectional effects.
It changes with context: what's protected in the US (race) can be different in other countries.

Partial solutions:

Proxy testing (counterfactuals with swapped markers, without needing self-identified data).
Demographic sampling (surveying a sub-sample for analysis).
Careful inference (ZIP → race with documented error rates).

The module tackles these difficulties instead of ignoring them.

3. Why aren't the impossibility theorems an excuse for not applying fairness?

The theorems prove that you can't satisfy all the metrics simultaneously. They do NOT prove that you can't satisfy any of them.

The right implication: you have to choose which one to prioritize based on context.

For hiring: equalized odds (equal error rates) is probably more relevant.
For healthcare: calibration (equal confidence-to-outcome ratio).
For advertising: demographic parity (equal exposure per group).

Every context has a naturally more relevant metric. Choosing it, justifying it, documenting it, monitoring it is engineering judgment, not an escape.

What you can't do is say "since I can't satisfy everything, I'll apply nothing." That would be like saying "since I can't prevent every bug, I don't test."

4. Why does a bias audit have to be continuous, not one-time?

Three kinds of drift affect AI systems:

Model drift: the model's behavior changes with scale, and edge cases show up in production.
Data drift: the distribution of inputs changes over time. Demographics shift, new use cases appear.
Concept drift: what "good outcome" means changes (regulation, social norms, economic conditions).

Any of the three can introduce bias that wasn't there in pre-deploy testing.

Solution: bias metrics as continuous monitoring in production, with alerts when they exceed thresholds. A formal re-audit every 6-12 months, or whenever there's a significant change.

The mini-project's Bias Audit Toolkit is designed for this: reusable, automatable, integrable into CI/monitoring.

Summary and next step

Bias is the most frequent, measurable, and actionable ethical risk in AI.
Concrete mathematical metrics (demographic parity, equalized odds, calibration, disparate impact) replace opinions.
Real trade-offs exist: fairness vs. accuracy, and impossibility theorems between metrics. The module teaches you to navigate them, not to hide them.
Defining a "protected group" is non-trivial: data availability, legal constraints, intersectionality.
A bias audit is continuous, not one-time. The mini-project's toolkit is designed for integration into CI and monitoring.

Checkpoint: you should be able to articulate why bias is where "ethics becomes engineering" in AI systems.

Bridge to the next capsule: capsule 02 introduces the most common metric: demographic parity (also called "independence" or "statistical parity"). You'll see the formula, implement it in code, apply it to a real dataset, and understand when it's the right metric (and when it isn't). It's the first step in turning your intuition about "bias" into a number you can measure and monitor.

Resources

Fairness in Machine Learning — Solon Barocas, Moritz Hardt, Arvind Narayanan (textbook) — a free academic reference.
AI Fairness 360 — IBM open source — a comprehensive toolkit.
Fairlearn — Microsoft open source — a Python alternative.
Fairness Indicators — Google — for TF/Keras.
Survey of Bias in ML (Mehrabi et al., 2021) — a survey paper.

Next: 02-fairness-metrics-demographic-parity.md — The first fairness metric: demographic parity.

Capsule 01 of 08 — Module 2 — AI Ethics & Compliance Guide

← Previous8. Mini-project: Ethics Impact Analysis Next →2. Fairness Metric: Demographic Parity