Module 1: Why Ethics Matters in AI Engineering

3. Case Study #2: Facial Recognition Bias

Capsule description

In 2018, Joy Buolamwini (MIT Media Lab) and Timnit Gebru published Gender Shades — a study measuring the accuracy of three commercial facial recognition systems (IBM, Microsoft, Face++) at gender classification, broken down by the intersection of race and gender. The results were scandalous:

White men: error rate 0.8%.
Black women: error rate 34.7%.
Difference: ~43x worse for Black women.

That study was the catalyst. Over the next 5 years: documented wrongful arrests, municipal bans (San Francisco 2019, Boston 2020, Portland 2020), proposed federal moratoriums, and multi-million-dollar civil lawsuits.

This capsule covers the case. It isn't theory — these are real people arrested, booked, and prosecuted because a model with unequal error rates was deployed into policing infrastructure without subgroup testing.

If Amazon Hiring is the case of "it could have been reversed before it reached production," facial recognition is the case of "it already reached the physical world, the harm is documented, and regulation is scrambling to catch up."

What happened

Gender Shades (2018)

Buolamwini and Gebru built a test set of 1,270 images of parliamentarians from 3 African countries and 3 European countries, balanced by gender (54% women, 46% men) and by skin tone (Fitzpatrick scale).

They tested it against:

IBM Watson Visual Recognition.
Microsoft Azure Face API.
Face++ (a Chinese company).

Results (error rate on binary gender classification):

Subgroup	IBM	Microsoft	Face++
Lighter men	0.3%	0.0%	0.7%
Lighter women	7.1%	1.7%	6.0%
Darker men	12.0%	5.9%	0.7%
Darker women	34.7%	20.8%	34.5%

The gap between lighter men and darker women: 30-40 percentage points. It wasn't marginal — it was structural.

The companies' initial reaction

IBM (June 2018): admits the problem, pulls the original training dataset, and publishes a more balanced one ("Diversity in Faces"). In 2020, Arvind Krishna (IBM's CEO) announces that IBM is abandoning facial recognition entirely, citing "the fight against racial bias and police misuse."
Microsoft (June 2018): ships an updated version with dramatically reduced error rates. Publishes a commitment not to sell facial recognition to police until there is federal regulation.
Amazon Rekognition: initially denies the problem applies to its systems. Independent studies (ACLU 2018, NIST 2019) show similar problems. In 2020, after the killing of George Floyd, Amazon imposes a 1-year moratorium on police use. Later extended indefinitely.

Real cases: wrongful arrests

The following are documented cases (2019-2023) where facial recognition produced wrongful arrests of innocent people:

Robert Williams (Detroit, 2020)

Facts: Williams, a Black man, was arrested at his home in front of his two daughters. Charge: shoplifting from a Shinola store.
Evidence: a facial recognition match between a security still and his driver's license photo.
Reality: Williams had never been in that store. The match was a false positive.
Processing: held for 30 hours. Charges eventually dismissed. He sued the city of Detroit.

Nijeer Parks (New Jersey, 2019)

Facts: Parks, a Black man, arrested for shoplifting + fleeing police + assault.
Evidence: a facial recognition match against a DMV photo.
Reality: Parks was 30 miles from the scene at the time of the crime. He had receipts and witnesses.
Processing: held for 10 days. Charges dismissed. He sued.

Randal Reid (Louisiana, 2022)

Facts: Reid, a Black man, arrested for stealing luxury handbags in Louisiana. He lived in Atlanta and had never been to Louisiana.
Evidence: a facial recognition match.
Reality: total misidentification.
Processing: held for 6 days. He sued.

Porcha Woodruff (Detroit, 2023)

Facts: Woodruff, a Black woman, 8 months pregnant, arrested at her home in front of her children. Charge: carjacking.
Evidence: a facial recognition match against a 2015 mugshot.
Reality: misidentification. Woodruff sued.

Pattern: in every documented case of a wrongful arrest driven by facial recognition, the victim was a Black person. That's not a random sample — it's the direct consequence of the model having higher error rates for darker skin.

The regulatory response

2019: San Francisco becomes the first US city to ban facial recognition use by municipal agencies.
2020: Boston, Portland (OR), Oakland, and other cities follow.
2020-2021: states like Massachusetts and Virginia limit police use.
2024: the EU AI Act classifies real-time biometric identification in public spaces as "high-risk" or "prohibited" in most cases.
2024: the state of New York imposes a moratorium on use by schools.

Why it happened: the technical mechanism

Imbalanced training data

The canonical datasets used to train facial recognition in the 2010s — LFW (Labeled Faces in the Wild), CelebA, IMDB-Wiki — were severely imbalanced.

LFW (one of the most used):

~77% men.
~83% light-skinned people (Fitzpatrick I-III).
~5% very dark-skinned people (Fitzpatrick V-VI).

If you train a model on that data, the model sees vastly more examples of white men than of Black women. As a result, it learns the distinctive features of white men better. For Black women it has fewer examples to generalize from — and the error goes up.

It's the same mathematical principle any statistician knows: fewer examples → higher variance in the estimate → more error. Applied to ML: less training data for a subgroup → a worse model for that subgroup.

The model learns what it sees

It's not that the models are "racist" in any cognitive sense. It's that they learn the dataset's patterns. If the dataset is 80% light skin, the model specializes in light skin.

The technical architecture also contributed:

Traditional color thresholding: many pre-2018 systems used color thresholds that assumed uniform lighting. Dark skin, under common lighting conditions (offices with warm light, a nighttime security photo), has less contrast, which affects feature detection.
Feature extraction calibrated for lighting: a similar problem. The face as a whole may not be detected at all before recognition even begins.

Inadequate testing

As in Amazon Hiring, the traditional tests passed. A global accuracy of 95% sounds fine — but it hides 0.8% for white men and 34.7% for Black women. The average is misleading.

What was missing: disaggregated testing by demographic subgroup, reported explicitly, and blocking the deploy if the gaps exceeded a threshold.

Buolamwini and Gebru didn't invent a new technique. They simply measured what the companies weren't measuring.

Deploying to high-stakes contexts without a disclaimer

The companies sold these systems to:

Police departments (to identify suspects).
Airports (identity verification).
Schools (access control).
Borders (immigration screening).

These are high-stakes: errors carry severe consequences (arrests, deportations, denying access to legitimate people).

A basic ethical decision would have been: do NOT sell systems with known unequal error rates for high-stakes use until the gaps close. Or at minimum, sell them with explicit requirements of no standalone use (always with human verification).

That didn't happen until AFTER the scandals.

The measurable cost

Human cost

Each of the documented wrongful arrests involved:

Detention (hours to days).
Criminal charges (which stay on the record even when dismissed).
Legal costs for the defense.
Personal and family trauma (some arrests happened in front of children).
Job loss in some cases.
Persistent reputational damage.

And the documented ones are a fraction. It's likely there have been hundreds or thousands of facial-recognition-driven arrests with mistaken identifications, where the suspect never sued and the case simply "went badly" without coverage.

Financial cost to the companies

IBM: abandoned the entire facial recognition product line (2020). Cost: years of lost revenue + written-off R&D.
Amazon Rekognition: an ongoing moratorium. The US government market, lost.
Microsoft: lost government market share during its voluntary moratorium.
Lawsuits: Robert Williams sued Detroit for $X million (settled), Parks sued NJ, Reid sued Louisiana.

Conservative total financial cost across the sector: hundreds of millions to billions of dollars in lost revenue + legal costs + written-off R&D.

Regulatory cost

Dozens of US cities with bans.
States with limits.
The EU AI Act with a high-risk/prohibited classification.
The reputation of the entire AI industry, damaged.

The case is cited in every legislative proposal on AI risk in the US and Europe since 2019.

How it could have been prevented: the missing techniques

Technique 1: Disaggregated testing by subgroup

Missing: testing that reports accuracy/error by demographic subgroup (race × gender) on every release.

Implementation:

def test_subgroup_performance(model, test_set):
    subgroups = test_set.groupby(['race', 'gender'])
    
    results = {}
    for name, group in subgroups:
        predictions = model.predict(group['image'])
        accuracy = (predictions == group['label']).mean()
        results[name] = {
            'n': len(group),
            'accuracy': accuracy,
            'error_rate': 1 - accuracy,
        }
    
    # Blocking: if any subgroup's error rate is > 2x the best one
    best_error = min(r['error_rate'] for r in results.values())
    worst_error = max(r['error_rate'] for r in results.values())
    
    if worst_error / best_error > 2:
        raise FailedQAGate(
            f"Disparate error rates detected. "
            f"Best: {best_error:.2%}, Worst: {worst_error:.2%}"
        )
    
    return results

This test, wired into CI, would have blocked the problematic releases.

Technique 2: Balanced training datasets

Missing: validating the demographic composition of the training datasets before using them.

Implementation: for each dataset, compute the distribution by demographics (wherever it's inferable). If subgroups have < 10% representation, that alone is a signal of future underperformance in that group.

Mitigations:

Oversampling of minority subgroups.
Data augmentation specific to darker skin (carefully, so you don't introduce new artifacts).
New datasets: like IBM's Diversity in Faces (2018) or FairFace (Karkkainen & Joo, 2021).

Technique 3: Adversarial testing

Missing: testing with edge cases specifically designed to detect bias.

Example: take pairs of images that differ only in skin tone (same facial type, same pose, different tone). Measure whether the model gives consistent results. If it doesn't, there's bias.

Technique 4: Explicit confidence thresholds

Missing: deploying with confidence thresholds calibrated per subgroup. If the model has 99% confidence for white men but 70% for Black women, that should propagate into the output.

Implementation: the system's output must include the confidence interval, calibrated by subgroup. If confidence < threshold, don't return a match — return "uncertain, requires human review."

This alone, applied to the wrongful arrest cases, would have prevented them. Williams, Parks, Reid, Woodruff — in every case, the match's confidence was uncertain, but the system returned "match" as if it were certainty.

Technique 5: Disclaimers and use restrictions

Missing: contractual documentation prohibiting standalone use in high-stakes decisions.

Implementation: the sales contract with police should have included clauses like:

"Do not use as the sole evidence for an arrest."
"Always require human verification of a match."
"Report per-subgroup performance on the local deploy before use."

Microsoft eventually adopted something like this. The other companies didn't, until regulation forced them.

The structural lesson

Models with unequal error rates across demographic subgroups, deployed in high-stakes decisions, produce disproportionate harm to the groups they serve worst.

This applies to:

Facial recognition (this case).
Speech recognition (worse on non-native accents).
Medical imaging (worse on groups underrepresented in the training data).
Credit scoring (capsule 04).
Any model whose global accuracy is presented without a breakdown.

Average accuracy is misleading. Subgroup analysis is non-negotiable when the system affects people.

Traps and common mistakes in interpreting this case

1. "It's an industry problem, not my small project's problem"

False. Any project that uses pre-trained models (including OpenAI, Anthropic, and Google APIs) inherits their biases. If you build on those APIs in a high-stakes context, the biases are your problem.

2. "The solution is just more data"

Partly. More data helps, but balanced data matters more than more data. 10x more imbalanced data doesn't solve the problem; 1x balanced data can.

3. "My system isn't high-stakes"

Define high-stakes. If your system:

Decides whether a person gets a loan, an offer, a discount, priority → high-stakes.
Moderates user content → high-stakes (false positives are censorship).
Recommends content to millions of users → high-stakes (scale amplifies effects).

Few AI systems in production are truly low-stakes. Assume high-stakes and test accordingly.

4. "If the global accuracy is high, it's fine"

The core mistake of the facial recognition case. Global accuracy hides subgroup gaps. Always break it down.

Self-check

1. Why isn't 95% global accuracy enough?

Because it can hide an unequal distribution. 95% global can mean:

99% for group A (the majority).
70% for group B (the minority).

If group B is 5% of the population but 100% of the worst-served cases, the individual experience of a group B user is 70% accuracy, not 95%.

The right metrics:

Accuracy by subgroup (race × gender × age, for example).
Error rate ratio: worst / best. If > 2x, there's structural bias.
False positive rate by subgroup: especially important in verification systems.
False negative rate by subgroup.

Report all four or report nothing.

2. Why are wrongful arrests from facial recognition always Black people?

Because the system has a higher error rate for darker skin. A higher error rate → a higher probability of a false match → a higher probability of misidentifying someone.

If the system has:

A false positive rate of 0.1% for light skin.
A false positive rate of 5% for dark skin.

Then, when it's run against a mugshot database, an innocent person with dark skin is 50x more likely to be falsely matched to an actual suspect.

It's math, not intent. Intent doesn't matter when the effect is discriminatory.

3. What changes when the system requires "human verification" before acting?

A lot — but it isn't a complete solution.

What improves: if a human verifies the match before an arrest, the model's errors get filtered out before they have consequences. In theory, false positives never reach the physical world.

Why it isn't a complete solution:

Automation bias: humans tend to trust machine outputs, especially when they have no contradicting information. If the model says "98% match," the human reviewer probably accepts it.
Confirmation bias: the human reviewer knows the model said "match" → they go looking for confirming evidence.
Volume: if a reviewer gets 1,000 matches/day, they can't investigate each one carefully.

A more robust solution: a high confidence threshold + matches limited to leads, not actionable evidence + explicit training for human reviewers on the model's limitations.

4. Why did IBM abandon facial recognition entirely?

Because they concluded that the reputational + ethical + legal risk outweighs the revenue. The product line was small in revenue terms for IBM, but the brand damage from being associated with problematic use in policing was large.

It's a reasonable business decision. When the costs of the externalities exceed the benefits, you drop the line.

This logic applies to other systems too. Before investing in an AI line, ask: does the potential harm from misuse exceed the likely revenue? If yes, don't build it.

Summary and next step

Gender Shades (2018) revealed error rates 30-40 points worse for Black women in commercial facial recognition systems.
Real documented cases of wrongful arrests (Williams 2020, Parks 2019, Reid 2022, Woodruff 2023) — all of them Black people.
Mechanism: imbalanced training data → a model that's better on the majority → global testing hides the gaps → deploy into high-stakes with no disaggregated metrics.
Cost: human (lives upended), financial (IBM drops the product, lawsuits, settlements), regulatory (municipal bans, EU AI Act high-risk).
Structural lesson: global accuracy deceives. Disaggregated testing by subgroup is non-negotiable in systems that affect people.

Checkpoint: you should be able to explain why a system with 95% global accuracy can be flatly unacceptable once you break the numbers down.

Bridge to the next capsule: capsule 04 covers the third case study: Apple Card and credit scoring — how a scoring system for a credit card triggered public accusations of gender discrimination (David Heinemeier Hansson, Steve Wozniak, and others), ended up under investigation by the NY Department of Financial Services within 60 days, and demonstrated how indirect gender proxies produce disparate impact even when gender is not an explicit feature. The technical payoff: how to detect proxy variables.

Resources

Gender Shades (Buolamwini & Gebru, 2018) — the original paper with the results.
Robert Williams ACLU case — the first documented wrongful arrest.
NIST FRVT — Face Recognition Vendor Test — ongoing benchmarking.
FairFace dataset (Karkkainen & Joo, 2021) — an open source balanced dataset.
Joy Buolamwini — TED talk — conceptual context.

Next: 04-case-study-credit-scoring.md — Apple Card and discrimination through indirect proxies.

Capsule 03 of 08 — Module 1 — AI Ethics & Compliance Guide

← Previous2. Case Study #1: Amazon Hiring Algorithm Next →4. Case Study #3: Apple Card and Credit Scoring