Module 1: Why Ethics Matters in AI Engineering

2. Case Study #1: Amazon Hiring Algorithm

Next →

Capsule description

In 2014, Amazon began building an ML system to automate resume review. The promise was simple: take the tens of thousands of applications they received every year, score them with a model trained on historical patterns of "what good hires look like," and return a ranking. Recruiters would review only the top 10%.

In 2018, Reuters revealed that the system had been quietly decommissioned. The reason: it systematically discriminated against women, and despite multiple attempts to fix it, the bias was structural in the data.

This capsule breaks the case down. Not to assign moral blame, but to extract the technical failure mechanism and the practices that would have prevented it. Because this wasn't a freak accident — it was the predictable result of applying ML to data that reflected a biased history, without adequate testing.

What happened

Timeline of events

  • 2014: Amazon's Edinburgh ML team starts building the system. Goal: automatic CV scoring on a 1-5 star scale.
  • 2014-2015: model training. Data: 10 years of CVs received by Amazon (2004-2014), with labels derived from who was hired and who was promoted.
  • 2015: the team discovers the model penalizes CVs containing the word "women's" (e.g., "captain of women's chess club"). It also lowers scores for graduates of two all-women colleges.
  • 2015-2017: mitigation attempts. They edit the model to treat specific terms as neutral. The issues persist.
  • 2017: the team concludes it cannot guarantee the model is fair. They shut it down for hiring in technical roles. They repurpose it for auxiliary tasks like data deduplication.
  • 2018: Reuters publishes the report. Amazon confirms the system was abandoned.

Technical magnitude

  • Training volume: ~500,000 CVs.
  • Roles covered: engineering, software development, data science.
  • Internal usage period: ~3 years before being discontinued.
  • Team involved: ~12 people on the Edinburgh ML team.

Why it happened: the technical failure mechanism

This is the critical section. It wasn't bad intent — it was a structural failure pattern that any model trained on historical data tends to reproduce if you don't do explicit bias testing.

Step 1: the dataset reflected a biased history

The training data was 10 years of CVs received by Amazon between 2004 and 2014, with positive labels for the CVs that led to hires and promotions.

Over those 10 years:

  • The tech industry was roughly 70-80% men in technical roles.
  • Amazon, like many tech companies, received proportionally fewer CVs from women.
  • The women who applied had similar hire rates — but the absolute volume was far smaller.

The dataset therefore carried a strong correlation between features associated with men and the positive label. Not because women were worse candidates, but because there were far fewer positive examples of women.

Step 2: the model learned the correlation, not the causation

An ML model, by design, learns statistical correlations. It has no notion of causation. It cannot distinguish between:

  • "This feature predicts 'good candidate' because it indicates technical skill."
  • "This feature predicts 'good candidate' because it correlates with male gender, and the dataset has a historical bias toward men."

The model learned both things with equal force. It detected that words like "executed" or "captured" were predictive of the positive label — because they were more common in men's CVs in the dataset. It detected that "women's" was predictive of the negative label — because CVs with that word had fewer positive labels in the dataset.

Step 3: explicit features weren't the only problem

Amazon, naturally, did not include "gender" as an explicit feature. That would have been obviously problematic.

The problem was that the model found proxies:

  • Words directly associated with gender ("women's").
  • Names of all-women colleges.
  • Language patterns correlated with gender in the CV corpus.
  • Probably other, subtler proxies that were never fully identified.

Removing the explicit proxy ("women's") did not remove the bias. The model simply recaptured it through other, subtler proxies. This is a known phenomenon called proxy discrimination, or redlining 2.0.

Step 4: the bias wasn't caught by traditional testing

The standard ML tests — accuracy, precision, recall, F1, AUC — passed. The model was good at predicting "who got an offer" because, as we said, that's what it learned.

What traditional testing did not measure:

  • Disparate impact: was the "high score" rate similar across demographic groups?
  • Equal opportunity: given equal qualifications, was the score similar across groups?
  • Calibration: for a given score, was the real probability of success similar across groups?

These are fairness metrics, distinct from accuracy. And they're the ones that would have caught the problem in week one.

The measurable cost

Direct financial cost

  • Lost development: ~3 years × ~12 people × senior ML eng salary ($180K) ≈ $6.5M in personnel cost.
  • Infrastructure: GPU compute for training + inference servers. Estimated at ~$500K-$1M.
  • Opportunity cost: the team could have built systems with shippable value.

Total estimate: $7-10M in direct cost.

Reputational cost

  • Reuters coverage, picked up by the NYT, Washington Post, Wall Street Journal, BBC, and dozens of tech outlets. Weeks of coverage.
  • A canonical case study in engineering faculties, business schools, and compliance training. Every new ML ethics course cites Amazon Hiring. That's permanent reputational damage.
  • Impact on hiring: senior women candidates report the case affected their decision to apply to Amazon. Hard to quantify, real.

Legal / regulatory cost

  • There was no direct lawsuit (the system never made binding hiring decisions before it was shut down, according to Amazon).
  • It accelerated regulation: the case is cited in the EU AI Act proposal, in NYC's AEDT Law (2023), and in US state-level proposals.
  • It increased scrutiny of Amazon's other systems: Rekognition, advertising, etc.

Cost to the people affected

If the system was used for 3 years to filter CVs even partially:

  • Women applicants who received a rejection without any real human review, based on a discriminatory score.
  • Women who were underranked and never made it to an interview.

Impossible to know the exact number — Amazon never released the data. But given the volume (tens of thousands of CVs/year), the order of magnitude is thousands to tens of thousands of affected candidates.

How it could have been prevented: the techniques that were missing

What follows isn't Monday-morning quarterbacking. The techniques existed in 2014. Some teams applied them. Amazon, like many others, didn't.

Technique 1: Explicit bias testing

What was done: testing accuracy, precision, recall.

What was missing: testing disparate impact and equal opportunity by demographic group (gender, age, ethnicity wherever it was inferable).

Concrete implementation:

# Pseudo-code for the test that was missing
def test_disparate_impact(model, test_set, protected_attribute):
    """
    Measures whether the model has significantly different
    positive-prediction rates across groups.
    """
    group_a = test_set[test_set[protected_attribute] == 'A']
    group_b = test_set[test_set[protected_attribute] == 'B']
    
    rate_a = model.predict(group_a).mean()
    rate_b = model.predict(group_b).mean()
    
    ratio = min(rate_a, rate_b) / max(rate_a, rate_b)
    
    # The 80% rule: ratio < 0.80 indicates disparate impact
    return {
        "ratio": ratio,
        "passes_4_5_rule": ratio >= 0.80,
    }

Applied to Amazon Hiring, this test would have come out red from the model's very first version.

Technique 2: Subgroup error analysis

What was done: measuring aggregate error.

What was missing: measuring error by subgroup. Does the model have a similar false-negative rate for men and women? Between graduates of all-women and all-male colleges? By age?

If the global error is 5% but the error on women's CVs is 15% and on men's is 3%, the model is not 95% accurate for women. It's 85%.

Technique 3: Counterfactual testing

What was done: test sets of real data.

What was missing: tests with identical CVs except for gender markers. If you swap "Mary Smith, captain of women's chess club" for "John Smith, captain of chess club" and the score changes, that is causal discrimination, not correlation.

Concrete implementation:

# Pseudo-code for a counterfactual test
def test_counterfactual_fairness(model, cv_template, gender_markers):
    cv_male = cv_template.replace_markers(gender_markers['male'])
    cv_female = cv_template.replace_markers(gender_markers['female'])
    
    score_male = model.score(cv_male)
    score_female = model.score(cv_female)
    
    delta = abs(score_male - score_female)
    
    # If the only difference is gender, the scores should be ~equal
    return {
        "delta": delta,
        "fair": delta < 0.05,  # configurable tolerance
    }

Applied to Amazon Hiring, this test would have revealed the bias in the first month.

Technique 4: Diversity on the evaluation team

What was done: a predominantly male team evaluating the outputs.

What was missing: including people from the potentially affected groups on the evaluation team. The women on the team would have noticed patterns the men didn't — because they know the language and the associated contexts.

This isn't performative ethics. It's information. Diverse teams catch failure modes that homogeneous teams don't see.

Technique 5: Documenting limitations (model cards)

What was done: standard internal technical documentation.

What was missing: a public model card (even for internal use) explicitly documenting:

  • Training data: M/F ratio, ratio by age, ratio by ethnicity.
  • Performance by subgroup, not just aggregate.
  • Known limitations.
  • Cases where the model should NOT be used.

If the model card had said "this model has 85% accuracy for women's CVs vs 95% for men's," the deploy to real hiring would have been blocked by compliance.

The structural lesson

Amazon wasn't a victim of bad luck or a lack of talent. The Edinburgh ML team was among the best in the world at the time.

The structural lesson is:

Models trained on historical data reproduce historical biases. Without an explicit bias testing process, this is practically guaranteed.

This applies to:

  • Hiring algorithms (Amazon).
  • Credit scoring (capsule 04).
  • Predictive policing.
  • Healthcare resource allocation.
  • Recommendation systems.
  • Any system trained on data from the past to predict the future.

Bias is not an anomaly — it's the default. The ethical process is what prevents it.

Traps and common mistakes in interpreting this case

1. "Amazon's engineers were bad"

No. They were very good. That's exactly the lesson — good engineers + biased data + inadequate process = discriminatory system. Team quality is not protection.

2. "The problem was the data, not the model"

Partly. The data was problematic, yes. But the full problem is:

  • Biased data +
  • A model that learns correlations without distinguishing causation +
  • Testing that didn't include bias metrics +
  • No process for detecting proxy variables +
  • A homogeneous team that didn't raise early flags.

"Bad data" alone doesn't capture the systemic failure.

3. "Just removing the gender feature solves the problem"

False, as Amazon proved. The model found proxies. Removing explicit features doesn't remove bias when proxies are available.

4. "This only applies to hiring; my system is different"

False. The mechanism (biased historical data → the model learns the bias → proxies make the problem invisible → traditional testing doesn't catch it) applies to any system trained on data from the past. If your system makes decisions that affect people, the risk applies.

Self-check

1. Why didn't removing "gender" as an explicit feature solve the problem?

Because the model found proxies — features correlated with gender that the model used to reconstruct the information. Examples:

  • Directly associated words: "women's" in clubs, sports.
  • All-women colleges appearing in CVs.
  • Language patterns (verbs, syntax) that differ statistically between genders in the corpus.

This is called proxy discrimination. Any sufficiently rich dataset will have proxies. The only way to mitigate it is to:

  1. Detect them explicitly (counterfactual tests).
  2. Apply post-training fairness techniques (re-weighting, adversarial debiasing).
  3. Validate disparate impact even after removing the explicit features.
2. What technical test would have caught the problem in the first month?

Disparate impact testing + counterfactual fairness testing.

Disparate impact: measure whether the "high score" rate was similar across men's and women's CVs in the test set. The 80% rule (4/5ths rule) would have come out red.

Counterfactual: take 100 real CVs, generate versions with the opposite gender markers (same content, different pronouns and gender-specific references). Run both versions through the model. Measure the score delta. If the delta exceeds tolerance, there's causal bias.

Both tests are cheap (they require no retraining), automatable (they run in CI), and were available in 2014.

3. Why is diversity on the evaluation team information, not virtue signaling?

Because people from affected groups catch failure modes that unaffected people don't notice. That's empirical information.

Example: a woman engineer on the Amazon team would have noticed faster that the model was lowering scores for phrases like "women's chess club" — because she recognizes the pattern of how women describe extracurricular activities on CVs differently than men do.

This isn't theory. It's a statistical signal that a brain trained in that context picks up. Homogeneous teams lose that signal systematically.

It's analogous to this: if you're testing a system in Spanish, you want testers who speak Spanish, not just English. Demographic diversity is the same idea applied to systems that affect diverse populations.

4. What's the general rule you take away from this case?

Models trained on historical data reproduce historical biases. Without an explicit bias testing and mitigation process, this is practically guaranteed.

That means for any AI system you're going to build:

  1. Assume your dataset has historical biases.
  2. Assume the model is going to learn them, even if you remove the obvious features.
  3. Assume traditional testing (accuracy/F1) will not catch them.
  4. Implement fairness-specific testing BEFORE you deploy.
  5. Document per-subgroup performance in model cards.
  6. Re-evaluate after deploy with real production data.

Bias is the default. The absence of bias is what takes work.

Summary and next step

  • Amazon built an AI hiring system that discriminated against women for ~3 years before it was decommissioned.
  • The mechanism: biased historical data → the model learns gender correlations through proxies → traditional testing doesn't catch it → deploy with no explicit bias testing.
  • The cost: ~$7-10M direct + permanent reputational damage + thousands of affected candidates + accelerated regulation.
  • The missing techniques: disparate impact testing, counterfactual fairness, subgroup error analysis, a diverse team, model cards.
  • The structural lesson: bias is the default when you train on historical data. The ethical process is what prevents it.

Checkpoint: you should be able to explain why Amazon failed to fix the system simply by removing the word "women's" from the features.

Bridge to the next capsule: capsule 03 covers the second case study: facial recognition with unequal error rates by race, which ended up causing real wrongful arrests. You'll see how a technical problem (imbalanced training data) translated into consequences in the physical world, not just in digital hiring. And how the regulatory response (municipal bans in San Francisco, Boston, and others) changed the legal landscape in under 24 months.

Resources

  1. Reuters report on Amazon hiring algorithm (2018) — the original story.
  2. Disparate impact analysis — fairness 101 — the legal and technical concept.
  3. Model Cards for Model Reporting (Mitchell et al.) — the canonical paper on model documentation.
  4. Fairness Indicators (Google) — open source tooling.
  5. The 4/5 Rule in EEOC guidelines — the US legal reference.

Next: 03-case-study-facial-recognition.md — Facial recognition with error rates 100x higher for people of color.

Capsule 02 of 08 — Module 1 — AI Ethics & Compliance Guide

← Previous1. Module introduction: Why Ethics Matters in AI EngineeringNext →3. Case Study #2: Facial Recognition Bias