Module 1: Why Ethics Matters in AI Engineering

7. Ethics Impact Analysis: the framework

Capsule description

The first 6 capsules established why ethics matters in AI. This capsule gives you the how: a concrete framework for doing an Ethics Impact Analysis on any AI system.

It's the equivalent of the threat model in security: it doesn't guarantee the system is ethical/secure, but it guarantees the decisions were made with visibility into the risks.

The framework has 5 phases:

System characterization — what it does, in what context, with what data.
Stakeholder identification — who affects it or is affected by it.
Failure mode mapping — how it can cause harm.
Severity and probability scoring — how bad, how likely.
Prioritization and mitigation — what to handle first, with what techniques.

By the end, you'll be able to produce a professional, defensible, reusable 3-5 page document. Capsule 08 walks you through applying it to a real case.

Before you start: why do an Impact Analysis

Three reasons, in increasing order of force:

1. Risk visibility

With no process, risks stay implicit. Every team member assumes "someone else" is thinking about them. Result: nobody thinks about them explicitly.

An Impact Analysis forces the question: what can go wrong and to whom? It doesn't skip things because "they're obvious" — it documents them.

2. Communication with stakeholders

When you talk to a PM, legal, an exec — you need a shared artifact everyone reads and signs off on. The Impact Analysis is that artifact.

"Look, I identified these risks, scored the severity, and I'm proposing these mitigations. Are we aligned?"

With no document, everything lives in conversations that get forgotten. With a document, there's a paper trail.

3. Compliance evidence

Under modern regulation (EU AI Act, NIST AI RMF), you have to demonstrate due diligence. The Impact Analysis is the primary evidence.

If a regulator or a lawsuit shows up: "what due diligence did you do before deploying?" Answer: "here's the Ethics Impact Analysis, dated, signed, with mitigations implemented and verified."

With no document, the answer is "we thought about it" — which legally is equivalent to not having done it.

Phase 1: System characterization

Before you identify risks, you have to describe exactly what it is you're analyzing. Vague = useless.

Characterization questions

Answer with enough detail. If you can't, the system isn't defined well enough to deploy.

About the system:

What does it do? One sentence, no jargon.
- Good: "Recommends candidates to interview based on their résumé and the job description."
- Bad: "Intelligent AI system for recruiting optimization."
What type of model?
- Classification, regression, ranking, generation, retrieval, agent.
- Trained from scratch, fine-tuned, or an external API.
Who provides the model?
- In-house, OpenAI, Anthropic, Google, open source (Llama), etc.
What is the formal output?
- A 0-1 score, a ranking, a binary decision, generated text, an embedding, an action.
How is the output used?
- An automatic decision (no human in the loop).
- An assisted decision (a human approves).
- Informational only.

About the data:

Where does the training data come from?
- Public datasets, scraped, generated internally, purchased.
What kinds of personal data does it contain?
- Direct PII, indirect PII, sensitive data (health, biometrics, religion, orientation, etc.).
Is there clear consent?
- Documented, explicit, granular, revocable.
What's the demographic composition of the dataset?
- Wherever inferable: gender, age, ethnicity, geography, language, etc.

About the deployment:

What jurisdictions will it be used in?
- US only, US + EU, global.
- This determines which regulations apply.
Who are the users?
- Individual end users, companies, governments, healthcare workers, etc.
At what scale?
- Dozens of people, thousands, millions, billions.
How reversible is the output?
- Reversible (a product recommendation).
- Hard to reverse (a hiring rejection).
- Irreversible (publishing content, a medical decision).

An example characterization

System: Resume Screener — recommends the Top 10 candidates to interview from the applicants to a job posting.

Type: Classifier + ranking. LLM-based (GPT-4) with prompt engineering, no fine-tuning.

Output: a ranked list of candidate IDs with a 0-1 score.

Use: the recruiter sees the ranking and decides who to interview. An "assisted decision" — a human makes the final call.

Data: applicants' résumés (free-form). Internal job descriptions. We do NOT use historical hire data (a deliberate decision to avoid the Amazon Hiring scenario).

Composition: applicants are mostly US (85%), tech roles (engineering, product), ages 22-50.

Jurisdictions: US (EEOC, NYC AEDT Law) + UK + EU (EU AI Act high-risk, given that it's hiring).

Scale: ~50,000 applications/year at mid-size companies; potential to scale 10x.

Reversibility: medium. The human recruiter can overrule, but automation bias is documented — a lot of people don't overrule.

Phase 2: Stakeholder identification

Who interacts with the system, or is affected by its decisions?

Stakeholder categories

Direct (they interact with the system):

Primary users: whoever operates the system (e.g., recruiters in Resume Screener).
Subjects: whoever the system makes decisions about (e.g., applicants).

Indirect (affected by the outputs):

Family/dependents of the subjects (affected by decisions that impact them).
Communities represented or under-represented.
Future employers / colleagues of the subject.

Systemic:

Regulators: agencies that can investigate or fine you.
Society at large: if the system scales, macro effects (perpetuating social biases).

For each stakeholder identified, document

Who they are (a concrete description).
How they interact with the system.
Which of the system's decisions affect them.
What outcomes are possible for them (positive and negative).
Specific vulnerabilities of the group (if applicable).

An example stakeholder map

Stakeholder	How they interact	Decisions that affect them	Possible outcomes	Vulnerabilities
Applicants	Submit a résumé	The ranking that decides the interview	Interview vs. no interview	Women, minorities, age > 50, non-standard profiles
Recruiters	Operate the system	Trust in the ranking; time invested	Efficiency (positive); blindness to outliers (negative)	Automation bias
Hiring managers	Receive pre-filtered candidates	A restricted candidate pool	Good hires vs. missed talent	Indirect (no access to the rejected)
The company	Builds/deploys it	Productivity; legal exposure	Efficient hiring vs. lawsuits	EEOC/AEDT/EU AI Act
Under-represented communities	Under-represented in hiring outcomes	Reduced access to opportunity	Access (positive if fair); exclusion (negative if biased)	Historically affected by hiring bias
Society at large	Aggregate hiring patterns	Economic mobility, equality	The distribution of opportunity	Concentration of power, occupational segregation

Phase 3: Failure mode mapping

How can the system cause harm?

Standard failure mode categories

Bias / Discrimination:

Disparate impact by gender, race, age, etc.
Indirect proxy discrimination.
Unequal subgroup error rates.

Privacy:

Leakage of personal data in the outputs.
Re-identification of individuals in "anonymous" datasets.
Inference of sensitive attributes from the outputs.

Accuracy:

Hallucinations (LLMs).
Confidence misrepresentation.
Edge case failures.

Misuse:

Use for unintended purposes.
Input manipulation (adversarial).
Use by bad actors.

Transparency:

Lack of explainability.
A false sense of objectivity.
Trust calibration issues.

Accountability:

Diffusion of responsibility.
No way to appeal.
Lock-in of algorithmic decisions.

Safety:

Physical harm (in systems with physical effects).
Psychological harm (manipulation, addiction).
Economic harm.

For each failure mode, document

The failure mode: a concrete description.
The mechanism: how it would happen technically.
Affected stakeholders: which ones from Phase 2.
The harm outcome: what happens to the stakeholder.

An example (continuing with Resume Screener)

Failure mode	Mechanism	Stakeholders	Harm outcome
Gender bias	The LLM learned from an internet with historical bias → it favors masculine-coded language in résumés	Female applicants	Under-ranked, fewer interviews
Age bias	The LLM implicitly penalizes résumés with > 20 years of experience	Applicants > 50	Under-ranked, ageism
Non-standard résumé bias	Model trained on Western résumés, fails on different formats	International applicants	Under-ranked
Hallucination	The LLM invents a "match" between skills not present in the résumé	Recruiters, applicants	A decision based on false data
Privacy leak	The prompt includes the full résumé → it can leak into logs	Applicants	Exposure of personal data
Automation bias	The recruiter trusts the ranking 100% without review	Mid-ranked applicants	Good talent discarded
No appeal	Applicants don't know why they were rejected	Applicants	No possibility of feedback/improvement

Phase 4: Severity and probability scoring

Each failure mode is scored on two axes:

Severity

How bad is the harm if it happens?

Level	Description	Examples
Low	Inconvenience, minor economic harm	A bad product recommendation
Medium	Significant economic harm, frustration	Rejection in a job process
High	Material harm, lost opportunities, psychological harm	Systematic discrimination in hiring
Critical	Physical harm, liberty, life; irreversible	A wrongful arrest, a wrong medical decision

Probability

How likely is it to happen?

Level	Description	Expected frequency
Low	A rare edge case	< 1% of cases
Medium	Uncommon but recurring	1-10% of cases
High	A common case	10-50% of cases
Very high	Nearly the default	> 50% of cases

The Severity × Probability matrix

              LOW   MEDIUM  HIGH  VERY HIGH  ← Probability

CRITICAL       🟡    🔴     🔴     🔴
HIGH           🟢    🟡     🔴     🔴
MEDIUM         🟢    🟢     🟡     🔴
LOW            🟢    🟢     🟢     🟡

↑
Severity

🟢 = acceptable risk / monitor 🟡 = mitigation required 🔴 = deploy blocker

An example (continuing with Resume Screener)

Failure mode	Severity	Probability	Risk	Justification
Gender bias	HIGH	HIGH	🔴	LLMs show documented bias; the impact in hiring is high
Age bias	HIGH	HIGH	🔴	Same as gender
Non-standard résumé bias	MEDIUM	HIGH	🔴	Common but solvable
Hallucination	HIGH	MEDIUM	🔴	LLMs demonstrably hallucinate
Privacy leak	HIGH	LOW	🟡	Low if logs are handled correctly
Automation bias	HIGH	HIGH	🔴	Recruiters tend to trust rankings
No appeal	MEDIUM	VERY HIGH	🔴	100% by default

7 of the 7 identified failure modes land in the 🔴 or 🟡 zone. The system cannot be deployed without mitigations.

Phase 5: Prioritization and mitigation

For each 🔴 or 🟡 risk, propose a concrete mitigation.

The structure of a mitigation

Each mitigation must include:

A specific technique (not an abstract one).
A metric to verify it works.
An acceptance criterion (when it counts as mitigated).
Ownership (who's responsible).
Estimated implementation time.

Example mitigations

Risk	Mitigation	Metric	Acceptance criterion	Owner	Time
Gender bias	Counterfactual testing in CI: 100 résumés with gender markers swapped, measure the score delta	Mean score delta	< 0.05	ML eng	2 weeks
Age bias	Disparate impact testing by age bracket	Approval ratio worst/best	> 0.80	ML eng	2 weeks
Non-standard résumé bias	Diversify the prompt examples; test with international résumés	Subgroup accuracy	> 0.85 in every subgroup	ML eng	1 week
Hallucination	Force constrained output: only skills/companies present in the résumé	Hallucination rate	< 1%	ML eng	1 week
Privacy leak	Don't persist the résumé in logs; redact in prompts	Audit logs	Zero PII in logs	DevOps	1 week
Automation bias	UI changes: require the recruiter to view at least the top 30, not the top 10. Show the limitations explicitly	Recruiter usage analytics	80% review beyond the top 10	Frontend	2 weeks
No appeal	Implement a feedback channel: applicants can request an explanation	Feedback request rate	Works at p99 < 24h	Eng + customer ops	4 weeks

Total estimate: 4 weeks of work + ongoing monitoring

Until the mitigations are in place, the system must NOT be deployed to production.

The final document

The Ethics Impact Analysis is delivered as a 3-5 page document with this structure:

# Ethics Impact Analysis: [System]

## 1. Executive summary
- System: [one sentence]
- Primary stakeholders: [list]
- Critical risks: [N risks in the 🔴 zone]
- Recommendation: [proceed with mitigations / pause / do not proceed]

## 2. System characterization
[Phase 1]

## 3. Identified stakeholders
[Phase 2 — table]

## 4. Failure modes
[Phase 3 — table]

## 5. Risk assessment
[Phase 4 — matrix + table]

## 6. Mitigation plan
[Phase 5 — table with ownership and timelines]

## 7. Post-deploy monitoring plan
[Metrics that will be checked periodically]

## 8. Decision log
- [Date] Analysis completed by [author]
- [Date] Reviewed by [stakeholders]
- [Date] Approved by [decision maker]

## 9. Next review
[Date — typically 6 months post-deploy, or whenever there's a significant change]

When and how to use this framework

When to apply it

Before starting a new AI system that affects people.
Before deploying to a new audience/jurisdiction.
After a significant change: new dataset, new model, new domain.
Periodically post-deploy (every 6-12 months).

When NOT to apply it (it's overkill)

Systems with no impact on people (e.g., an internal song recommender for your own playlist).
Prototypes in dev only (but apply it before any external exposure).
Systems with full human review always (where the AI is 100% advisory with no automation).

Who does it

Owner: the system's lead engineer. It's a technical responsibility.

Reviewers:

PM (to understand the product risks).
Legal (regulatory compliance).
Security (overlap with privacy).
Senior eng (technical quality review).

Decision maker: the director/VP who approves the deploy. They sign the document.

Traps and common mistakes

1. Doing the analysis but not implementing the mitigations

A document with no action is useless. If you identify a 🔴 risk and deploy without mitigation, you've made it worse (now there's a paper trail proving you knew).

2. Generic analysis

"Bias could exist" is not analysis. "Gender bias detectable via counterfactual testing with threshold X" is analysis.

3. Skipping Phase 1 (characterization)

If the characterization is vague, all the rest of the analysis is vague. Push for detail until the system is unambiguously described.

4. Forgetting indirect stakeholders

The most affected are often not the direct users. In hiring, the applicants (subjects) are more affected than the recruiters (users).

5. Subjective severity

"It's low because it's only a recommendation, not a decision" — but if the recruiter always trusts it, it's effectively a decision. Severity has to be based on real impact, not on intent.

6. Probability without data

"Low probability" with no measurement isn't useful. If you're going to estimate probability without measuring, it's high or very high by conservative default.

Self-check

1. Why is Phase 1 (characterization) a prerequisite for the other 4?

Because without a clear characterization:

You don't know who the stakeholders are (Phase 2 impossible).
You don't know which failure modes are relevant (Phase 3 vague).
You can't score severity (Phase 4 conjectural).
You can't propose specific mitigations (Phase 5 abstract).

If your Phase 1 says "the system is intelligent AI for hiring optimization," everything after it is noise. If it says "a ranking of the Top 10 candidates to interview from the applicants to a job posting, used by a recruiter to decide interviews, in US/UK/EU jurisdictions, at a scale of 50K applications/year, an assisted decision with a human making the final call," the rest of the analysis becomes concrete.

A vague characterization = a vague analysis = a false sense of due diligence.

2. What do you do if every failure mode you identified is 🔴 (blocking)?

Three options:

Implement all the mitigations before deploy. This delays the launch but produces a robust system.
Reduce the scope: launch for a subset of use cases where some risks don't apply. E.g., launch Resume Screener in the US only to start (reduces EU AI Act regulatory exposure); or only for internal roles (reduces the stakes).
Don't launch: if every risk is irreducible at a reasonable cost, the project may not be viable as designed. Rethink the architecture or cancel.

What you do NOT do: launch anyway. Blocking = blocking. If you launch, you expose the company and yourself personally.

Real-world: many AI systems need this pause. The argument is "the cost of delaying is enormous" — and the answer is "the cost of an incident is 50x higher; the math is against rushing."

3. How do you assign severity when you have no precedents?

Conservative heuristics:

By default, assume high severity if it affects people. Lower it only with evidence.
Consider reversibility: irreversible (publishing, a medical decision) > hard to reverse (a rejection) > reversible (a product recommendation).
Consider scale: 1 person affected vs. millions. The same per-capita harm, but a different aggregate cost.
Consider the vulnerabilities of the affected group: historically affected minorities carry a higher severity because harm compounds.
Consider precedents in similar systems: if Amazon Hiring was HIGH severity, your hiring system probably is too.

When in doubt, raise it: it's easier to lower severity later with evidence than to raise it after an incident.

4. How do you balance the Impact Analysis against development velocity?

The right balance:

Low-risk systems (don't affect people, low scale): skip it or do a light version. 1 page max.
Medium-risk systems (affect people but low scale, reversible): full analysis but light. 2-3 pages. 1-2 days of work.
High-risk systems (affect people at scale, irreversible): full analysis. 5+ pages. 1-2 weeks of work, including stakeholder reviews.

The cost calculation: 1-2 weeks of Impact Analysis vs. $4M in expected incident cost (capsule 05). Overwhelmingly positive ROI.

For velocity: integrate the Impact Analysis into the design, not as post-design overhead. Do it in the feature's sprint planning, not after.

Summary and next step

The Ethics Impact Analysis is the equivalent of the threat model in security: risk visibility + a paper trail.
5 phases: characterization → stakeholders → failure modes → severity/probability → prioritization/mitigation.
The final document is 3-5 pages, professional, defensible, reusable.
Who does it: the lead engineer, with review by PM/legal/security/senior eng.
When it's done: before starting a new system, before significant changes, and periodically post-deploy.
The cost calculation massively favors doing the analysis: 1-2 weeks vs. $4M+ in expected incident cost.

Checkpoint: you should be able to describe the 5 phases, what happens in each one, and what deliverable it produces.

Bridge to the next capsule: capsule 08 is the mini-project. You'll apply the framework to a real AI system (it can be your path capstone, a system from your job, or a case we provide). The deliverable is a complete 3-5 page Ethics Impact Analysis that demonstrates mastery of the process. That document is the first tangible artifact of your ethical workflow, and you'll refine it in later modules as you learn specific techniques (bias testing in M2, privacy in M3, compliance in M4-6).

Resources

Microsoft Responsible AI Impact Assessment Template — a commercial template.
Google Model Cards Toolkit — open source.
NIST AI RMF Playbook — step-by-step guidelines.
EU AI Act — Risk Assessment guidance — regulatory reference.
Algorithmic Impact Assessment (Canada Treasury Board) — a government framework.

Next: 08-mini-project-ethics-impact-analysis.md — Produce your own Ethics Impact Analysis.

Capsule 07 of 08 — Module 1 — AI Ethics & Compliance Guide

← Previous6. The engineer's ethical responsibility Next →8. Mini-project: Ethics Impact Analysis