Module 3: Privacy and Data Protection Fundamentals

8. Mini-project: Privacy Assessment

Next →

You close the module (and Phase 1 of the guide) by applying everything you learned to a comprehensive Privacy Assessment of a real AI system.

It's the equivalent of the Ethics Impact Analysis (M1/08) but focused on privacy specifically. A 4-6 page document that demonstrates mastery of the process and produces a reusable artifact.

Deliverable spec

Minimum viable

  • ✅ System defined: a data-focused characterization.
  • ✅ Data inventory: a list of all data types, sources, and lifecycles.
  • ✅ Minimization assessment: each data type evaluated for necessity.
  • ✅ Anonymization decisions: for each piece of PII, which technique to apply.
  • ✅ Consent flow: documented for each purpose.
  • ✅ Retention policy: a table with periods + automation.
  • ✅ AI-specific risks: an assessment of membership inference, model inversion, prompt leakage, memorization.
  • ✅ Privacy by Design checklist: the 7 principles evaluated.
  • ✅ Mitigation plan: concrete actions with owners and timelines.

Stretch goals

  • ⭐ Implementation: code at least 1 mitigation (PII filter, consent manager, etc.).
  • ⭐ Stakeholder review: legal/security feedback.
  • ⭐ GDPR pre-mapping: identify which articles each section addresses.
  • ⭐ A periodic re-evaluation calendar: schedule the next review.

Choosing the system

The same options as M1/08 and M2/08:

Option A: your AI Engineering path Capstone.

Option B: a system from your job (with permission).

Option C: a provided case study:

C1: Healthcare Triage System (recommended for a privacy focus)

An AI system that takes patient symptoms via web chat and classifies urgency. LLM-based with prompt engineering. Training data: 10,000 anonymized historical conversations. Deploy: a clinic with 50K visits/year in the US (HIPAA + state laws).

Particularly rich for privacy because:

  • HIPAA applies (US).
  • Health data is "sensitive" under GDPR (a special category).
  • The LLM provider is a third party.
  • Conversations potentially contain PII.

C2: Customer Support Chatbot

A RAG-based system for customer support. Conversations stored, indexed, used for FAQ generation. 1M users globally (US + EU + LATAM).

Privacy aspects:

  • GDPR (EU users).
  • CCPA (CA users).
  • LATAM regulations (Brazil's LGPD, Argentina's PDPA).
  • A cross-jurisdictional compliance challenge.

C3: Loan Approval System

As in M1/08. Predicts default probability. Trained on 5 years of internal data including financial PII.

Privacy aspects:

  • GLBA (US financial).
  • FCRA (credit reporting).
  • GDPR (EU customers).
  • Sensitive financial data → high stakes if leaked.

Document structure

# Privacy Assessment: [System Name]

## 1. Executive Summary
- System overview (1 paragraph)
- Privacy risk level (low/medium/high)
- Key findings
- Critical mitigations needed

## 2. System Characterization
[See section below]

## 3. Data Inventory
[Comprehensive list]

## 4. Data Minimization Assessment
[Per data type evaluation]

## 5. Anonymization & Pseudonymization Decisions
[What technique per data type]

## 6. Consent Management
[Flows, purposes, granularity]

## 7. Retention Policy
[Periods, automation, deletion processes]

## 8. AI-Specific Privacy Risks
[Each attack assessed]

## 9. Privacy by Design Evaluation
[7 principles checklist]

## 10. Mitigation Plan
[Concrete actions]

## 11. Monitoring & Re-evaluation
[Ongoing privacy posture]

## 12. Decision Log
[Authors, reviewers, dates]

4-6 pages. Compact but comprehensive.

Detailed implementation

Section 2: System Characterization

Different from the Ethics Impact Analysis — the focus is on data flows, not just functionality.

## 2. System Characterization

**System**: Healthcare Triage Chatbot

**Functionality**: Receives patient symptoms via web chat, classifies urgency 
(immediate / 24h / week / non-urgent), recommends action.

**Data flows**:
1. User submits symptoms via web form (TLS to our server)
2. Server constructs prompt with symptoms + medical knowledge context
3. Prompt sent to LLM API (OpenAI, US region)
4. LLM response received
5. Response shown to user
6. Conversation saved (encrypted)

**Data lifecycle stages**:
- **Input**: user types symptoms in web form
- **Processing**: prompt construction, LLM call, response classification
- **Storage**: conversation stored 90 days (per medical record retention)
- **Training**: 5% of conversations (with consent) used for fine-tuning quarterly

**Jurisdictions**: US (HIPAA, state-specific medical privacy laws)

**Third parties**: 
- OpenAI (LLM provider, BAA signed for HIPAA)
- AWS (hosting, BAA signed)
- Stripe (payment, no medical data)

**User base**: US adults seeking medical guidance; ~50K patients/year

Section 3: Data Inventory

## 3. Data Inventory

| # | Data Type | Source | Sensitivity | Storage | Retention |
|---|-----------|--------|-------------|---------|-----------|
| 1 | Patient name | User input | High (PII) | DB encrypted | Account life + 90d |
| 2 | DOB | User input | Medium | DB encrypted | Account life + 90d |
| 3 | Symptoms text | User input | Critical (PHI) | DB encrypted | 90 days |
| 4 | Triage result | System output | Medium | DB encrypted | 90 days |
| 5 | LLM prompt | Generated | Critical (PHI) | LLM API logs | 30 days (per OpenAI) |
| 6 | LLM response | Generated | High | DB + logs | 90 days |
| 7 | User account | User input | Medium | DB encrypted | Until deletion |
| 8 | Payment data | Stripe | High | Stripe (not us) | Per Stripe |
| 9 | Audit logs | System | Low | Encrypted store | 6 years |

**Total data types**: 9
**With PHI**: 4 (#1, #3, #5, #6)
**With PII**: 7 (excluding #4, #9)

Section 4: Data Minimization Assessment

## 4. Data Minimization Assessment

For each data type:

### Patient name (#1)
- **Necessary?** YES — required for medical record creation
- **Granularity?** Full name needed (legal medical record requirement)
- **Recommendations?** None — required minimum

### DOB (#2)
- **Necessary?** YES — for age-based triage decisions
- **Granularity?** Full DOB for medical accuracy. Could be year only?
  - **Decision**: Full DOB. Medical accuracy outweighs privacy gain of bracket.

### Symptoms text (#3)
- **Necessary?** YES — core functionality
- **Granularity?** As detailed as user provides
- **Recommendations?** None — minimum for triage

### LLM prompt (#5)
- **Necessary?** YES — required to query LLM
- **Improvements**:
  - Strip patient name, DOB before sending to LLM (LLM doesn't need)
  - Only symptoms + age bracket sent
  - Pre-action: implement prompt sanitization

### Audit logs (#9)
- **Necessary?** YES — HIPAA compliance requires audit trail
- **Improvements**:
  - Reduce content: don't log full prompts/responses, only IDs
  - Implement PII redaction in logs

**Action items**:
- [ ] Implement LLM prompt sanitization (strip name, exact DOB)
- [ ] Implement log PII redaction
- [ ] Owner: ML engineer + DevOps. Timeline: 2 weeks.

Section 5: Anonymization Decisions

## 5. Anonymization & Pseudonymization Decisions

| Data | Strategy | Reasoning |
|------|----------|-----------|
| Patient name | None (required identifier) | Medical record requires identification |
| Symptoms (in prompt to LLM) | Pseudonymize patient name; remove if mentioned | LLM doesn't need name to assess symptoms |
| Training data (5% sample) | k-anonymity (k=10) + name removal | Aggregates while preserving pattern signal |
| Logs | PII redaction | Don't need actual values for debugging |
| Aggregated stats (analytics) | Differential privacy | Public stats need DP guarantee |

**Decision**: NOT using DP for training because:
- Accuracy cost too high (medical accuracy critical)
- Sample size sufficient for k-anonymity to be meaningful
- Combined with consent + access controls, sufficient privacy

Section 6: Consent Management

## 6. Consent Management

**Purposes** (each with separate consent):

1. **Service operation** (REQUIRED):
   - "Use my data to provide medical triage service"
   - Cannot use service without

2. **Anonymized research** (OPTIONAL):
   - "Allow my anonymized conversations to be used for AI safety research"
   - Default OFF

3. **Model improvement** (OPTIONAL):
   - "Allow my anonymized conversations to be used to improve our AI"
   - Default OFF

4. **Marketing** (OPTIONAL):
   - "Allow contact about new health features"
   - Default OFF

**Implementation**:
- Consent UI separate checkboxes, all unchecked by default except #1
- Consent recorded with timestamp + version
- Consent revocable from privacy settings
- Revocation triggers exclusion from future training (within 24h)
- Existing models contain residual data (documented to user)

**Information disclosure**:
- Plain-language privacy notice
- Layered: summary, details, technical
- FAQ accessible

Section 7: Retention Policy

## 7. Retention Policy

| Data Type | Retention | Mechanism | Verification |
|-----------|-----------|-----------|--------------|
| Active conversations | 30 days | Daily cron job | Weekly audit query |
| Triage results | 90 days | Daily cron job | Weekly audit query |
| Account data (active) | While active + 90d post-cancel | Auto-trigger | Monthly audit |
| Backups | 90-day rolling | S3 lifecycle | Auto-managed |
| Audit logs | 6 years | Immutable storage | Compliance review |
| Training data (5% sample) | Until consent revoked OR 1 year | Quarterly job | Per quarter check |

**Right to Erasure handling**:
- API endpoint for user request
- 24h verification + processing
- Deletes from primary, marks for backup cleanup
- Excludes from future training
- Documents limitation: existing trained models contain traces
- 30-day completion timeline

**Action items**:
- [ ] Implement automated retention cron jobs
- [ ] Implement Right to Erasure API
- [ ] Document handling of model traces in privacy notice

Section 8: AI-Specific Risks

## 8. AI-Specific Privacy Risks

### Risk 1: Membership Inference

**Concern**: Attacker queries the model to determine if specific patient was in training set.

**Severity**: Medium (revealing membership in healthcare dataset is sensitive)

**Probability**: Low (limited model access; rate limiting in place)

**Mitigations**:
- DP NOT used (accuracy trade-off too high for medical)
- Output regularization: return classification only, not confidence scores
- Rate limiting: 100 queries/hour/user
- Monitoring: anomalous query patterns

**Status**: Acceptable with mitigations.

### Risk 2: Model Inversion

**Concern**: Reconstruct training symptoms from model.

**Severity**: High (medical data reconstructed)

**Probability**: Low (LLM with much larger pre-training dilutes inversion)

**Mitigations**:
- Training data minimal in fine-tuning
- API access only (no weight access)
- Rate limited

**Status**: Acceptable but monitor research developments.

### Risk 3: Prompt Leakage

**Concern**: System prompt contains medical context; injection could leak.

**Severity**: High

**Probability**: Medium (LLMs have known injection vulnerabilities)

**Mitigations**:
- Patient PII NOT in system prompt (already implemented)
- Output filter for sensitive medical terms
- Prompt structure: clear separation of instructions and user input
- Two-LLM pattern for high-stakes queries

**Status**: Mitigated.

### Risk 4: Memorization

**Concern**: Fine-tuned model reproduces verbatim training patient data.

**Severity**: Critical

**Probability**: Medium (LLMs known to memorize)

**Mitigations**:
- Training data deduplicated
- PII redacted before training (names, DOBs, addresses, etc.)
- Periodic memorization auditing (quarterly)
- If memorization detected, retrain with stricter PII filter

**Status**: Mitigated, monitoring required.

Section 9: Privacy by Design Evaluation

## 9. Privacy by Design Evaluation

| # | Principle | Status | Evidence |
|---|-----------|--------|----------|
| 1 | Proactive | ✅ | This Assessment + Ethics Impact Analysis pre-deployment |
| 2 | Default | ✅ | Consent OFF for non-essential; minimal data collected |
| 3 | Embedded | ✅ | Architecture review with privacy annotations; data flow diagram |
| 4 | Full functionality | ✅ | Privacy + functionality co-designed; not zero-sum |
| 5 | End-to-end security | ✅ | TLS, encryption at rest, audit logs, access controls |
| 6 | Visibility | ⚠️ | Privacy policy yes; model card pending |
| 7 | Respect for users | ✅ | Easy access, deletion, settings; clear notices |

**Action items**:
- [ ] Publish model card (#6 partial gap)
- [ ] Owner: ML lead. Timeline: 1 week.

Section 10: Mitigation Plan

## 10. Mitigation Plan

| # | Action | Priority | Owner | Timeline |
|---|--------|----------|-------|----------|
| 1 | Implement LLM prompt sanitization | High | ML eng | 2 weeks |
| 2 | Implement log PII redaction | High | DevOps | 2 weeks |
| 3 | Automated retention cron jobs | Medium | DevOps | 4 weeks |
| 4 | Right to Erasure API | High | Backend | 4 weeks |
| 5 | Memorization auditing setup | Medium | ML eng | 6 weeks |
| 6 | Publish model card | Medium | ML lead | 1 week |
| 7 | Two-LLM pattern for high-stakes | Low | ML eng | 8 weeks |
| 8 | Privacy notice update | High | Product + legal | 2 weeks |

**Total estimated effort**: 8 weeks of distributed work.
**Critical path**: items #1, #2, #4, #6, #8 must complete before deploy.

Section 11: Monitoring & Re-evaluation

## 11. Monitoring & Re-evaluation

**Continuous metrics**:
- Failed consent checks (alert if non-zero)
- PII detected in logs (alert if non-zero)
- Right to erasure processing time (target < 24h)
- Memorization audit results (target < 1%)
- Privacy-related user complaints (track trend)

**Re-evaluation calendar**:
- Quarterly: review metrics, update if needed
- Semi-annually: full Privacy Assessment review
- On material change: any architecture change triggers re-review

**Triggers for ad-hoc review**:
- New jurisdiction added
- New data source added
- New third-party integration
- Significant feature change
- Privacy incident (internal or industry)

Section 12: Decision Log

## 12. Decision Log

| Date | Action | By | Notes |
|------|--------|-----|-------|
| 2026-05-08 | Privacy Assessment v1 created | [Author] | Initial assessment |
| 2026-05-15 | Reviewed by legal | [Legal lead] | No changes; approved |
| 2026-05-22 | Reviewed by security | [Security lead] | Approved with note on #2 |
| 2026-06-01 | Approved for deployment | [VP Product] | Conditional on items #1, #2, #4, #6, #8 |
| 2026-12-01 | Scheduled re-review | — | Calendar reminder |

How to proceed

Step 1: block the time (6-10 hours)

This is more laborious than the Bias Audit. Set aside the right amount of time.

Step 2: the data inventory first (2-3h)

Without a complete inventory, everything else is vague. Worth the time.

Step 3: the minimization assessment (1-2h)

For each data type, ask the hard question: is it really necessary? Most teams find 20%+ is over-collection.

Step 4: the technical sections (2-3h)

Anonymization, consent, retention, AI risks. Concrete decisions.

Step 5: the PbD checklist (30 min)

Evaluate against the 7 principles. Identify the gaps.

Step 6: the mitigation plan (1h)

Concrete actions. Owners. Timelines.

Step 7: review

Stakeholder review (legal, security, ML lead). Iterate.

Closing the module

8 capsules

  1. Module introduction.
  2. Data minimization.
  3. Anonymization vs. pseudonymization.
  4. Consent in AI.
  5. Retention and lifecycle.
  6. AI-specific privacy risks.
  7. Privacy by Design.
  8. Mini-project: Privacy Assessment (this capsule).

What you have now

Three integrated tools (Phase 1 complete):

  1. Ethics Impact Analysis (M1) — risks + stakeholders.
  2. Bias Audit Toolkit (M2) — fairness measurement.
  3. Privacy Assessment (M3) — a comprehensive privacy posture.

Applied to a system, they give you a 360° view of its responsibility.

We start in the next module

Module 4: EU AI Act Deep Dive marks the start of Phase 2: Regulatory Compliance.

The transition:

"You understand impact, bias, and privacy as fundamentals. Now we look at how the world's most important regulation (the EU AI Act) formalizes these concepts into legal obligations with risk categories, mandatory requirements, and fines up to €35M."

Privacy connects directly: the EU AI Act has data quality and governance requirements that overlap with minimization. GDPR (M5) formalizes it completely.

Prep work: none. M3's conceptual framework prepares you naturally.

Resources for the exercise

  1. Microsoft Privacy Threat Modeling — the methodology.
  2. NIST Privacy Framework — the complete framework.
  3. ICO Privacy Impact Assessment template — the UK regulator's template.
  4. GDPR Art 35 — Data Protection Impact Assessment — when a DPIA is required.
  5. HIPAA Security Risk Assessment Tool — for healthcare.

Capsule 08 of 08 — Module 3 — AI Ethics & Compliance Guide

End of module 3. End of Phase 1: Ethics Foundations. Continue with module 4 (EU AI Act Deep Dive).

← Previous7. Privacy by Design in AI SystemsNext →1. Module introduction: EU AI Act Deep Dive