Module 3: Privacy and Data Protection Fundamentals

1. Module introduction: Privacy and Data Protection Fundamentals

Capsule description

Module 2 covered how your system can discriminate. This module covers how your system can violate the privacy of the people whose data it processes.

These are the two faces of impact on people in AI:

Bias affects groups: systematic discrimination.
Privacy affects individuals: personal data mishandled.

Both risks often hit the same people: the groups that get discriminated against are frequently the most vulnerable to privacy mishandling too.

This module closes Phase 1 of the guide (Ethics Foundations). When you finish, you'll have three actionable tools:

Ethics Impact Analysis (M1).
Bias Audit Toolkit (M2).
Privacy Assessment (M3, this module).

With those, you're ready for Phase 2 (Regulatory Compliance: EU AI Act M4, GDPR M5, etc.) — because regulation formalizes these principles into legal obligations. Without understanding the principles, regulation is empty memorization.

Why is privacy in AI different?

Privacy in traditional software is already complex. But AI introduces unique risks that traditional frameworks don't cover well.

Traditional software vs. AI

Traditional software:

Stores data.
Displays it when the user asks.
Privacy risks: leaks (compromised databases), unauthorized access, improper retention.
Solutions: encryption, access control, retention policies.

AI systems:

They learn from the data: they transform it, integrate it into models, use it for future inference.
Individual data becomes part of the model.
Unique risks:
- Model inversion: reconstructing training data from the model.
- Membership inference: determining whether an individual was in the training set.
- Data leakage in prompts: sensitive information exposed via LLMs.
- Memorization: the model "remembers" training data verbatim.

These risks don't show up in generic privacy tutorials. And they're your technical responsibility as an AI engineer.

A concrete example: model inversion

Imagine a face recognition model trained on 10,000 faces. An attacker:

Gets access to the model (download, API).
Generates random faces.
For each face, measures how "confident" the model is.
Iterates: modifies the face until the model gives high confidence for a specific identity.
Result: a reconstruction of a face similar to those in the training set.

This doesn't require access to the training data. Only to the model.

For a face recognition system, that means: your deployed model can leak the faces of the people in your dataset.

A concrete example: data leakage in prompts

You built a RAG system for customer support. The system prompt includes:

You are an AI assistant. Customer information:
- Email: maria@company.com
- Internal ID: 12345
- Subscription tier: Premium
- Last support ticket: complaint about billing

Answer the customer's question.

The user asks a question. The LLM answers — but it can also:

Mention the customer's email in the response (if the LLM "hallucinates" or if the user asks for it).
Log the full prompt in monitoring systems → the email persists in the logs.
If you use an external API (OpenAI), the prompt may end up in their logs (depends on the contract).

Each one is a technical privacy leak that a traditional system wouldn't have.

What you'll learn

8 capsules:

Introduction (this capsule) — why privacy in AI is different.
Data minimization — what data you need vs. what you collect "just in case."
Anonymization vs. pseudonymization — when each applies.
Consent in AI — informed consent, granularity, revocation.
Retention and lifecycle — purge policies, deletion requests, the problem with already-trained models.
AI-specific privacy risks — model inversion, membership inference, data leakage.
Privacy by Design — privacy built into the architecture.
Mini-project — a Privacy Assessment for a real system.

By the end:

You apply data minimization to concrete systems.
You distinguish and choose between anonymization and pseudonymization.
You design consent flows appropriate for AI.
You define retention policies that account for models.
You identify AI-specific privacy risks and apply mitigations.
You produce a Privacy Assessment for a real system.
You connect it all to GDPR — ready for the regulatory module.

The mindset shift in this module

Before:

"Privacy = encrypt the database."

After:

"Privacy in AI = data minimization + informed consent + anonymization + retention that accounts for models + mitigation of AI-specific attacks. Each one requires different architectural decisions."

Privacy in AI is multi-dimensional. No single technique (encryption) covers the risks. Defense in depth, just like with bias.

The tension: privacy vs. utility

Worth acknowledging from the start: privacy and model utility are often in tension.

More data in training → a more accurate model.
But: more data = more PII potentially exposed.
Anonymization protects privacy.
But: it can destroy predictive signal.
Data minimization reduces risk.
But: it can reduce useful features.

This module doesn't pretend there's no trade-off. There is one. What it teaches is how to navigate it in an informed way:

Which data is truly necessary vs. nice-to-have?
What level of privacy preservation is appropriate for your context?
How do you combine techniques for the best of both?

Sometimes the right answer is "less data, a less accurate model, but far more respectful of privacy." Sometimes it's "more data with strong protections." The module gives you the framework to decide.

Privacy as an architectural decision

The crucial point: privacy is implemented in the architecture, not as a patch.

If your system stores PII unnecessarily, adding encryption doesn't solve the fundamental problem. The right question is: why are we storing this in the first place?

Privacy by Design (Cavoukian, 2009) sets out 7 principles:

Proactive not Reactive: prevent issues, don't respond to them.
Privacy as Default: the system must be private with no configuration.
Privacy Embedded: built into the design, not bolted on.
Full Functionality: privacy + functionality, not zero-sum.
End-to-End Security: protection across the whole lifecycle.
Visibility and Transparency: stakeholders understand the system.
Respect for User Privacy: the user at the center.

Every design decision has to pass through these lenses. Capsule 7 covers this in detail.

Who has to read this module

If your system:

✅ Processes people's data (even public data can be personal data depending on the jurisdiction).
✅ Stores chat conversations (a user's text is PII if it contains a name, email, address, etc.).
✅ Trains models on user data.
✅ Uses external LLM APIs with prompts containing user info.
✅ Has logs that can contain inputs/outputs.

Then this module applies.

For systems that do NOT process personal data at all (e.g., a model that classifies images of inanimate objects), the module is less critical — but the principles of minimization and design are still valuable.

How this connects to the rest of the guide

M1 Impact Analysis      ← stakeholders, risks
   ↓
M2 Bias Detection       ← fairness in outputs
   ↓
M3 Privacy (this one)   ← protecting individuals
   ↓
M4 EU AI Act            ← AI regulation
   ↓
M5 GDPR                 ← data protection regulation (formalizes M3)
   ↓
M6 NIST AI RMF + others ← complementary frameworks
   ↓
M7 Responsible AI Framework  ← integration
   ↓
M8 Final Audit          ← end-to-end application

This module is foundational for Phase 2. The principles you learn here are what regulation formalizes. GDPR didn't invent data minimization — it formalizes it as a legal obligation. The EU AI Act didn't invent data quality — it formalizes it for high-risk systems.

Without understanding the technical principles, regulation feels arbitrary. With them, regulation makes sense.

Common traps on this topic

1. "We have encryption, we're private"

Encryption protects against unauthorized access to storage. It does not prevent model inversion, membership inference, or leakage in prompts. Privacy in AI requires more than encryption.

2. "We anonymized the data, there's no PII"

K-anonymity can be broken with auxiliary data. Perfect anonymization is very hard when the dataset is rich. Assume re-identification is possible and design defense in depth.

3. "It's the user's responsibility to give us consent"

Under modern regulation (GDPR, CCPA), consent must be explicit, informed, granular, revocable. A boilerplate "I agree to terms" doesn't cut it. The engineer is responsible for implementing consent flows correctly.

4. "If the data is public, we can use it"

False. Data scraped from the internet without consent is problematic. GDPR explicitly requires a lawful basis for processing, even of public data. New AI-specific regulations also restrict scraping.

5. "Once trained, the model is safe"

False. The model contains information from the training samples. Model inversion and membership inference attacks demonstrate this. "Right to be forgotten" requests are particularly hard for trained models.

Self-check

1. Why is privacy in AI different from privacy in traditional software?

Three reasons:

The model learns from the data: individual data gets "integrated" into the parameters. It isn't stored explicitly, but it can be recovered via model inversion attacks.
Inference is PII processing: when a user query interacts with a model trained on other users' data, there's indirect propagation of PII.
LLM-specific risks: prompts can contain PII that gets logged, transmitted to external APIs, or leaked in outputs.

Result: traditional techniques (encryption, access control) are necessary but not sufficient. Privacy in AI requires mitigations specific to each risk.

GDPR and other regulations are updating to reflect this. The EU AI Act has provisions specific to AI privacy. NIST AI RMF includes explicit privacy considerations.

2. Why isn't "we have encryption" a sufficient answer for privacy in AI?

Encryption protects against unauthorized access to storage. But it does not prevent:

Model inversion: an attacker with legitimate access to the model (API, downloaded) reconstructs training data.
Membership inference: an attacker determines whether a specific individual was in the training set.
Memorization: the model "remembers" training samples verbatim and reproduces them with specific prompts.
Prompt leakage: PII in prompts gets logged, transmitted to APIs, shows up in outputs.
Inference attacks: sensitive attributes of the user inferred from queries.

Encryption is one layer. Defense in depth requires:

Data minimization (don't collect what you don't need).
Anonymization of training data where possible.
Differential privacy or federated learning for training.
Output filtering.
Strict logging policies.
Retention policies that account for models.

Encryption alone is like saying "we have a firewall" as your entire security strategy.

3. What's the central tension between privacy and AI utility?

More data and more detail = a better model. But also = more PII exposure.

Specifically:

Training data size: more samples = better generalization. But each sample is a person whose data is exposed.
Feature richness: more features per sample = more predictive signal. But more PII per individual.
Anonymization: protects privacy but destroys signal.
Differential privacy: adds noise = better privacy = worse accuracy.

How to navigate it:

Data minimization: collect what's genuinely needed, not "just in case."
Aggregation where possible: use cohort stats instead of individual records.
Federated learning: train without centralizing the data.
Differential privacy: noise calibrated to a privacy budget.
Sophisticated anonymization (k-anonymity, l-diversity, t-closeness).

Each one has a cost. Make the trade-off explicit, not hidden.

What does NOT work: pretending there's no trade-off ("we applied best practices and everything's fine").

4. How does this module connect to GDPR (M5)?

GDPR formalizes this module's principles into legal obligations:

Principle (M3)	GDPR Article
Data minimization	Art 5(1)(c) — "adequate, relevant, and limited to what is necessary"
Anonymization vs pseudonymization	Recital 26, Art 4(5)
Consent	Art 6, 7 — lawful basis and conditions for consent
Retention	Art 5(1)(e) — "kept for no longer than necessary"
Right to erasure	Art 17
Privacy by Design	Art 25

Without understanding the technical principles, GDPR is a list of articles to memorize. With them, GDPR is a legal framework that formalizes your engineering thinking.

Result: M5 will feel natural if M3 is internalized. "This legal obligation is exactly the data minimization principle" instead of "what does this article even mean?"

That's why M3 comes before M5: technical fundamentals first, the regulation that formalizes them second.

Summary and next step

Privacy in AI has unique risks that traditional software doesn't: model inversion, membership inference, prompt leakage, memorization.
Privacy is implemented in the architecture, not patched in. Privacy by Design.
There's a real tension between privacy and utility. The module teaches you to navigate it in an informed way.
The connection to GDPR: this module establishes the technical principles. GDPR (M5) formalizes them as legal obligations.
Defense in depth: encryption is one layer, not the whole solution.

Checkpoint: you should be able to articulate why privacy in AI requires specific mitigations beyond traditional software techniques.

Bridge to the next capsule: capsule 02 covers data minimization — the first practical principle. You'll learn to distinguish between "data we need" and "data we collect just in case," and how to apply minimization concretely to AI systems (which features, what retention, what granularity).

Resources

Privacy by Design (Cavoukian, 2009) — the foundational paper.
GDPR — Recital 26 (anonymization) — the legal definition.
Membership Inference Attacks (Shokri et al., 2017) — the technical paper.
Model Inversion (Fredrikson et al., 2015) — the technical paper.
Confidentiality of LLM-based Systems (OpenAI policy) — a provider's policy.

Next: 02-data-minimization.md — Data minimization applied to AI systems.

Capsule 01 of 08 — Module 3 — AI Ethics & Compliance Guide

← Previous8. Mini-project: Bias Audit Toolkit Next →2. Data Minimization in AI Systems