7. Privacy by Design in AI Systems

Capsule description

Privacy by Design (Cavoukian, 2009) is the framework for building privacy in from the design stage, not as a post-deployment patch.

It has 7 principles:

1. Proactive not Reactive — anticipate issues, don't respond to them.
2. Privacy as the Default Setting — private with no configuration.
3. Privacy Embedded into Design — integral, not bolted on.
4. Full Functionality — privacy + functionality, not zero-sum.
5. End-to-End Security — protection across the whole lifecycle.
6. Visibility and Transparency — stakeholders understand it.
7. Respect for User Privacy — the user at the center.

This capsule covers each one applied to AI systems specifically, translating the principles into concrete architectural decisions.

GDPR Art 25 makes Privacy by Design a legal obligation. That's why covering these principles is a prerequisite for the regulatory modules (M5 GDPR).

Principle 1: Proactive not Reactive

The concept

Anticipate privacy issues before they happen, don't respond afterward.

Application to AI

Reactive (the default mode):

• Deploy the AI system.
• A privacy issue surfaces.
• Patch the issue.
• Repeat.

Proactive:

• Identify privacy risks during design.
• Apply the Ethics Impact Analysis (M1) before code.
• Apply the Privacy Assessment before code.
• Build the mitigations into the architecture.

Concrete actions

## Pre-design checklist

- [ ] Ethics Impact Analysis completed (M1/08)
- [ ] Privacy Assessment completed (M3/08)
- [ ] Threat model includes AI-specific risks (membership inference, model inversion, etc.)
- [ ] Data flow diagram created with privacy annotations
- [ ] Identified all third parties involved (and their privacy practices)
- [ ] Stakeholder review of architecture for privacy implications

Doing this before writing code prevents 80% of privacy issues from existing at all.

Principle 2: Privacy as the Default

The concept

The system must be private without the user configuring anything. "Opt in to share," not "opt out to stay private."

Application to AI

Default behaviors should be privacy-preserving:

• Conversations NOT used for training unless the user opts in.
• Logs anonymized by default; raw mode requires an explicit enable.
• Personalization off unless the user enables it.
• Data shared with third parties: NOT by default.

The concrete pattern

# UI/Settings page

class UserSettings:
    def __init__(self, user_id):
        self.user_id = user_id
        # Privacy-preserving defaults
        self.training_consent = False           # Default OFF
        self.analytics_consent = False           # Default OFF
        self.personalization = False             # Default OFF
        self.share_with_partners = False         # Default OFF
        # Necessary for service: defaults to ON
        self.service_operation = True            # Required for service

The user can opt in to additional uses. The service works fully with the defaults (privacy-preserving).

The counter-example

# ❌ NOT privacy by default
class UserSettings:
    training_consent = True       # Default ON
    analytics_consent = True       # Default ON
    share_with_partners = True     # Default ON

Common in commercial products. It violates privacy by default.

Principle 3: Privacy Embedded into Design

The concept

Privacy is an architectural concern, not a feature bolted on at the end.

Application to AI

Architectural decisions with privacy implications:

Every architectural decision should explicitly consider privacy.

Concrete: a data flow diagram with privacy annotations

[User Input]
    │ encrypted in transit (TLS)
    ▼
[API Gateway]
    │ minimal logging, no PII
    │ rate limited
    ▼
[App Server]
    │ PII fetched only if needed
    │ system prompt with no static PII
    ▼
[LLM API]   →  Third party (OpenAI)
    │           Contract: no training on data
    │           Logs: 30 days max
    │ response filtered for PII leaks
    ▼
[App Server]
    │ minimal logging
    │ response cached 24h max
    ▼
[Response]
    encrypted in transit (TLS)
    │
    ▼
[User]

Every arrow has a privacy annotation. The architectural review covers each one.

Principle 4: Full Functionality

The concept

Privacy and functionality are not zero-sum. Good design achieves both.

Application to AI

A common false dichotomy: "if we want personalization, we have to collect everything."

Reality:

• Personalization without storing PII: use an embeddings/preferences vector, not raw history.
• Recommendations without tracking: collaborative filtering on hashed user IDs.
• Analytics without individual tracking: differential privacy aggregates.
• AI improvement without training on personal data: synthetic data, federated learning.

Pattern: privacy-preserving personalization

# ❌ Privacy-violating
class Personalizer:
    def get_recommendations(self, user_id):
        # Pull entire user history
        history = db.get_full_history(user_id)
        # Pull demographic data
        demographics = db.get_demographics(user_id)
        # Send all to recommender
        return recommend_model.predict(history, demographics)

# ✅ Privacy-preserving
class Personalizer:
    def __init__(self):
        self.user_vectors = load_user_vectors()  # Pre-computed embeddings
    
    def get_recommendations(self, user_id):
        # Use only the user's preference vector (anonymous embedding)
        user_vector = self.user_vectors.get(user_id)
        # Recommend based on vector similarity
        return self.recommend_by_similarity(user_vector)
    
    def update_vector(self, user_id, action):
        # Update vector incrementally based on action
        # No need to store the action itself long-term
        self.user_vectors[user_id] = update(
            self.user_vectors[user_id],
            action
        )
        # Optionally, action discarded after vector update

It achieves personalization with a much smaller PII surface.

Principle 5: End-to-End Security

The concept

Protection across the whole lifecycle: collection, processing, storage, transmission, deletion.

Application to AI

Data lifecycle stages with security:

1. Collection
   ├ TLS for transmission
   ├ Input validation
   └ Minimization at collection
   
2. Processing
   ├ Authorization checks
   ├ PII redaction in logs
   └ Audit trails
   
3. Training
   ├ Encryption at rest
   ├ Access controls
   ├ Differential privacy (where applicable)
   └ Audit who accessed
   
4. Inference (production)
   ├ TLS to model API
   ├ Output filtering
   ├ Rate limiting
   └ Monitoring abnormal queries
   
5. Storage
   ├ Encryption at rest (always)
   ├ Backups encrypted
   └ Access logged
   
6. Deletion
   ├ Secure deletion (overwrite, not just unlink)
   ├ Cascade across systems
   └ Document

Every stage has security measures. A single failure → exposure. Defense in depth.

Concrete: encryption everywhere

# At rest
db.execute("CREATE TABLE conversations (... ) WITH (encryption='AES256')")

# In transit
app.config['SSL_REQUIRED'] = True
app.config['MIN_TLS_VERSION'] = 'TLSv1.2'

# In API calls to third parties
openai_client = OpenAI(
    api_key=settings.openai_key,
    # Always TLS
)

# In backups
backup_config = {
    'encryption': True,
    'kms_key': settings.kms_backup_key,
}

Principle 6: Visibility and Transparency

The concept

Stakeholders (users, regulators, internal teams) must understand how the system handles privacy.

Application to AI

Transparency artifacts:

1. A privacy policy (user-facing): plain language, comprehensive, current.

2. A model card: technical documentation of the model's behavior, limitations, biases, and data sources.

3. A data flow diagram: how data moves through the system.

4. A privacy impact assessment (the Privacy Assessment from M3/08): a formal evaluation.

5. Audit trails: what data was processed, when, and by whom.

Concrete: a model card example

# Model Card: Customer Support Chatbot

## Model Details
- **Type**: Fine-tuned LLM (Llama 3 base)
- **Version**: v2.1
- **Date**: 2026-04
- **Fine-tuning data**: 50K anonymized customer support conversations

## Intended Use
- **Primary**: respond to customer support queries
- **Out of scope**: medical, legal, financial advice

## Training Data
- **Source**: internal customer support tickets
- **Anonymization**: PII redacted (names, emails, phones, addresses)
- **Consent**: customers consented at ticket submission with opt-out option
- **Time period**: 2020-2025

## Performance
- **Accuracy**: 87% (on test set)
- **By language**: English 89%, Spanish 85%, French 80%
- **Limitations**: known issues with technical jargon

## Privacy Considerations
- **No PII in system prompt**: customer-specific data fetched on-demand
- **Conversations not used for training without explicit consent**
- **Logs anonymized within 30 days**
- **Output filtering for PII leaks**

## Bias Considerations
- **Tested**: equal performance across age groups, languages
- **Known limitation**: lower accuracy in non-Western names

## Maintenance
- **Re-evaluation**: every 6 months
- **Contact**: ai-team@acme.com

Open documentation builds trust + supports compliance.

Principle 7: Respect for User Privacy

The concept

The user at the center. Their interests are the primary consideration, not an afterthought.

Application to AI

Concrete behaviors:

1. Privacy-preserving defaults (Principle 2).
2. Easy access to the user's own data (Right to Access).
3. Easy deletion of the user's data (Right to Erasure).
4. Easy export of the user's data (Right to Portability).
5. Easy withdrawal of consent.
6. Clear notification when data practices change.
7. Genuine accountability when issues occur.

Concrete: user-facing privacy controls

[User Settings - Privacy]

Data & Privacy:

┌─────────────────────────────────────────────────────────┐
│  📊  See your data                                      │
│      What we know about you                          [→]│
├─────────────────────────────────────────────────────────┤
│  📥  Download your data                                 │
│      Get a copy in JSON format                       [→]│
├─────────────────────────────────────────────────────────┤
│  🗑️  Delete your account                                │
│      Permanently remove your data                    [→]│
├─────────────────────────────────────────────────────────┤
│  ⚙️  Privacy settings                                    │
│      Control how your data is used                   [→]│
├─────────────────────────────────────────────────────────┤
│  📝  Privacy policy                                     │
│      How we handle data                              [→]│
└─────────────────────────────────────────────────────────┘

All accessible. Self-service. No support tickets required.

Integrated application: an AI system with Privacy by Design

An example: a customer support chatbot with all the principles:

# 1. Proactive: completed Privacy Assessment before code
# 2. Privacy default: training consent OFF until explicitly granted
# 3. Embedded: data flow diagram + architectural review

class ChatbotService:
    def __init__(self):
        self.consent_manager = ConsentManager()
        self.audit_logger = AuditLogger()
    
    async def handle_query(self, user_id, query):
        # Audit log (Principle 5: end-to-end security)
        self.audit_logger.log({
            'user_id': hash(user_id),  # Hashed in logs
            'query_hash': hash(query),
            'timestamp': now(),
        })
        
        # Fetch only what's needed (Principle 4: full functionality, minimum data)
        user_context = self.fetch_minimal_context(user_id)
        
        # Generate response without PII in prompt (Principle 3: embedded)
        response = await self.generate_response(query, user_context)
        
        # Filter response (defense in depth)
        safe_response = self.filter_pii_from_response(response)
        
        # Optionally, save conversation (only if user consented)
        if self.consent_manager.has_consent(user_id, 'training'):
            await self.save_for_training(query, response, user_id)
        
        return safe_response
    
    def fetch_minimal_context(self, user_id):
        """Fetch only fields needed for context."""
        return self.db.fetch_fields(user_id, fields=['subscription_tier', 'language'])
        # NOT: full user profile, payment info, etc.

# Architectural decisions documented

architecture:
  data_storage:
    primary: PostgreSQL with encryption at rest
    cache: Redis with 24h TTL
    backups: encrypted, 90-day rotation
  
  api_security:
    transport: TLS 1.3
    auth: JWT with 1h expiry
    rate_limiting: 100/hour/user
  
  llm_provider:
    vendor: OpenAI
    contract: enterprise (no training on data)
    region: US (data residency)
  
  logging:
    level: minimal
    pii_redaction: automatic
    retention: 30 days
  
  monitoring:
    privacy_metrics:
      - failed_consent_checks
      - pii_in_logs (alert if non-zero)
      - data_purge_completion

# Public artifacts (Principle 6)

- /privacy: full privacy policy
- /privacy/data: how data is handled
- /privacy/your-data: user's own data
- /privacy/preferences: privacy settings
- /privacy/contact: data protection contact

- Model card: github.com/.../model-card.md
- Data flow diagram: in privacy policy
- Privacy assessment: available on request to regulators

This is a system with Privacy by Design fully implemented.

Common traps

1. "Privacy is the legal team's problem"

Privacy is architectural. The legal team can advise, but engineers implement. "By Design" means engineers own it.

2. Adding privacy "later"

When it's easier (shipping fast), privacy gets deferred. Then it becomes "we'll fix it next release." It never gets done. Build it in from the start or never.

3. Privacy theater

The UI says "your privacy matters" while the data is shared with 50 partners. Users notice. Eventually regulators notice. Be authentic or don't claim it.

4. Not auditing the architecture

Privacy by Design only works if it's verified periodically. Re-review the architecture every 6 months. New features can break your privacy posture.

5. Treating Privacy by Design as a checklist

It's a mindset + a framework, not just checkboxes. Engineers should think about privacy whenever they think about anything.

Self-check

architecture:
  user_db:
    location: EU (Frankfurt)
    encryption: AES-256 at rest
    backups: encrypted, 30-day retention
    access_logs: enabled, 1-year retention
  
  api:
    minimum_response_fields: true
    rate_limiting: enabled
    audit_logging: enabled
    pii_in_logs: filtered
  
  third_parties:
    - openai: enterprise contract, no training, US region
    - cloudflare: TLS termination only, no PII access

Summary and next step

• Privacy by Design = 7 principles (Cavoukian, 2009): proactive, default, embedded, full functionality, end-to-end, visibility, respect.
• GDPR Art 25 makes it a legal obligation.
• Applying it to AI requires translating each principle into specific architectural decisions.
• Privacy as the default is the shift: from opt-out to opt-in for anything non-essential.
• End-to-end security + transparency = defense in depth.
• Full functionality is achievable alongside privacy — it's a false dichotomy.

Checkpoint: you should be able to evaluate an AI system against the 7 principles and identify the gaps.

Bridge to the next capsule: capsule 08 is the mini-project: the Privacy Assessment for a real system. You'll apply everything you've learned — minimization, anonymization, consent, retention, AI-specific risks, Privacy by Design — in a comprehensive evaluation. A 4-6 page document, defensible, reusable.

Resources

1. Privacy by Design — Cavoukian (2009) — the foundational paper.
2. GDPR Art 25 — Data protection by design and by default — the legal reference.
3. NIST Privacy Framework — the US framework.
4. Federated Learning (McMahan et al., 2017) — privacy-preserving training.
5. Differential Privacy 101 — an accessible introduction.

Next: 08-mini-project-privacy-assessment.md — A Privacy Assessment for your system.

Capsule 07 of 08 — Module 3 — AI Ethics & Compliance Guide