Module 3: Privacy and Data Protection Fundamentals

5. Retention and the Data Lifecycle

Capsule description

Collecting data is only the beginning. The rest of the lifecycle — how long, where stored, when purged, how to delete on request — defines a large part of your privacy posture.

The default tendencies in software:

Data persists forever unless explicitly deleted.
Backups pile up.
Logs are never purged.
Caches keep copies.
"Just in case" wins every decision.

Privacy requires inverting that default. This capsule covers:

Defining retention periods based on purpose.
Auto-purge mechanisms: implementation.
Right to erasure requests: how to handle them.
Backups, caches, logs: the hidden problem.
Trained models: the irreversible problem.
A retention table model: an applied example.

Defining retention periods

The principle

GDPR Art 5(1)(e): data kept for no longer than necessary for the purpose.

"No longer than necessary" requires:

Defining the purpose.
Defining the minimum period necessary for the purpose.
Implementing deletion at the end of the period.

For every type of data, you have to have an answer.

Retention table example

| Data type             | Purpose                       | Retention | Deletion   |
|----------------------|-------------------------------|-----------|------------|
| User account info     | Service operation             | While active + 90 days | Auto-purge after 90 days inactive |
| Transaction history   | Legal, tax records            | 7 years   | Auto-purge year 8 |
| Conversations         | Service operation             | 30 days   | Auto-purge daily job |
| Conversations (training) | Improve AI model           | Until consent revoked | Triggered on revoke |
| Logs (operational)    | Debugging, security           | 90 days   | Auto-rotate |
| Logs (audit)          | Compliance                    | 6 years   | Per regulation |
| Backups               | Disaster recovery             | 90 days rolling | Auto-overwrite |
| Cached data           | Performance                   | 24 hours  | TTL-based |
| Model artifacts       | Versioning, rollback          | 1 year    | Auto-purge old versions |

Each entry has a clear purpose + period + deletion mechanism.

Why "until consent revoked" requires immediate action

For training data with consent revocation:

The data could be used for the next training round (within days/weeks).
Auto-trigger the removal flag immediately.
The retraining schedule excludes flagged users.

def revoke_training_consent(user_id):
    # Immediate: mark for exclusion
    db.execute("UPDATE users SET training_excluded = TRUE WHERE id = ?", user_id)
    
    # Immediate: remove from current training queue
    training_queue.exclude(user_id)
    
    # Immediate: remove from RAG/retrieval store
    rag_store.delete_by_user(user_id)
    
    # Document
    audit_log.record(f"User {user_id} training consent revoked at {now()}")

Auto-purge mechanisms

Manual cleanup never happens. Automation, or it isn't real.

Approach 1: TTL-based in the database

A PostgreSQL example with auto-cleanup:

-- Table with retention policy
CREATE TABLE conversations (
    id SERIAL PRIMARY KEY,
    user_id INT,
    content TEXT,
    created_at TIMESTAMP DEFAULT NOW(),
    expires_at TIMESTAMP DEFAULT NOW() + INTERVAL '30 days'
);

-- Index for fast cleanup queries
CREATE INDEX idx_conversations_expires ON conversations(expires_at);

-- Daily cleanup job (cron)
DELETE FROM conversations WHERE expires_at < NOW();

Approach 2: Redis with natural TTL

# Set conversation cache with TTL
redis.setex(
    f"conv:{user_id}:{session_id}",
    timedelta(hours=24),  # Auto-expire
    conversation_json,
)

Redis handles expiration automatically. No cleanup job needed.

Approach 3: Scheduled background jobs

# Daily cron task
@scheduled(cron="0 2 * * *")  # 2 AM daily
def purge_old_data():
    cutoff = datetime.now() - timedelta(days=30)
    
    deleted = db.execute(
        "DELETE FROM conversations WHERE created_at < ?",
        cutoff
    )
    
    logger.info(f"Purged {deleted.rowcount} old conversations")

Approach 4: S3 lifecycle policies

# AWS S3 lifecycle.yaml
Rules:
  - Id: "expire-old-logs"
    Status: Enabled
    Filter:
      Prefix: "logs/"
    Transitions:
      - Days: 30
        StorageClass: GLACIER
    Expiration:
      Days: 90  # Delete after 90 days

Cloud storage tools have native lifecycle management. Use it.

Verifying the purge actually happens

Don't trust that the automation works. Verify:

def audit_purge_mechanism():
    # Sample query: there should be no records older than retention
    old_count = db.execute(
        "SELECT COUNT(*) FROM conversations WHERE created_at < ?",
        datetime.now() - timedelta(days=31)  # 1 day grace
    ).scalar()
    
    if old_count > 0:
        alert(f"Retention violation: {old_count} records past retention!")

Run it weekly. Alert on an anomaly.

Right to Erasure (Right to be Forgotten)

GDPR Art 17 grants users the right to request deletion. The process must be:

Available: an easy way for users to request it.
Verifiable: confirm the requester is the data subject.
Comprehensive: delete from all systems.
Timely: respond within 30 days (GDPR's default requirement).
Documented: a paper trail of what was deleted.

Implementation

class ErasureRequestHandler:
    def handle_request(self, user_id, request_id):
        """
        Process a right-to-erasure request.
        """
        # 1. Verify requestor is data subject
        if not self.verify_identity(user_id, request_id):
            raise UnauthorizedError()
        
        # 2. Document the request
        self.audit_log.record({
            'request_id': request_id,
            'user_id': user_id,
            'received_at': now(),
            'type': 'erasure',
        })
        
        # 3. Delete from primary stores
        deleted_records = self.delete_from_primary(user_id)
        
        # 4. Delete from caches
        cache_deletions = self.delete_from_caches(user_id)
        
        # 5. Mark for exclusion in training/derivatives
        self.exclude_from_training(user_id)
        
        # 6. Trigger backup cleanup (next backup will not include)
        self.flag_for_backup_cleanup(user_id)
        
        # 7. Document what was done
        self.audit_log.record({
            'request_id': request_id,
            'completed_at': now(),
            'records_deleted': deleted_records,
            'cache_deletions': cache_deletions,
            'training_excluded': True,
            'limitations': self.document_limitations(),  # What can't be removed
        })
        
        # 8. Notify user
        self.notify_user(user_id, "Your data has been deleted.")
    
    def document_limitations(self):
        """What we couldn't fully delete."""
        return {
            'trained_models': "May contain statistical traces; not individually identifiable",
            'old_backups': "Will be overwritten in 90 days per backup policy",
            'audit_logs': "Retained for legal compliance (6 years)",
        }

Limitations to document

Be honest about what can't be deleted:

Trained models: as discussed in capsule 04, they contain statistical traces. Document this.
Audit logs: legally required to retain (compliance). Don't delete them; explain.
Backups: will be overwritten according to the backup policy. Document the expected timeline.
External processors: if the data was shared with third parties (OpenAI, AWS), you may need to coordinate deletion. Document the chain.
Anonymized aggregates: typically excluded from the "personal data" definition. No need to delete.

Backups: the hidden problem

Backups frequently contain copies of data after primary deletion. Without explicit handling, you have a privacy violation.

The problem

Day 1: user María's account is created. The backup runs, includes María. Day 30: María's account is deleted from the primary store. Day 31: the backup still has María (1 month old). Day 60: the backup still has María (2 months old). Day 90: the backup rotates out, María's data is finally gone.

Between day 31 and 90, your data state contradicts the user's deletion request.

Solutions

Option A: short backup retention

If backups rotate every 30 days, the data is gone within 30 days max post-deletion. Acceptable for most.

Option B: incremental deletion

Apply the deletion to each backup as part of the erasure process. Technically complex.

Option C: encrypted backups + key deletion

Store backups encrypted. To "delete," destroy the encryption key. Without the key, the backup is unreadable.

def setup_backup_with_per_user_keys():
    # Each user has individual encryption key
    user_key = generate_key(user_id)
    
    # Store backup encrypted with user key
    encrypted_backup = encrypt(user_data, user_key)
    
    # Key stored separately
    keystore.store(user_id, user_key)
    
    # On erasure: just delete the key
    # Backup remains but is unreadable

def handle_erasure_with_backup_keys(user_id):
    # Delete user's encryption key
    keystore.delete(user_id)
    
    # All backups encrypted with that key are now unreadable = effectively deleted

Sophisticated, but it solves the problem mathematically.

Option D: document the limitation

The most practical: document the backup policy in the privacy notice. "Your data will be deleted from primary stores immediately, and from backups within 90 days as backup rotation occurs." Most regulators accept this as reasonable.

Logs: persistent privacy debt

Logs are a notorious privacy concern:

Often retained longer than the primary data.
Frequently contain raw PII (full prompts, full responses).
Multiple systems (app logs, API gateway, monitoring, analytics) → many places.
Long-term storage is often skipped in privacy reviews.

Logging policies

Levels of log content:

# ❌ Verbose, PII-heavy
logger.info(f"User {user.email} from {user.address} requested {full_query}")

# 🟡 Reduced PII
logger.info(f"User {user.id} (region: {user.region}) made request")

# ✅ Minimal, anonymized
logger.info(f"Request from anon-{hash(user.id)[:8]} processed in {duration}ms")

Default to minimal. Add detail only when provably needed for debugging/security.

Log retention

# logs.yaml
operational_logs:
  retention: 30 days
  pii_redaction: true
  
debug_logs:
  retention: 7 days
  pii_redaction: true
  storage: encrypted
  
audit_logs:
  retention: 6 years  # Compliance
  pii_redaction: false  # Required for compliance
  storage: immutable
  access: restricted

Different log types have different policies. Operational ≠ audit.

PII redaction in logs

Implement it at the logging layer:

class PIIRedactingFormatter(logging.Formatter):
    PII_PATTERNS = [
        (r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[EMAIL]'),
        (r'\b\d{3}-\d{2}-\d{4}\b', '[SSN]'),
        (r'\b\d{16}\b', '[CREDIT_CARD]'),
        (r'\b\(\d{3}\)\s?\d{3}-?\d{4}\b', '[PHONE]'),
    ]
    
    def format(self, record):
        msg = super().format(record)
        for pattern, replacement in self.PII_PATTERNS:
            msg = re.sub(pattern, replacement, msg)
        return msg

# Apply globally
logging.getLogger().handlers[0].setFormatter(PIIRedactingFormatter())

Even if a developer accidentally logs PII, it gets redacted at the output.

Trained models: the fundamental problem

As discussed, data inside models is hard to remove. The realistic strategy:

Layer 1: minimize PII in the training data from the start

Don't put it in. Capsule 02 (data minimization) and capsule 03 (anonymization) are the primary defenses.

Layer 2: track which data was used in each model version

CREATE TABLE training_runs (
    model_version VARCHAR(50),
    user_ids_included INT[],
    training_date TIMESTAMP,
    consent_version VARCHAR(20)
);

When a user revokes, you know which model versions contain their data.

Layer 3: retrain periodically, excluding revocations

def schedule_periodic_retrain():
    """Monthly retrain excluding all revoked consents."""
    excluded_users = get_users_with_revoked_consent()
    
    training_data = get_training_data().filter(
        lambda x: x.user_id not in excluded_users
    )
    
    new_model = train(training_data)
    deploy(new_model)

A monthly cadence means revoked data is excluded from production within ~30 days.

Layer 4: machine unlearning research

An active area. SISA, exact unlearning, etc. Watch for production-ready techniques.

Layer 5: be honest with users

The privacy notice:

"When you revoke training consent, we immediately exclude your data from future model training and remove it from our retrieval databases. Existing trained models may contain statistical traces of your data, which cannot be fully removed without complete model retraining. We retrain periodically; your data will be fully excluded within [N] months."

Honest > perfect. Most regulators accept reasonable effort + transparency.

Common traps

1. "We'll delete it manually when needed"

Manual processes don't happen consistently. Automation, or no privacy.

2. Forgetting caches

Redis cache, CDN cache, ML feature store — they all have copies. They all need lifecycle management.

3. Infinite backups

Many companies have backups going back years "for safety." Each one is accumulating privacy debt.

4. Retention contracts with third parties

If you pass data to OpenAI, AWS, etc., their retention policies apply too. Verify alignment.

5. Audit logs with no policy

Audit logs frequently get exempted from retention rules ("we need them"). But they contain PII. Have an explicit policy + access controls.

Self-check

1. Why is automation essential for retention?

Manual deletion processes:

Don't happen consistently: developers forget, priorities shift.
Aren't auditable: no record of what was deleted when.
Don't scale: manually reviewing each user's data is impossible at scale.
Are vulnerable to human error: deleting the wrong data, missing some data.

Automated deletion:

Consistent: runs the same way every time.
Auditable: logs of each deletion.
Scales: handles millions of records the same as 100.
Verifiable: you can audit "are there records past retention?" with one query.

Implementation: TTL in the database/Redis, scheduled cleanup jobs, S3 lifecycle policies, event-driven triggers on consent revocation.

If you can't automate it, the retention policy isn't enforceable.

2. How do you handle right-to-erasure when the data is in old backups?

Three approaches:

Short backup retention: if backups rotate every 30 days, the data disappears within 30 days. Document this in the privacy notice.
Encrypted backups + key deletion: use per-user encryption keys. To "delete," destroy the key. The backup remains but is unreadable. Mathematically equivalent to deletion.
Document the limitation: the privacy notice explicitly states "Your data will be deleted from primary stores immediately and from backups within [X] days as backup rotation occurs." Most regulators accept this as reasonable effort.

Option 3 is the most practical. Combined with option 1 (reasonable backup retention), it satisfies regulator expectations.

What you CANNOT do:

Pretend the backups don't have the data.
Indefinitely retain backups while claiming deletion.
Charge for backup deletion or refuse to address it.

Honest disclosure + a reasonable timeline = compliance.

3. Why are logs frequently the biggest privacy debt?

Reasons:

Often longer retention than the primary data: months or years.
Multiple systems: app logs, API gateway, monitoring, analytics — each one is a copy.
They contain PII by default: developers print(user) and the full object goes into the logs.
Frequently exempted from privacy reviews: "they're just logs."
Hard to audit: log volumes are huge, hard to find a specific user's data.
Required for ops: you can't just delete them (debugging needs them).

Solutions:

PII redaction at the logging layer: even accidental PII gets redacted automatically.
Layered retention: operational (30 days) ≠ audit (6 years).
Audit log access controls: not everyone gets to read them.
Log volume monitoring: anomalies get detected.
Periodic audit of log content: a random sample, checked for PII.

Treat logs as data with the same privacy obligations as primary data, with retention adjusted to the legitimate purpose.

4. What's the realistic strategy for data in trained models?

Layered defense:

Layer 1 - prevent: minimize PII in the training data from the start (capsules 02-03).
Layer 2 - track: document which data was used in each model version. You know when a revocation affects what.
Layer 3 - retrain periodically: a monthly retrain excluding revoked users. That means a ~30-day delay before they're fully excluded from production.
Layer 4 - machine unlearning: emerging research. Watch for production-ready techniques.
Layer 5 - honest disclosure: the privacy notice explicitly states the limitations. Users know what to expect.

What this combination achieves:

Most data is minimized from the start.
Tracking lets you know which versions are affected.
Periodic retrains ensure the most recent production model excludes revoked users.
Honest disclosure manages expectations + provides legal cover.

This is reasonable effort. Most regulators don't require perfect deletion when it's technically infeasible — they require demonstrable effort + transparency.

Companies that pretend they achieve perfect deletion, or ignore the issue, face regulatory action. Companies that document limitations + apply best effort are generally OK.

Summary and next step

Define retention for each data type with a clear purpose + period + deletion mechanism.
Automate deletion: TTL, scheduled jobs, lifecycle policies. Manual = no privacy.
Right to erasure requires a comprehensive process: primary stores, caches, backups, training exclusion, documentation.
Backups are hidden privacy debt. Short retention, or encryption with key deletion.
Logs are persistent. PII redaction + layered retention + audit.
Trained models can't be fully erased. Reasonable effort + transparency = compliance.

Checkpoint: you should be able to design a retention table for a real system and identify the auto-purge mechanisms.

Bridge to the next capsule: capsule 06 covers AI-specific privacy risks — model inversion, membership inference, prompt leakage, memorization. You'll understand how these attacks work and which mitigations apply.

Resources

GDPR Art 5(1)(e), 17 — retention, right to erasure.
AWS S3 Lifecycle Policies — implementation.
Encryption with key deletion — papers on key-based deletion.

Next: 06-ai-specific-privacy-risks.md — Model inversion, membership inference, prompt leakage.

Capsule 05 of 08 — Module 3 — AI Ethics & Compliance Guide

← Previous4. Consent in AI Systems Next →6. AI-Specific Privacy Risks