Módulo 3: Privacy and Data Protection Fundamentals

5. Retention y Data Lifecycle

Descripción de la cápsula

Recolectar data es solo el inicio. El resto del lifecycle — how long, where stored, when purged, how to delete on request — define gran parte del privacy posture.

Default tendencies en software:

Data persiste forever unless explicitly deleted.
Backups se acumulan.
Logs nunca se purgan.
Caches mantienen copies.
"Por si acaso" wins en cada decision.

Privacy requires invertir ese default. Esta cápsula cubre:

Definir retention periods según purpose.
Auto-purge mechanisms: implementación.
Right to erasure requests: handling.
Backups, caches, logs: el problema oculto.
Models entrenados: el problema irreversible.
Tabla de retention model: ejemplo aplicado.

Definir retention periods

El principio

GDPR Art 5(1)(e): data kept for no longer than necessary for the purpose.

"No longer than necessary" requires:

Definir el purpose.
Definir el período mínimo necesario para purpose.
Implement deletion al final del período.

Por cada type de data, tenés que tener answer.

Retention table example

| Data type             | Purpose                       | Retention | Deletion   |
|----------------------|-------------------------------|-----------|------------|
| User account info     | Service operation             | While active + 90 days | Auto-purge after 90 days inactive |
| Transaction history   | Legal, tax records            | 7 years   | Auto-purge year 8 |
| Conversations         | Service operation             | 30 days   | Auto-purge daily job |
| Conversations (training) | Improve AI model           | Until consent revoked | Triggered on revoke |
| Logs (operational)    | Debugging, security           | 90 days   | Auto-rotate |
| Logs (audit)          | Compliance                    | 6 years   | Per regulation |
| Backups               | Disaster recovery             | 90 days rolling | Auto-overwrite |
| Cached data           | Performance                   | 24 hours  | TTL-based |
| Model artifacts       | Versioning, rollback          | 1 year    | Auto-purge old versions |

Each entry tiene clear purpose + period + deletion mechanism.

Why "until consent revoked" requires immediate action

For training data with consent revocation:

Data could be used for next training round (within days/weeks).
Auto-trigger removal flag immediately.
Re-training schedule excludes flagged users.

def revoke_training_consent(user_id):
    # Immediate: mark for exclusion
    db.execute("UPDATE users SET training_excluded = TRUE WHERE id = ?", user_id)
    
    # Immediate: remove from current training queue
    training_queue.exclude(user_id)
    
    # Immediate: remove from RAG/retrieval store
    rag_store.delete_by_user(user_id)
    
    # Document
    audit_log.record(f"User {user_id} training consent revoked at {now()}")

Auto-purge mechanisms

Manual cleanup never happens. Automation o no es real.

Approach 1: TTL-based en database

PostgreSQL example with auto-cleanup:

-- Table with retention policy
CREATE TABLE conversations (
    id SERIAL PRIMARY KEY,
    user_id INT,
    content TEXT,
    created_at TIMESTAMP DEFAULT NOW(),
    expires_at TIMESTAMP DEFAULT NOW() + INTERVAL '30 days'
);

-- Index for fast cleanup queries
CREATE INDEX idx_conversations_expires ON conversations(expires_at);

-- Daily cleanup job (cron)
DELETE FROM conversations WHERE expires_at < NOW();

Approach 2: Redis con TTL natural

# Set conversation cache with TTL
redis.setex(
    f"conv:{user_id}:{session_id}",
    timedelta(hours=24),  # Auto-expire
    conversation_json,
)

Redis handles expiration automatically. No cleanup job needed.

Approach 3: Scheduled background jobs

# Daily cron task
@scheduled(cron="0 2 * * *")  # 2 AM daily
def purge_old_data():
    cutoff = datetime.now() - timedelta(days=30)
    
    deleted = db.execute(
        "DELETE FROM conversations WHERE created_at < ?",
        cutoff
    )
    
    logger.info(f"Purged {deleted.rowcount} old conversations")

Approach 4: S3 lifecycle policies

# AWS S3 lifecycle.yaml
Rules:
  - Id: "expire-old-logs"
    Status: Enabled
    Filter:
      Prefix: "logs/"
    Transitions:
      - Days: 30
        StorageClass: GLACIER
    Expiration:
      Days: 90  # Delete after 90 days

Cloud storage tools tienen native lifecycle management. Use them.

Verifying purge actually happens

Don't trust that automation works. Verify:

def audit_purge_mechanism():
    # Sample query: should be no records older than retention
    old_count = db.execute(
        "SELECT COUNT(*) FROM conversations WHERE created_at < ?",
        datetime.now() - timedelta(days=31)  # 1 day grace
    ).scalar()
    
    if old_count > 0:
        alert(f"Retention violation: {old_count} records past retention!")

Run weekly. Alert if anomaly.

Right to Erasure (Right to be Forgotten)

GDPR Art 17 grants users right to request deletion. Process must be:

Available: easy way for users to request.
Verifiable: confirm requestor is data subject.
Comprehensive: delete from all systems.
Timely: respond within 30 days (default GDPR requirement).
Documented: paper trail of what was deleted.

Implementation

class ErasureRequestHandler:
    def handle_request(self, user_id, request_id):
        """
        Process a right-to-erasure request.
        """
        # 1. Verify requestor is data subject
        if not self.verify_identity(user_id, request_id):
            raise UnauthorizedError()
        
        # 2. Document the request
        self.audit_log.record({
            'request_id': request_id,
            'user_id': user_id,
            'received_at': now(),
            'type': 'erasure',
        })
        
        # 3. Delete from primary stores
        deleted_records = self.delete_from_primary(user_id)
        
        # 4. Delete from caches
        cache_deletions = self.delete_from_caches(user_id)
        
        # 5. Mark for exclusion in training/derivatives
        self.exclude_from_training(user_id)
        
        # 6. Trigger backup cleanup (next backup will not include)
        self.flag_for_backup_cleanup(user_id)
        
        # 7. Document what was done
        self.audit_log.record({
            'request_id': request_id,
            'completed_at': now(),
            'records_deleted': deleted_records,
            'cache_deletions': cache_deletions,
            'training_excluded': True,
            'limitations': self.document_limitations(),  # What can't be removed
        })
        
        # 8. Notify user
        self.notify_user(user_id, "Your data has been deleted.")
    
    def document_limitations(self):
        """What we couldn't fully delete."""
        return {
            'trained_models': "May contain statistical traces; not individually identifiable",
            'old_backups': "Will be overwritten in 90 days per backup policy",
            'audit_logs': "Retained for legal compliance (6 years)",
        }

Limitations to document

Honest about what can't be deleted:

Trained models: as discussed in cápsula 04, contain statistical traces. Document this.
Audit logs: legally required to retain (compliance). Don't delete; explain.
Backups: will be overwritten according to backup policy. Document expected timeline.
External processors: if data shared with third parties (OpenAI, AWS), may need to coordinate deletion. Document chain.
Anonymized aggregates: typically excluded from "personal data" definition. Don't need to delete.

Backups: el problema oculto

Backups frequently contain copies of data after primary deletion. Without explicit handling, you have privacy violation.

El problema

Day 1: User María's account created. Backup runs, includes María. Day 30: María's account deleted from primary. Day 31: Backup still has María (1 month old). Day 60: Backup still has María (2 months old). Day 90: Backup rotates out, María's data finally gone.

Between day 31 and 90, your data state contradicts user's deletion request.

Solutions

Option A: short backup retention

If backups rotate every 30 days, data is gone in 30 days max post-deletion. Acceptable for most.

Option B: incremental deletion

Apply deletion to each backup as part of erasure process. Technically complex.

Option C: encrypted backups + key deletion

Store backups encrypted. To "delete", destroy the encryption key. Without key, backup is unreadable.

def setup_backup_with_per_user_keys():
    # Each user has individual encryption key
    user_key = generate_key(user_id)
    
    # Store backup encrypted with user key
    encrypted_backup = encrypt(user_data, user_key)
    
    # Key stored separately
    keystore.store(user_id, user_key)
    
    # On erasure: just delete the key
    # Backup remains but is unreadable

def handle_erasure_with_backup_keys(user_id):
    # Delete user's encryption key
    keystore.delete(user_id)
    
    # All backup encrypted with that key are now unreadable = effectively deleted

Sophisticated but solves the problem mathematically.

Option D: document the limitation

Most practical: document backup policy in privacy notice. "Your data will be deleted from primary stores immediately, and from backups within 90 days as backup rotation occurs." Most regulators accept this as reasonable.

Logs: persistent privacy debt

Logs are notorious privacy concern:

Often retained longer than primary data.
Frequently contain raw PII (full prompts, full responses).
Multiple systems (app logs, API gateway, monitoring, analytics) → many places.
Long-term storage often skipped in privacy reviews.

Logging policies

Levels of log content:

# ❌ Verbose, PII-heavy
logger.info(f"User {user.email} from {user.address} requested {full_query}")

# 🟡 Reduced PII
logger.info(f"User {user.id} (region: {user.region}) made request")

# ✅ Minimal, anonymized
logger.info(f"Request from anon-{hash(user.id)[:8]} processed in {duration}ms")

Default to minimal. Add detail only when provably needed for debugging/security.

Log retention

# logs.yaml
operational_logs:
  retention: 30 days
  pii_redaction: true
  
debug_logs:
  retention: 7 days
  pii_redaction: true
  storage: encrypted
  
audit_logs:
  retention: 6 years  # Compliance
  pii_redaction: false  # Required for compliance
  storage: immutable
  access: restricted

Different log types have different policies. Operational ≠ audit.

PII redaction in logs

Implement at logging layer:

class PIIRedactingFormatter(logging.Formatter):
    PII_PATTERNS = [
        (r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[EMAIL]'),
        (r'\b\d{3}-\d{2}-\d{4}\b', '[SSN]'),
        (r'\b\d{16}\b', '[CREDIT_CARD]'),
        (r'\b\(\d{3}\)\s?\d{3}-?\d{4}\b', '[PHONE]'),
    ]
    
    def format(self, record):
        msg = super().format(record)
        for pattern, replacement in self.PII_PATTERNS:
            msg = re.sub(pattern, replacement, msg)
        return msg

# Apply globally
logging.getLogger().handlers[0].setFormatter(PIIRedactingFormatter())

Even if developer accidentally logs PII, it gets redacted at output.

Models entrenados: el problema fundamental

Como discussed, datos en modelos son hard to remove. Realistic strategy:

Layer 1: minimize PII en training data desde el inicio

Don't put it in. Cápsula 02 (data minimization) y cápsula 03 (anonymization) son las defenses primarias.

Layer 2: track which data se usó en cada model version

CREATE TABLE training_runs (
    model_version VARCHAR(50),
    user_ids_included INT[],
    training_date TIMESTAMP,
    consent_version VARCHAR(20)
);

Cuando user revokes, sabés cuáles model versions contain their data.

Layer 3: re-train periodically excluding revocations

def schedule_periodic_retrain():
    """Monthly retrain excluding all revoked consents."""
    excluded_users = get_users_with_revoked_consent()
    
    training_data = get_training_data().filter(
        lambda x: x.user_id not in excluded_users
    )
    
    new_model = train(training_data)
    deploy(new_model)

Monthly cadence means revoked data is excluded from production within ~30 days.

Layer 4: machine unlearning research

Active area. SISA, exact unlearning, etc. Watch for production-ready techniques.

Layer 5: be honest with users

Privacy notice:

"When you revoke training consent, we immediately exclude your data from future model training and remove it from our retrieval databases. Existing trained models may contain statistical traces of your data, which cannot be fully removed without complete model retraining. We retrain periodically; your data will be fully excluded within [N] months."

Honest > perfect. Most regulators accept reasonable effort + transparency.

Trampas comunes

1. "We'll delete it manually when needed"

Manual processes don't happen consistently. Automation o no privacy.

2. Olvidar caches

Redis cache, CDN cache, ML feature store — all have copies. All need lifecycle management.

3. Backups infinitos

Many companies have backups going back years "for safety". Each is privacy debt accumulating.

4. Retention contracts con third parties

Si pasás data a OpenAI, AWS, etc., su retention policies aplican también. Verify alignment.

5. Audit logs sin policy

Audit logs frequently exempted from retention rules ("we need them"). But they contain PII. Have explicit policy + access controls.

Auto-verificación

1. ¿Por qué automation es esencial para retention?

Manual deletion processes:

Don't happen consistently: developers forget, priorities shift.
Not auditable: no record of what was deleted when.
Don't scale: manual review of each user's data is impossible at scale.
Vulnerable to human error: deleting wrong data, missing some data.

Automated deletion:

Consistent: runs same way every time.
Auditable: logs of each deletion.
Scales: handles millions of records same as 100.
Verifiable: can audit "are there records past retention?" with one query.

Implementation: TTL en database/Redis, scheduled cleanup jobs, S3 lifecycle policies, event-driven triggers on consent revocation.

If you can't automate, the retention policy isn't enforceable.

2. ¿Cómo manejás right-to-erasure cuando data está en backups antiguos?

Tres approaches:

Short backup retention: si backups rotate cada 30 días, data desaparece en 30 días. Document esto en privacy notice.
Encrypted backups + key deletion: usar per-user encryption keys. Para "delete", destroy the key. Backup remains but is unreadable. Mathematically equivalent a deletion.
Document the limitation: privacy notice explicitly states "Your data will be deleted from primary stores immediately and from backups within [X] days as backup rotation occurs." Most regulators accept this as reasonable effort.

Option 3 is most practical. Combined with option 1 (reasonable backup retention), satisfies regulator expectations.

What you CANNOT do:

Pretend backups don't have the data.
Indefinitely retain backups while claiming deletion.
Charge for backup deletion or refuse to address.

Honest disclosure + reasonable timeline = compliance.

3. ¿Por qué logs son frequently el biggest privacy debt?

Razones:

Often longer retention than primary data: months or years.
Multiple systems: app logs, API gateway, monitoring, analytics — each is a copy.
Default contains PII: developers print(user) and full object goes to logs.
Frequently exempted from privacy reviews: "they're just logs".
Hard to audit: log volumes are huge, hard to find specific user's data.
Required for ops: can't just delete (debugging needs).

Solutions:

PII redaction at logging layer: even accidental PII gets redacted automatically.
Layered retention: operational (30 days) ≠ audit (6 years).
Audit log access controls: not everyone reads them.
Log volume monitoring: anomalies detected.
Periodic audit of log content: random sample, check for PII.

Treat logs as data with same privacy obligations as primary data, with adjusted retention based on legitimate purpose.

4. ¿Cuál es la realistic strategy para data en modelos entrenados?

Layer defense:

Layer 1 - prevent: minimize PII en training data desde inicio (cápsulas 02-03).
Layer 2 - track: documentar qué data fue used en cada model version. Sabés cuándo revocation affects qué.
Layer 3 - retrain periodically: monthly retrain excluyendo revoked users. Means ~30 day delay before fully excluded de production.
Layer 4 - machine unlearning: emerging research. Watch para production-ready techniques.
Layer 5 - honest disclosure: privacy notice explicitly states limitations. Users know what to expect.

What this combination achieves:

Most data minimized desde start.
Tracking enables knowing which versions are affected.
Periodic retrains ensure most recent production excludes revoked users.
Honest disclosure manages expectations + provides legal cover.

This is reasonable effort. Most regulators don't require perfect deletion when technically infeasible — they require demonstrable effort + transparency.

Companies that pretend perfect deletion or ignore the issue face regulatory action. Companies that document limitations + apply best effort generally OK.

Resumen y siguiente paso

Definir retention for each data type with clear purpose + period + deletion mechanism.
Automate deletion: TTL, scheduled jobs, lifecycle policies. Manual = no privacy.
Right to erasure requires comprehensive process: primary stores, caches, backups, training exclusion, documentation.
Backups are hidden privacy debt. Short retention or encryption with key deletion.
Logs are persistent. PII redaction + layered retention + audit.
Trained models can't be fully erased. Reasonable effort + transparency = compliance.

Checkpoint: deberías poder design retention table for a real system y identify auto-purge mechanisms.

Puente a la siguiente cápsula: la cápsula 06 cubre AI-specific privacy risks — model inversion, membership inference, prompt leakage, memorization. Vas a entender cómo estos attacks funcionan y qué mitigations aplican.

Recursos

GDPR Art 5(1)(e), 17 — retention, right to erasure.
AWS S3 Lifecycle Policies — implementation.
Encryption with key deletion (boem) — papers on key-based deletion.

Siguiente: 06-ai-specific-privacy-risks.md — Model inversion, membership inference, prompt leakage.

Cápsula 05 de 08 — Módulo 3 — AI Ethics & Compliance Guide

← Anterior4. Consent en Sistemas AI Siguiente →6. AI-Specific Privacy Risks