4. High-Risk Obligations: Las 6 Requisitos

Descripción de la cápsula

Si tu sistema es high-risk, tenés 6 obligations principales bajo EU AI Act:

1. Risk management system (Art. 9).
2. Data y data governance (Art. 10).
3. Technical documentation (Art. 11).
4. Record-keeping y logging (Art. 12).
5. Transparency y information to deployers (Art. 13).
6. Human oversight (Art. 14).

Plus crítico: accuracy, robustness, and cybersecurity (Art. 15) y conformity assessment + registration (Art. 43, 49, 60).

Esta cápsula traduce cada obligation de legal language a engineering action específica. Al terminar, vas a tener un checklist concreto de qué hacer para cumplir cada una.

---

Obligation 1: Risk Management System (Art. 9)

Lo que pide el Act

"A risk management system shall be established, implemented, documented and maintained..."

Traducción técnica

Continuous process de:

1. Identify risks del sistema.
2. Estimate y evaluate known/foreseeable risks.
3. Adopt mitigation measures.
4. Test mitigations effectiveness.
5. Update continuously.

Engineering action

## Risk Management System (RMS)

### 1. Risk Register
| Risk | Likelihood | Severity | Mitigation | Status | Owner |
|------|-----------|----------|------------|--------|-------|
| Bias por género | High | High | Bias testing en CI | In progress | ML eng |
| Privacy leak en logs | Medium | High | PII redaction | Implemented | DevOps |
| Hallucination | Medium | Medium | Output filter | Implemented | ML eng |
| ... | | | | | |

### 2. Mitigation testing
- Quarterly: re-run bias tests.
- Continuously: monitoring privacy metrics.
- Per-release: regression tests.

### 3. Updates
- After incidents.
- After feature changes.
- After regulatory updates.

Conexión con M1: Ethics Impact Analysis es input directo al Risk Register.

---

Obligation 2: Data y Data Governance (Art. 10)

Lo que pide el Act

Training, validation, testing data sets debe cumplir requisitos de:

• Relevance y representatividad para purpose.
• Free of errors as far as possible.
• Statistical properties appropriate (distribution, etc.).
• Bias examination y mitigation.
• Documentation de data sources.

Engineering action

## Data Governance

### 1. Data Sources Documentation
- Source 1: Internal database. Period: 2020-2025.
- Source 2: Public dataset X. License: CC-BY.
- Source 3: Synthetic data generated por process Y.

### 2. Data Quality
- Completeness: 95% (5% have missing fields, dropped).
- Accuracy: validated against ground truth where available.
- Freshness: refreshed quarterly.

### 3. Bias Examination
- Demographic distribution analyzed (M2 Bias Audit).
- Identified imbalances + mitigations applied.
- Documented in model card.

### 4. Statistical Validation
- Train/val/test split: 70/15/15.
- No data leakage between splits (verified).
- Distribution similar between splits (verified).

Conexión con M2: Bias Audit Toolkit produces documentation directly satisfying esta obligation.

Conexión con M3: Privacy Assessment + minimization aplica acá también.

---

Obligation 3: Technical Documentation (Art. 11)

Lo que pide el Act

Documentation que demonstra compliance con todos los requirements. Debe incluir (Annex IV):

• General description del sistema.
• Detailed description del design y development process.
• Detailed information sobre monitoring, functioning, control.
• Risk management documentation.
• Data documentation.
• Performance metrics.
• Cybersecurity measures.
• Etc.

Engineering action

Documentation se mantiene como artefacto vivo. Format:

## Technical Documentation v1.0

### 1. System Description
- Purpose: medical triage classification
- Architecture: LLM + RAG + classifier
- Inputs: user symptoms text
- Outputs: triage urgency category

### 2. Design Decisions
- LLM choice: GPT-4 (enterprise)
- Decision rationale: trade-off accuracy vs cost
- Alternatives considered: open-source models

### 3. Training Data
- Source documents (linked)
- Quality measures (linked to bias audit)

### 4. Performance
- Accuracy: 88% on test set
- Per-subgroup performance (linked)
- Limitations: known issues with rare conditions

### 5. Risk Mitigations
- (Linked to Risk Management Doc)

### 6. Human Oversight
- (Linked to Human Oversight Doc)

### 7. Monitoring
- Production metrics tracked
- Alerts configured
- Re-evaluation schedule

### 8. Cybersecurity
- (Linked to Security Doc)

Best practice: maintain en repository (Git), versionado, accessible al regulator if requested.

---

Obligation 4: Record-Keeping y Logging (Art. 12)

Lo que pide el Act

System debe automatically generate logs:

• Period of each use.
• Reference of inputs.
• Persons involved (humans-in-the-loop).
• Para enabling traceability y enforcement.

Engineering action

# Required logging para high-risk AI
@app.post("/predict")
async def predict(request):
    log_record = {
        'timestamp': now(),
        'user_id': hash(request.user_id),  # Pseudonymized
        'inputs_hash': hash(request.inputs),  # Don't log PII raw
        'model_version': model.version,
        'output_summary': summarize_output(prediction),
        'human_reviewer': request.assigned_reviewer if applicable else None,
        'override_flag': prediction.was_overridden,
    }
    
    audit_logger.log(log_record)
    
    return prediction

Retention: typically 6+ years for compliance (varies by use case).

---

Obligation 5: Transparency y Information (Art. 13)

Lo que pide el Act

Sistema debe ser transparent enough that deployers can:

• Understand how it works.
• Interpret outputs correctly.
• Use it appropriately.

Provider must furnish:

• Instructions for use.
• Description of intended use.
• Limitations.
• Performance characteristics.
• Required human oversight.
• Maintenance requirements.

Engineering action

Model card + deployer guide + API documentation.

Ejemplo de model card (extract):

## Model Card: Medical Triage Classifier v2.1

### Intended Use
- Triage classification para non-emergency adult cases.
- NOT for: emergencies, pediatrics, mental health crises.

### Performance
- Accuracy on validation set: 88%.
- Reliability: highest for English, 85% Spanish, 80% French.
- Edge cases: poor performance on rare conditions (<5% prevalence).

### Required Human Oversight
- Healthcare professional must review urgent classifications.
- For non-urgent: can be acted upon directly by user.

### Deployment Requirements
- Must be integrated with patient communication system.
- Healthcare professional must be available within 1 hour for urgent cases.
- Must comply with HIPAA + local medical regulations.

### Limitations
- Cannot diagnose specific conditions.
- May misclassify edge cases.
- Bias mitigation: see bias audit (linked).

### Updates
- Re-trained quarterly.
- Major changes communicated to deployers.

---

Obligation 6: Human Oversight (Art. 14)

Lo que pide el Act

System debe permit effective human oversight:

• Understanding what the system is doing.
• Monitoring its operation.
• Interpreting outputs.
• Decideing to override or not use output.
• Stopping the system if needed.

Engineering action

Human oversight es decisión arquitectónica, no solo UI feature.

class HumanOversightAwareSystem:
    def __init__(self):
        self.review_queue = ReviewQueue()
        self.override_log = OverrideLog()
    
    async def make_prediction(self, input_data, threshold=0.7):
        # Generate prediction
        prediction = self.model.predict(input_data)
        confidence = prediction.confidence
        
        # Reject option: low confidence → human review
        if confidence < threshold:
            self.review_queue.add(input_data, prediction)
            return {
                'status': 'pending_human_review',
                'estimated_time': '2 hours',
            }
        
        # High confidence: still log for audit
        self.audit_logger.log_prediction(input_data, prediction)
        return prediction
    
    def human_override(self, prediction_id, new_decision, reason):
        """Allow humans to override AI decisions."""
        self.override_log.record({
            'prediction_id': prediction_id,
            'original': self.get_prediction(prediction_id),
            'overridden_to': new_decision,
            'reason': reason,
            'reviewer': current_user(),
            'timestamp': now(),
        })
        
        # Update output
        self.update_output(prediction_id, new_decision)
        
        # Feed back into improvements
        self.training_feedback.add(prediction_id, new_decision)

Levels of human oversight

EU AI Act distinguishes:

1. Human-in-the-loop: human approves each decision.
2. Human-on-the-loop: human monitors, intervenes when needed.
3. Human-in-command: human can stop the system.

Different use cases require different levels. High-stakes (medical) typically requires 1 or 2. Lower-stakes can be 3 only.

---

Adicional: Accuracy, Robustness, Cybersecurity (Art. 15)

Requirements

• Accuracy: appropriate level para intended purpose.
• Robustness: works correctly under varied conditions.
• Cybersecurity: protected against unauthorized changes.

Engineering action

## Accuracy
- Defined accuracy metrics for the use case.
- Threshold: at least 85% (justified for medical triage).
- Per-subgroup performance documented.

## Robustness
- Tested with adversarial inputs.
- Tested with distribution shift scenarios.
- Failover mechanism: if confidence < threshold, human review.

## Cybersecurity
- Model weights encrypted at rest.
- API authenticated y rate-limited.
- Adversarial robustness testing.
- Vulnerability scanning.
- Incident response plan.

---

Adicional: Conformity Assessment + Registration (Art. 43, 49, 60)

Conformity assessment

Before deploying high-risk AI:

• Self-assessment for some categories.
• Third-party assessment for others (notified body).
• Documentation of assessment.

Registration

• Register el sistema en EU database de high-risk AI.
• Update registration when system changes.
• Public-facing (some details).

---

Trampas comunes

1. Documentation sin substance

Documents que describen what the system does sin actually addressing requirements. Regulators read carefully — empty documentation invalida compliance claim.

2. Human oversight como UI checkbox

"Approve?" button que humans always click is not real oversight. Need genuine decision authority + override capability + actual usage tracking.

3. Risk management as one-time

RMS es continuous process. Document updates. Re-run risk assessments quarterly.

4. Skipping cybersecurity

Often deprioritized. But Art. 15 makes it mandatory. AI specific threats (prompt injection, model inversion) deben ser addressed.

5. Conformity assessment delayed

Some categories require third-party assessment which takes weeks/months. Don't wait until last minute.

---

Auto-verificación

1. ¿Por qué documentation sin substance es problematic?

Regulators are not impressed by volume of documentation — they evaluate whether it actually addresses requirements.

Bad documentation:

• "We tested for bias." (no methodology, no metrics, no results)
• "Risk management is in place." (no register, no review process)
• "Human oversight implemented." (no description of how)

Good documentation:

• "Bias tested using demographic parity (4/5 rule), equalized odds (5% threshold), counterfactual fairness. Results: 0.85, 4.2%, 0.05. See appendix B for methodology and full results."
• "Risk register maintained at [link]. Reviewed quarterly. Last review: [date]. Three risks identified, two mitigated, one in progress."
• "Human reviewer required for confidence < 0.7. Reviewer dashboard at [link]. Override functionality: see Section 4.3 of Operations Manual."

Specific, verifiable, traceable. That's what compliance documentation looks like.

2. ¿Cómo distinguís human oversight real de cosmetic?

Real human oversight:

• Humans actually review outputs (not just rubber-stamp).
• Humans have authority to override.
• Override decisions are tracked and affect future.
• Reviewers are trained for their role.
• Process is monitored for effectiveness.

Cosmetic oversight:

• Humans click "approve" on hundreds per hour without real review.
• Override available but rarely used (no time, no expertise).
• Override decisions disappear into void.
• Reviewers untrained.
• No metrics on oversight effectiveness.

Test:

• What % of decisions are overridden? (If <1%, may be cosmetic.)
• How long do reviewers spend per decision? (If <30 seconds for complex decisions, cosmetic.)
• Do reviewers have authority and ability to dissent?

Auditors check this. "We have a review step" is not enough — must be effective.

3. ¿Cuál es la diferencia entre human-in-the-loop y human-on-the-loop?

Human-in-the-loop:

• Human reviews every AI decision before action.
• Sequential: AI → human → action.
• Strongest oversight, slowest.
• Used for: high-stakes (medical, legal, criminal justice).

Human-on-the-loop:

• AI acts automatically; human monitors patterns.
• Parallel: AI → action + human can intervene if needed.
• Faster, less oversight.
• Used for: medium-stakes with clear failure modes.

Human-in-command:

• Human can stop system entirely if needed.
• Minimal oversight.
• Used for: low-stakes automation.

EU AI Act doesn't mandate specific level; depends on system risk + use case. Medical typically requires "in-the-loop". Customer service "on-the-loop". Recommendations "in-command".

Document choice + reasoning.

4. ¿Por qué Risk Management debe ser continuous, no one-time?

Sistemas evolucionan continuously:

• Data drift: production inputs differ from training data over time.
• Model updates: re-training introduces new behaviors.
• Use case shifts: systems used in unforeseen ways.
• New threats: adversarial techniques evolve.
• Regulatory updates: requirements change.

One-time risk assessment becomes outdated within months.

Continuous RMS:

• Quarterly review of risk register.
• Re-test mitigations after changes.
• Add new risks as they emerge.
• Update documentation continuously.
• Audit logs show review history.

Compliance evidence: regulator wants to see ongoing engagement, not single document from 2 years ago.

---

Resumen y siguiente paso

• 6 main obligations for high-risk AI: risk management, data governance, technical documentation, record-keeping, transparency, human oversight.
• Plus: accuracy/robustness/cybersecurity, conformity assessment, registration.
• Each requires engineering action, not just paperwork.
• Continuous processes: RMS, monitoring, updates.
• Conexión con previous modules: M1 risk analysis, M2 bias docs, M3 privacy assessment all feed into compliance documentation.

Checkpoint: deberías poder map cada obligation a a concrete engineering deliverable.

Puente a la siguiente cápsula: la cápsula 05 cubre Limited Risk — chatbots, deepfakes, transparency obligations. Mucho más sencillo que high-risk pero aún requires specific actions.

---

Recursos

1. EU AI Act Articles 9-15 — texto.
2. Annex IV — Technical documentation — full requirements.

---

Siguiente: 05-limited-risk-transparency.md — Limited Risk + transparency obligations.

Cápsula 04 de 08 — Módulo 4 — AI Ethics & Compliance Guide