4. High-Risk Obligations: The 6 Requirements

Capsule description

If your system is high-risk, you have 6 main obligations under the EU AI Act:

1. Risk management system (Art. 9).
2. Data and data governance (Art. 10).
3. Technical documentation (Art. 11).
4. Record-keeping and logging (Art. 12).
5. Transparency and information to deployers (Art. 13).
6. Human oversight (Art. 14).

Plus, critically: accuracy, robustness, and cybersecurity (Art. 15) and conformity assessment + registration (Art. 43, 49, 60).

This capsule translates each obligation from legal language into a specific engineering action. By the end, you'll have a concrete checklist of what to do to meet each one.

---

Obligation 1: Risk Management System (Art. 9)

What the Act asks for

"A risk management system shall be established, implemented, documented and maintained..."

The technical translation

A continuous process of:

1. Identifying the risks of the system.
2. Estimating and evaluating known/foreseeable risks.
3. Adopting mitigation measures.
4. Testing the effectiveness of the mitigations.
5. Updating continuously.

The engineering action

## Risk Management System (RMS)

### 1. Risk Register
| Risk | Likelihood | Severity | Mitigation | Status | Owner |
|------|-----------|----------|------------|--------|-------|
| Gender bias | High | High | Bias testing in CI | In progress | ML eng |
| Privacy leak in logs | Medium | High | PII redaction | Implemented | DevOps |
| Hallucination | Medium | Medium | Output filter | Implemented | ML eng |
| ... | | | | | |

### 2. Mitigation testing
- Quarterly: re-run bias tests.
- Continuously: monitor privacy metrics.
- Per-release: regression tests.

### 3. Updates
- After incidents.
- After feature changes.
- After regulatory updates.

The connection to M1: the Ethics Impact Analysis is a direct input to the Risk Register.

---

Obligation 2: Data and Data Governance (Art. 10)

What the Act asks for

Training, validation, and testing data sets must meet requirements for:

• Relevance and representativeness for the purpose.
• Free of errors as far as possible.
• Appropriate statistical properties (distribution, etc.).
• Bias examination and mitigation.
• Documentation of data sources.

The engineering action

## Data Governance

### 1. Data Sources Documentation
- Source 1: Internal database. Period: 2020-2025.
- Source 2: Public dataset X. License: CC-BY.
- Source 3: Synthetic data generated by process Y.

### 2. Data Quality
- Completeness: 95% (5% have missing fields, dropped).
- Accuracy: validated against ground truth where available.
- Freshness: refreshed quarterly.

### 3. Bias Examination
- Demographic distribution analyzed (M2 Bias Audit).
- Identified imbalances + mitigations applied.
- Documented in model card.

### 4. Statistical Validation
- Train/val/test split: 70/15/15.
- No data leakage between splits (verified).
- Distribution similar between splits (verified).

The connection to M2: the Bias Audit Toolkit produces documentation that directly satisfies this obligation.

The connection to M3: the Privacy Assessment + minimization apply here too.

---

Obligation 3: Technical Documentation (Art. 11)

What the Act asks for

Documentation that demonstrates compliance with all the requirements. It must include (Annex IV):

• A general description of the system.
• A detailed description of the design and development process.
• Detailed information about monitoring, functioning, and control.
• Risk management documentation.
• Data documentation.
• Performance metrics.
• Cybersecurity measures.
• Etc.

The engineering action

The documentation is maintained as a living artifact. Format:

## Technical Documentation v1.0

### 1. System Description
- Purpose: medical triage classification
- Architecture: LLM + RAG + classifier
- Inputs: user symptoms text
- Outputs: triage urgency category

### 2. Design Decisions
- LLM choice: GPT-4 (enterprise)
- Decision rationale: trade-off accuracy vs cost
- Alternatives considered: open-source models

### 3. Training Data
- Source documents (linked)
- Quality measures (linked to bias audit)

### 4. Performance
- Accuracy: 88% on test set
- Per-subgroup performance (linked)
- Limitations: known issues with rare conditions

### 5. Risk Mitigations
- (Linked to Risk Management Doc)

### 6. Human Oversight
- (Linked to Human Oversight Doc)

### 7. Monitoring
- Production metrics tracked
- Alerts configured
- Re-evaluation schedule

### 8. Cybersecurity
- (Linked to Security Doc)

Best practice: maintain it in a repository (Git), versioned, and accessible to the regulator if requested.

---

Obligation 4: Record-Keeping and Logging (Art. 12)

What the Act asks for

The system must automatically generate logs of:

• The period of each use.
• A reference to the inputs.
• The persons involved (humans in the loop).
• To enable traceability and enforcement.

The engineering action

# Required logging for high-risk AI
@app.post("/predict")
async def predict(request):
    log_record = {
        'timestamp': now(),
        'user_id': hash(request.user_id),  # Pseudonymized
        'inputs_hash': hash(request.inputs),  # Don't log raw PII
        'model_version': model.version,
        'output_summary': summarize_output(prediction),
        'human_reviewer': request.assigned_reviewer if applicable else None,
        'override_flag': prediction.was_overridden,
    }
    
    audit_logger.log(log_record)
    
    return prediction

Retention: typically 6+ years for compliance (it varies by use case).

---

Obligation 5: Transparency and Information (Art. 13)

What the Act asks for

The system must be transparent enough that deployers can:

• Understand how it works.
• Interpret the outputs correctly.
• Use it appropriately.

The provider must furnish:

• Instructions for use.
• A description of the intended use.
• Limitations.
• Performance characteristics.
• The human oversight required.
• Maintenance requirements.

The engineering action

A model card + a deployer guide + API documentation.

An example model card (excerpt):

## Model Card: Medical Triage Classifier v2.1

### Intended Use
- Triage classification for non-emergency adult cases.
- NOT for: emergencies, pediatrics, mental health crises.

### Performance
- Accuracy on validation set: 88%.
- Reliability: highest for English, 85% Spanish, 80% French.
- Edge cases: poor performance on rare conditions (<5% prevalence).

### Required Human Oversight
- A healthcare professional must review urgent classifications.
- For non-urgent: the user can act on it directly.

### Deployment Requirements
- Must be integrated with a patient communication system.
- A healthcare professional must be available within 1 hour for urgent cases.
- Must comply with HIPAA + local medical regulations.

### Limitations
- Cannot diagnose specific conditions.
- May misclassify edge cases.
- Bias mitigation: see bias audit (linked).

### Updates
- Re-trained quarterly.
- Major changes communicated to deployers.

---

Obligation 6: Human Oversight (Art. 14)

What the Act asks for

The system must permit effective human oversight:

• Understanding what the system is doing.
• Monitoring its operation.
• Interpreting the outputs.
• Deciding to override or not use the output.
• Stopping the system if needed.

The engineering action

Human oversight is an architectural decision, not just a UI feature.

class HumanOversightAwareSystem:
    def __init__(self):
        self.review_queue = ReviewQueue()
        self.override_log = OverrideLog()
    
    async def make_prediction(self, input_data, threshold=0.7):
        # Generate prediction
        prediction = self.model.predict(input_data)
        confidence = prediction.confidence
        
        # Reject option: low confidence → human review
        if confidence < threshold:
            self.review_queue.add(input_data, prediction)
            return {
                'status': 'pending_human_review',
                'estimated_time': '2 hours',
            }
        
        # High confidence: still log for audit
        self.audit_logger.log_prediction(input_data, prediction)
        return prediction
    
    def human_override(self, prediction_id, new_decision, reason):
        """Allow humans to override AI decisions."""
        self.override_log.record({
            'prediction_id': prediction_id,
            'original': self.get_prediction(prediction_id),
            'overridden_to': new_decision,
            'reason': reason,
            'reviewer': current_user(),
            'timestamp': now(),
        })
        
        # Update output
        self.update_output(prediction_id, new_decision)
        
        # Feed back into improvements
        self.training_feedback.add(prediction_id, new_decision)

Levels of human oversight

The EU AI Act distinguishes:

1. Human-in-the-loop: a human approves each decision.
2. Human-on-the-loop: a human monitors and intervenes when needed.
3. Human-in-command: a human can stop the system.

Different use cases require different levels. High-stakes (medical) typically requires 1 or 2. Lower-stakes ones can be 3 only.

---

Additional: Accuracy, Robustness, Cybersecurity (Art. 15)

The requirements

• Accuracy: an appropriate level for the intended purpose.
• Robustness: it works correctly under varied conditions.
• Cybersecurity: protected against unauthorized changes.

The engineering action

## Accuracy
- Defined accuracy metrics for the use case.
- Threshold: at least 85% (justified for medical triage).
- Per-subgroup performance documented.

## Robustness
- Tested with adversarial inputs.
- Tested with distribution shift scenarios.
- Failover mechanism: if confidence < threshold, human review.

## Cybersecurity
- Model weights encrypted at rest.
- API authenticated and rate-limited.
- Adversarial robustness testing.
- Vulnerability scanning.
- Incident response plan.

---

Additional: Conformity Assessment + Registration (Art. 43, 49, 60)

Conformity assessment

Before deploying high-risk AI:

• Self-assessment for some categories.
• Third-party assessment for others (a notified body).
• Documentation of the assessment.

Registration

• Register the system in the EU database of high-risk AI.
• Update the registration when the system changes.
• Public-facing (some details).

---

Common traps

1. Documentation with no substance

Documents that describe what the system does without actually addressing the requirements. Regulators read carefully — empty documentation invalidates the compliance claim.

2. Human oversight as a UI checkbox

An "Approve?" button that humans always click isn't real oversight. You need genuine decision authority + override capability + actual usage tracking.

3. Risk management as a one-time exercise

The RMS is a continuous process. Document the updates. Re-run risk assessments quarterly.

4. Skipping cybersecurity

Often deprioritized. But Art. 15 makes it mandatory. AI-specific threats (prompt injection, model inversion) must be addressed.

5. Delaying the conformity assessment

Some categories require a third-party assessment that takes weeks/months. Don't wait until the last minute.

---

Self-check

1. Why is documentation with no substance a problem?

Regulators aren't impressed by the volume of documentation — they evaluate whether it actually addresses the requirements.

Bad documentation:

• "We tested for bias." (no methodology, no metrics, no results)
• "Risk management is in place." (no register, no review process)
• "Human oversight implemented." (no description of how)

Good documentation:

• "Bias tested using demographic parity (4/5 rule), equalized odds (5% threshold), counterfactual fairness. Results: 0.85, 4.2%, 0.05. See appendix B for methodology and full results."
• "Risk register maintained at [link]. Reviewed quarterly. Last review: [date]. Three risks identified, two mitigated, one in progress."
• "Human reviewer required for confidence < 0.7. Reviewer dashboard at [link]. Override functionality: see Section 4.3 of the Operations Manual."

Specific, verifiable, traceable. That's what compliance documentation looks like.

2. How do you distinguish real human oversight from cosmetic oversight?

Real human oversight:

• Humans actually review the outputs (not just rubber-stamp them).
• Humans have the authority to override.
• Override decisions are tracked and affect the future.
• Reviewers are trained for their role.
• The process is monitored for effectiveness.

Cosmetic oversight:

• Humans click "approve" on hundreds per hour with no real review.
• Override is available but rarely used (no time, no expertise).
• Override decisions disappear into a void.
• Reviewers are untrained.
• No metrics on oversight effectiveness.

The test:

• What % of decisions get overridden? (If <1%, it may be cosmetic.)
• How long do reviewers spend per decision? (If <30 seconds for complex decisions, cosmetic.)
• Do reviewers have the authority and ability to dissent?

Auditors check this. "We have a review step" isn't enough — it has to be effective.

3. What's the difference between human-in-the-loop and human-on-the-loop?

Human-in-the-loop:

• A human reviews every AI decision before action.
• Sequential: AI → human → action.
• The strongest oversight, the slowest.
• Used for: high-stakes (medical, legal, criminal justice).

Human-on-the-loop:

• The AI acts automatically; a human monitors the patterns.
• Parallel: AI → action, and the human can intervene if needed.
• Faster, less oversight.
• Used for: medium-stakes with clear failure modes.

Human-in-command:

• A human can stop the system entirely if needed.
• Minimal oversight.
• Used for: low-stakes automation.

The EU AI Act doesn't mandate a specific level; it depends on the system's risk + use case. Medical typically requires "in-the-loop." Customer service, "on-the-loop." Recommendations, "in-command."

Document the choice + the reasoning.

4. Why must Risk Management be continuous, not one-time?

Systems evolve continuously:

• Data drift: production inputs differ from the training data over time.
• Model updates: retraining introduces new behaviors.
• Use case shifts: systems get used in unforeseen ways.
• New threats: adversarial techniques evolve.
• Regulatory updates: requirements change.

A one-time risk assessment becomes outdated within months.

A continuous RMS:

• Quarterly review of the risk register.
• Re-test the mitigations after changes.
• Add new risks as they emerge.
• Update the documentation continuously.
• Audit logs that show the review history.

The compliance evidence: the regulator wants to see ongoing engagement, not a single document from 2 years ago.

---

Summary and next step

• 6 main obligations for high-risk AI: risk management, data governance, technical documentation, record-keeping, transparency, human oversight.
• Plus: accuracy/robustness/cybersecurity, conformity assessment, registration.
• Each one requires an engineering action, not just paperwork.
• Continuous processes: RMS, monitoring, updates.
• The connection to previous modules: M1's risk analysis, M2's bias docs, and M3's privacy assessment all feed into the compliance documentation.

Checkpoint: you should be able to map each obligation to a concrete engineering deliverable.

Bridge to the next capsule: capsule 05 covers Limited Risk — chatbots, deepfakes, transparency obligations. Much simpler than high-risk, but it still requires specific actions.

---

Resources

1. EU AI Act Articles 9-15 — the text.
2. Annex IV — Technical documentation — the full requirements.

---

Next: 05-limited-risk-transparency.md — Limited Risk + transparency obligations.

Capsule 04 of 08 — Module 4 — AI Ethics & Compliance Guide