Módulo 7: Building a Responsible AI Framework

Diseñando el checklist: estructura

Siguiente →

Descripción

Antes de llenar contenido, diseñamos la estructura. Cómo organizamos 25-30 items, qué formato tiene cada uno, qué criterios determinan "pass".

Decisiones de estructura afectan usability. Bad structure → checklist abandonado. Good structure → tool real que team usa.

Al terminar vas a poder:

  • Organizar items en 5-7 sections coherentes
  • Definir formato de item con campos requeridos
  • Establecer criteria de pass/fail/N-A
  • Diseñar review workflow on top

Estructura overall: 5 sections + 1 meta

1. SECTION: Bias and Fairness (5-6 items)
   - Drawn from M2 work

2. SECTION: Privacy and Data Protection (5-6 items)
   - Drawn from M3 + M5 work

3. SECTION: Transparency and Explainability (4-5 items)
   - Cross-cutting

4. SECTION: Compliance (EU AI Act + GDPR) (5-6 items)
   - Drawn from M4 + M5

5. SECTION: Documentation and Governance (3-4 items)
   - Drawn from M6

META SECTION: Application & Review (3 items)
   - When applied, by whom, sign-offs

Total: 25-30 items + meta section.

Formato de cada item

Each item should have 6 fields:

### Item N.M: [Short title]

**Question**: [Specific verifiable question]

**Why this matters**: [1-2 sentences brief justification]

**Evidence required**: [What documents/data/code prove "yes"]

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**:
**Verified by**:
**Notes**:

Ejemplo de item bien formateado

### Item 1.2: Demographic parity tested

**Question**: Has demographic parity been calculated for all protected
demographic groups (gender, ethnicity if applicable, age buckets) for
the system's primary decision outputs?

**Why this matters**: Demographic parity disparities indicate potential
unfair treatment. Untested systems may discriminate without detection.

**Evidence required**:
- Bias audit report (M2 deliverable) within last 90 days
- Calculation methodology documented
- Disparate impact ratios captured

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**: 2026-05-11
**Verified by**: ML Engineer
**Notes**: DPR > 0.85 across all measured groups. Slight gap (0.82) in
age 65+ which is being investigated; tracked in ticket #2451.

Tipos de items (vagos vs específicos)

Item vago (NO USAR)

"Is the system ethical?"

Imposible de answer. No criteria. No evidence.

Item específico (USAR)

"Has bias testing been conducted in last 90 days with results documented
and any disparities >20% addressed via mitigation or accepted with
written justification?"

Specific, verifiable, evidence required.

Criteria de pass/fail/N-A

Define explicitly:

Yes (pass)

  • Question answered affirmatively
  • Evidence available and current (within timeframe)
  • Verified by appropriate party
  • No major caveats

No (fail)

  • Question can't be answered yes
  • Evidence missing or outdated
  • Mitigation required before deployment

N/A (not applicable)

  • Item doesn't apply to this system
  • Must justify why: "N/A because system doesn't process personal data"
  • Documentar to prevent abuse

Borderline cases

  • Add "Partial" option if needed: "Partial pass with caveats" + remediation timeline

Section ordering: lógica

Ordenamos sections en flujo natural de questioning:

1. Bias and Fairness → "Is the AI fair?"
2. Privacy → "Does it respect privacy?"
3. Transparency → "Is it understandable to users?"
4. Compliance → "Is it legally compliant?"
5. Documentation → "Is it documented for audit?"
6. Meta → "Was the review process valid?"

Logical progression: ethical → legal → operational.

Section weight (not all sections equal)

Not all sections matter equally para tu sistema. Define weight:

For Knowledge Assistant:
- Bias: 25% weight (high — affects users)
- Privacy: 30% weight (high — multi-tenant data)
- Transparency: 20% weight (medium — citations require)
- Compliance: 20% weight (high — GDPR + EU AI Act)
- Documentation: 5% weight (low — for audit not for users)

Total: 100%

Use weight for scoring: critical bias gap > minor documentation gap.

Alternative: all-or-nothing: any "No" in critical section = blocked deployment.

Status outputs: el dashboard

Each application of checklist produces a status:

## Checklist Application — System X, Release v1.5
Date: 2026-05-11
Reviewer: Jane Doe
Triggered by: Pre-release review

**Summary**:
- Total items: 28
- Yes: 24 (86%)
- No: 2 (7%) — both Critical
- N/A: 2 (7%)
- Score: 86% weighted; 5 items pending

**Decision**: DEPLOYMENT BLOCKED
**Reason**: 2 Critical items in Privacy section
**Remediation required**: Items 2.3 and 2.5 must reach "Yes" status

**Next review**: After remediation, expected 2026-05-20

Dashboard makes status obvious.

Versionado del checklist

v1.0 — 2026-05-11 — Initial framework
v1.1 — 2026-06-15 — Added item 4.2 (EU AI Act risk classification)
v1.2 — 2026-09-01 — Updated 1.2 to include new fairness metric
v2.0 — 2027-01-01 — Major refresh with new regulation

Each version dated, changes documented. Always use latest version.

Trampas comunes

Trampa 1 — Items "subjective". "Is the explanation clear?" depends on reviewer. Hace it objetivo: "Does explanation include source citations? (Y/N)"

Trampa 2 — Sin time bounds. "Has bias testing been conducted?" alguna vez → could be 2 years ago. Add timeframe: "in last 90 days".

Trampa 3 — Sin evidence required. "Yes" with no proof. Each "Yes" must have evidence link/reference.

Trampa 4 — All items equal. Trivial item ≠ Critical item. Use weighting or critical/non-critical classification.

Trampa 5 — No update cycle. Checklist v1 forever. Regulation changes. Practices evolve. Schedule updates.

Ejercicio

Diseñá la estructura de tu checklist:

  1. Lista 5-6 sections con item counts
  2. Para 3 items, write formal form (con todos los campos)
  3. Define weighting o critical/non-critical
  4. Diseñá output template
Ver solución (skeleton)

Sections:

  1. Bias and Fairness — 6 items
  2. Privacy and Data Protection — 7 items
  3. Transparency and Explainability — 5 items
  4. Compliance (EU AI Act + GDPR) — 7 items
  5. Documentation and Governance — 4 items
  6. Meta: Application and Review — 3 items

Total: 32 items.

Sample formal items: (use template above)

Weighting:

  • Bias: 25%
  • Privacy: 30%
  • Transparency: 15%
  • Compliance: 25%
  • Documentation: 5%

Critical items (any "No" blocks deployment):

  • All items in Privacy
  • Item 4.1 (EU AI Act classification)
  • Item 4.3 (GDPR Art. 22 compliance)
  • Item 4.5 (Data subject rights)

Output template: as shown in module.

Resumen

Aprendiste:

  • ✅ Structure: 5-6 sections + meta, 25-30 items total
  • ✅ Item format: question, why matters, evidence, status, date, verifier
  • ✅ Specific vs vague items
  • ✅ Pass/fail/N-A criteria
  • ✅ Section ordering, weighting, critical items
  • ✅ Output dashboard
  • ✅ Versionado

Checkpoint: si tu estructura es manageable + clear formato + actionable, estás listo.

Siguiente cápsula

03 — Contenido del checklist: bias + fairness. Llenamos la primera section con items específicos derivados de M2.

Recursos

  1. Microsoft Responsible AI Standard PDF.
  2. Google AI Principles.
  3. Aequitas — bias audit.
  4. SecurityScorecard's RAFT framework — example of operationalized checklist.
← AnteriorIntroducción al Responsible AI FrameworkSiguiente →Contenido del checklist: bias + fairness