Módulo 7: Building a Responsible AI Framework

Review processes

Siguiente →

Descripción

Tenés items. Falta el proceso: cuándo se aplica el checklist, quién participa, qué pasa con failures. Sin proceso, el checklist es decoración.

Al terminar vas a poder:

  • Definir triggers que activan checklist application
  • Asignar reviewers y sign-offs
  • Implementar remediation workflows
  • Documentar application records

Triggers: cuándo aplicar

5 triggers comunes:

Trigger 1: Pre-production release

Before any deployment a production. Mandatory.

Trigger: PR merge to main + deployment plan submitted
Required: Full checklist application
Approver: Tech Lead (minimum)
Blocking: Yes — release blocked if Critical items fail

Trigger 2: Significant model/data change

When the AI model is updated, retrained, or data sources changed.

Trigger: Model version update OR data source addition/removal
Required: Sections 1 (bias) + 2 (privacy) re-applied
Approver: ML Engineer + Tech Lead
Blocking: Yes for significant changes

Trigger 3: New use case for existing system

System repurposed or new feature added.

Trigger: Product feature spec involving AI
Required: Sections 1, 3, 4 re-evaluated
Approver: Tech Lead + Product

Trigger 4: Post-incident

After any AI-related incident affecting users.

Trigger: Incident ticket P1 or higher closed
Required: Full checklist (verify nothing else broken)
Approver: Tech Lead

Trigger 5: Periodic review

Annual or semi-annual deep review regardless of changes.

Trigger: Calendar (every 12 months minimum)
Required: Full checklist + framework update
Approver: Tech Lead + DPO (if applicable)

Reviewers: quién participa

Roles típicos

RoleResponsibility
AI Reviewer (Tech Lead or designate)Drives the review process, verifies items
ML EngineerVerifies items in bias/fairness section
DPO / Legal counselPrivacy + compliance sections
Product LeadUse case implications
Security (if separate)Security-related items

For small startup, one person can wear multiple hats. But document it.

Sign-offs

Critical items typically require dual sign-off:

  • AI Reviewer (item verified)
  • Stakeholder owner (accepts any risks)

For routine items, single sign-off sufficient.

Remediation workflow

When item fails (status "No"):

1. Document the gap:
   - Specific item failing
   - Reason for failure
   - Impact if not addressed

2. Classify:
   - Critical (blocks deployment)
   - High (must address within sprint)
   - Medium (must address within quarter)
   - Low (planned improvement)

3. Assign owner + target date

4. Plan mitigation:
   - Action items
   - Resources required
   - Dependencies

5. Track:
   - Status updates in tracking system
   - Re-verify when complete
   - Update checklist status

6. Sign-off when resolved

Example: failing item handled

**Failed item**: 1.6 (Ongoing bias monitoring in production)

**Gap**: No production monitoring of bias metrics; baseline measurement
only.

**Impact**: Cannot detect bias drift post-deployment. Risk: medium.

**Classification**: High (must address within sprint)

**Owner**: ML Engineer
**Target date**: 2026-Q3 sprint 2

**Plan**:
- Implement segmented metric collection (per protected attr)
- Setup dashboard in Datadog
- Configure alerts when DPR drops below 0.85
- Validate with synthetic data first

**Re-verification**: Once dashboard and alerts operational, re-apply
item 1.6 to confirm.

Application record

Each application of checklist produces a record:

# Checklist Application Record

**Date**: 2026-05-11
**Reviewer(s)**: Tech Lead (Jane Doe), DPO (John Smith)
**Triggered by**: Pre-production release v1.5
**Checklist version**: v1.1

**System**: AI Knowledge Assistant
**Release**: v1.5 (planned 2026-05-15)

**Application duration**: 2h 15min

## Summary
- Total items: 28
- Yes: 24 (86%)
- No: 3 (11%)
- N/A: 1 (3%)

## Critical items status
- All Critical items: Yes/No
- [list critical "Yes" items]
- [list critical "No" items]

## Items requiring action
| Item | Status | Owner | Target |
|------|--------|-------|--------|
| 1.6 | No | ML Engineer | 2026-Q3 |
| 4.6 | No | DPO | 2026-05-30 |
| 5.2 | No | Tech Lead | 2026-05-25 |

## Decision
[ ] APPROVED — ready for production
[X] CONDITIONALLY APPROVED — 1 item must reach Yes before release
[ ] BLOCKED — Critical items failing

## Conditions
- Item 4.6 (international transfers safeguards) must reach Yes
  - Status: SCCs being signed; expected 2026-05-25
- Items 1.6 and 5.2 can be remediated post-release as High priority

## Signatures
- AI Reviewer: ___________ Date: __________
- DPO: ___________ Date: __________
- Tech Lead: ___________ Date: __________

## Next review
- Conditional: re-verify 4.6 before deployment
- Periodic: 2026-11-11 (6 months)

Integrar con CI/CD

Para process embedded:

# .github/workflows/ai-deployment-gate.yml
name: AI Ethics Gate

on:
  pull_request:
    paths:
      - 'ai-service/**'
      - 'models/**'

jobs:
  ethics-check:
    runs-on: ubuntu-latest
    steps:
      - name: Check for ethics checklist application
        run: |
          if [ ! -f "ethics-reviews/release-${RELEASE_VERSION}.md" ]; then
            echo "::error::No ethics review for this release"
            exit 1
          fi
      
      - name: Verify critical items
        run: |
          # Script that parses review file
          # Fails if any Critical item is "No"
          python verify_ethics_review.py ethics-reviews/release-${RELEASE_VERSION}.md
      
      - name: Comment on PR
        # Adds summary of ethics review to PR
        # ...

Ethics becomes part of CI gate, not "thing we do later".

Trampas comunes

Trampa 1 — Review como rubber stamp. Reviewer signs off sin actually verificar. Sign-off must be meaningful.

Trampa 2 — Sin SLA en remediation. "Will address" no time. Items lingering forever. Target dates required.

Trampa 3 — No tracking re-verification. Item failed, fixed, never re-verified. Status stays "No" en records. Update procedure mandatory.

Trampa 4 — Trigger 4 (post-incident) skipped. "Incident closed, move on". Skipped re-check. Other items may have broken.

Trampa 5 — Reviewers sin training. Anyone signs off without understanding. Train reviewers properly.

Ejercicio

Diseñá los review processes for tu Capstone:

  1. Triggers específicos (5 mín)
  2. Reviewers para each trigger
  3. Remediation workflow
  4. CI/CD integration sketch

Resumen

Aprendiste:

  • ✅ 5 triggers: pre-release, model change, new use case, post-incident, periodic
  • ✅ Reviewer roles + sign-offs (dual for Critical)
  • ✅ Remediation workflow con classifications + targets
  • ✅ Application record template
  • ✅ CI/CD integration sketch
  • ✅ Trampas: rubber stamp, sin SLA, no re-verification

Checkpoint: si tu team puede ejecutar review process sin tu hand-holding, está OK.

Siguiente cápsula

06 — Documentation templates. Reviews producen documents. Templates assegurán consistency.

Recursos

  1. Atlassian — Approval workflows.
  2. GitHub PR templates.
  3. Microsoft Responsible AI Standard PDF — review workflow example.
← AnteriorPrivacy, GDPR y EU AI Act sections del checklistSiguiente →Documentation templates