Module 7: Building a Responsible AI Framework

Review processes

Description

You have the items. What's missing is the process: when the checklist gets applied, who participates, what happens with failures. Without a process, the checklist is decoration.

By the end you'll be able to:

Define the triggers that activate a checklist application
Assign reviewers and sign-offs
Implement remediation workflows
Document application records

Triggers: when to apply it

5 common triggers:

Trigger 1: Pre-production release

Before any deployment to production. Mandatory.

Trigger: PR merged to main + a deployment plan submitted
Required: A full checklist application
Approver: Tech Lead (minimum)
Blocking: Yes — the release is blocked if Critical items fail

Trigger 2: Significant model/data change

When the AI model is updated, retrained, or the data sources change.

Trigger: A model version update OR a data source addition/removal
Required: Sections 1 (bias) + 2 (privacy) re-applied
Approver: ML Engineer + Tech Lead
Blocking: Yes for significant changes

Trigger 3: A new use case for an existing system

The system is repurposed or a new feature is added.

Trigger: A product feature spec involving AI
Required: Sections 1, 3, 4 re-evaluated
Approver: Tech Lead + Product

Trigger 4: Post-incident

After any AI-related incident affecting users.

Trigger: An incident ticket P1 or higher is closed
Required: The full checklist (verify nothing else broke)
Approver: Tech Lead

Trigger 5: Periodic review

An annual or semi-annual deep review regardless of changes.

Trigger: The calendar (every 12 months minimum)
Required: The full checklist + a framework update
Approver: Tech Lead + DPO (if applicable)

Reviewers: who participates

Typical roles

Role	Responsibility
AI Reviewer (Tech Lead or a designate)	Drives the review process, verifies the items
ML Engineer	Verifies the items in the bias/fairness section
DPO / Legal counsel	The privacy + compliance sections
Product Lead	The use case implications
Security (if separate)	Security-related items

At a small startup, one person can wear multiple hats. But document it.

Sign-offs

Critical items typically require a dual sign-off:

The AI Reviewer (the item is verified)
The stakeholder owner (accepts any risks)

For routine items, a single sign-off is sufficient.

The remediation workflow

When an item fails (status "No"):

1. Document the gap:
   - The specific item failing
   - The reason for the failure
   - The impact if it isn't addressed

2. Classify it:
   - Critical (blocks deployment)
   - High (must be addressed within the sprint)
   - Medium (must be addressed within the quarter)
   - Low (a planned improvement)

3. Assign an owner + a target date

4. Plan the mitigation:
   - Action items
   - Resources required
   - Dependencies

5. Track it:
   - Status updates in the tracking system
   - Re-verify when complete
   - Update the checklist status

6. Sign off when resolved

An example: a failing item handled

**Failed item**: 1.6 (Ongoing bias monitoring in production)

**Gap**: No production monitoring of bias metrics; a baseline measurement
only.

**Impact**: We can't detect bias drift post-deployment. Risk: medium.

**Classification**: High (must be addressed within the sprint)

**Owner**: ML Engineer
**Target date**: 2026-Q3 sprint 2

**Plan**:
- Implement segmented metric collection (per protected attribute)
- Set up a dashboard in Datadog
- Configure alerts when DPR drops below 0.85
- Validate with synthetic data first

**Re-verification**: Once the dashboard and alerts are operational, re-apply
item 1.6 to confirm.

The application record

Each application of the checklist produces a record:

# Checklist Application Record

**Date**: 2026-05-11
**Reviewer(s)**: Tech Lead (Jane Doe), DPO (John Smith)
**Triggered by**: Pre-production release v1.5
**Checklist version**: v1.1

**System**: AI Knowledge Assistant
**Release**: v1.5 (planned 2026-05-15)

**Application duration**: 2h 15min

## Summary
- Total items: 28
- Yes: 24 (86%)
- No: 3 (11%)
- N/A: 1 (3%)

## Critical items status
- All Critical items: Yes/No
- [list of critical "Yes" items]
- [list of critical "No" items]

## Items requiring action
| Item | Status | Owner | Target |
|------|--------|-------|--------|
| 1.6 | No | ML Engineer | 2026-Q3 |
| 4.6 | No | DPO | 2026-05-30 |
| 5.2 | No | Tech Lead | 2026-05-25 |

## Decision
[ ] APPROVED — ready for production
[X] CONDITIONALLY APPROVED — 1 item must reach Yes before release
[ ] BLOCKED — Critical items failing

## Conditions
- Item 4.6 (international transfer safeguards) must reach Yes
  - Status: SCCs being signed; expected 2026-05-25
- Items 1.6 and 5.2 can be remediated post-release as High priority

## Signatures
- AI Reviewer: ___________ Date: __________
- DPO: ___________ Date: __________
- Tech Lead: ___________ Date: __________

## Next review
- Conditional: re-verify 4.6 before deployment
- Periodic: 2026-11-11 (6 months)

Integrating with CI/CD

For an embedded process:

# .github/workflows/ai-deployment-gate.yml
name: AI Ethics Gate

on:
  pull_request:
    paths:
      - 'ai-service/**'
      - 'models/**'

jobs:
  ethics-check:
    runs-on: ubuntu-latest
    steps:
      - name: Check for ethics checklist application
        run: |
          if [ ! -f "ethics-reviews/release-${RELEASE_VERSION}.md" ]; then
            echo "::error::No ethics review for this release"
            exit 1
          fi
      
      - name: Verify critical items
        run: |
          # Script that parses the review file
          # Fails if any Critical item is "No"
          python verify_ethics_review.py ethics-reviews/release-${RELEASE_VERSION}.md
      
      - name: Comment on PR
        # Adds a summary of the ethics review to the PR
        # ...

Ethics becomes part of the CI gate, not a "thing we do later."

Common traps

Trap 1 — Review as a rubber stamp. The reviewer signs off without actually verifying. The sign-off must be meaningful.

Trap 2 — No SLA on remediation. "Will address" with no timeline. Items linger forever. Target dates are required.

Trap 3 — No tracking of re-verification. An item failed, was fixed, and never re-verified. The status stays "No" in the records. The update procedure is mandatory.

Trap 4 — Trigger 4 (post-incident) skipped. "The incident is closed, move on." The re-check gets skipped. Other items may have broken.

Trap 5 — Untrained reviewers. Anyone signs off without understanding. Train the reviewers properly.

Exercise

Design the review processes for your Capstone:

Specific triggers (5 minimum)
Reviewers for each trigger
The remediation workflow
A CI/CD integration sketch

Summary

You learned:

✅ 5 triggers: pre-release, model change, new use case, post-incident, periodic
✅ Reviewer roles + sign-offs (dual for Critical items)
✅ The remediation workflow with classifications + targets
✅ The application record template
✅ A CI/CD integration sketch
✅ The traps: rubber stamping, no SLA, no re-verification

Checkpoint: if your team can execute the review process without your hand-holding, it's OK.

Next capsule

06 — Documentation templates. Reviews produce documents. Templates ensure consistency.

Resources

Atlassian — Approval workflows.
GitHub PR templates.
Microsoft Responsible AI Standard PDF — a review workflow example.

← PreviousThe Privacy, GDPR, and EU AI Act sections of the checklist Next →Documentation templates