Module 7: Building a Responsible AI Framework

Designing the checklist: structure

Next →

Description

Before filling in the content, we design the structure. How we organize 25-30 items, what format each one has, what criteria determine a "pass."

Structural decisions affect usability. Bad structure → an abandoned checklist. Good structure → a real tool the team uses.

By the end you'll be able to:

  • Organize items into 5-7 coherent sections
  • Define an item format with required fields
  • Establish pass/fail/N-A criteria
  • Design the review workflow on top of it

The overall structure: 5 sections + 1 meta

1. SECTION: Bias and Fairness (5-6 items)
   - Drawn from the M2 work

2. SECTION: Privacy and Data Protection (5-6 items)
   - Drawn from the M3 + M5 work

3. SECTION: Transparency and Explainability (4-5 items)
   - Cross-cutting

4. SECTION: Compliance (EU AI Act + GDPR) (5-6 items)
   - Drawn from M4 + M5

5. SECTION: Documentation and Governance (3-4 items)
   - Drawn from M6

META SECTION: Application & Review (3 items)
   - When applied, by whom, sign-offs

Total: 25-30 items + the meta section.

The format of each item

Each item should have 6 fields:

### Item N.M: [Short title]

**Question**: [A specific, verifiable question]

**Why this matters**: [1-2 sentences of brief justification]

**Evidence required**: [What documents/data/code prove a "yes"]

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**:
**Verified by**:
**Notes**:

An example of a well-formatted item

### Item 1.2: Demographic parity tested

**Question**: Has demographic parity been calculated for all protected
demographic groups (gender, ethnicity if applicable, age buckets) for
the system's primary decision outputs?

**Why this matters**: Demographic parity disparities indicate potential
unfair treatment. Untested systems may discriminate without detection.

**Evidence required**:
- Bias audit report (M2 deliverable) within the last 90 days
- Calculation methodology documented
- Disparate impact ratios captured

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**: 2026-05-11
**Verified by**: ML Engineer
**Notes**: DPR > 0.85 across all measured groups. A slight gap (0.82) in
age 65+ is being investigated; tracked in ticket #2451.

Item types (vague vs. specific)

A vague item (DO NOT USE)

"Is the system ethical?"

Impossible to answer. No criteria. No evidence.

A specific item (USE THIS)

"Has bias testing been conducted in the last 90 days with results documented
and any disparities >20% addressed via mitigation or accepted with
written justification?"

Specific, verifiable, with evidence required.

Pass/fail/N-A criteria

Define them explicitly:

Yes (pass)

  • The question is answered affirmatively
  • The evidence is available and current (within the timeframe)
  • Verified by the appropriate party
  • No major caveats

No (fail)

  • The question can't be answered yes
  • The evidence is missing or outdated
  • Mitigation is required before deployment

N/A (not applicable)

  • The item doesn't apply to this system
  • You must justify why: "N/A because the system doesn't process personal data"
  • Document it to prevent abuse

Borderline cases

  • Add a "Partial" option if needed: "Partial pass with caveats" + a remediation timeline

Section ordering: the logic

We order the sections in a natural flow of questioning:

1. Bias and Fairness → "Is the AI fair?"
2. Privacy → "Does it respect privacy?"
3. Transparency → "Is it understandable to users?"
4. Compliance → "Is it legally compliant?"
5. Documentation → "Is it documented for an audit?"
6. Meta → "Was the review process itself valid?"

A logical progression: ethical → legal → operational.

Section weight (not all sections are equal)

Not all sections matter equally for your system. Define a weight:

For the Knowledge Assistant:
- Bias: 25% weight (high — it affects users)
- Privacy: 30% weight (high — multi-tenant data)
- Transparency: 20% weight (medium — citations are required)
- Compliance: 20% weight (high — GDPR + the EU AI Act)
- Documentation: 5% weight (low — for the audit, not for users)

Total: 100%

Use the weight for scoring: a critical bias gap > a minor documentation gap.

The alternative: all-or-nothing: any "No" in a critical section = deployment blocked.

Status outputs: the dashboard

Each application of the checklist produces a status:

## Checklist Application — System X, Release v1.5
Date: 2026-05-11
Reviewer: Jane Doe
Triggered by: Pre-release review

**Summary**:
- Total items: 28
- Yes: 24 (86%)
- No: 2 (7%) — both Critical
- N/A: 2 (7%)
- Score: 86% weighted; 5 items pending

**Decision**: DEPLOYMENT BLOCKED
**Reason**: 2 Critical items in the Privacy section
**Remediation required**: Items 2.3 and 2.5 must reach "Yes" status

**Next review**: After remediation, expected 2026-05-20

The dashboard makes the status obvious.

Versioning the checklist

v1.0 — 2026-05-11 — Initial framework
v1.1 — 2026-06-15 — Added item 4.2 (EU AI Act risk classification)
v1.2 — 2026-09-01 — Updated 1.2 to include a new fairness metric
v2.0 — 2027-01-01 — Major refresh with new regulation

Each version is dated, the changes documented. Always use the latest version.

Common traps

Trap 1 — "Subjective" items. "Is the explanation clear?" depends on the reviewer. Make it objective: "Does the explanation include source citations? (Y/N)"

Trap 2 — No time bounds. "Has bias testing been conducted?" — at some point? It could have been 2 years ago. Add a timeframe: "in the last 90 days."

Trap 3 — No evidence required. A "Yes" with no proof. Every "Yes" must have an evidence link/reference.

Trap 4 — All items equal. A trivial item ≠ a critical item. Use weighting or a critical/non-critical classification.

Trap 5 — No update cycle. Checklist v1 forever. Regulation changes. Practices evolve. Schedule updates.

Exercise

Design the structure of your checklist:

  1. List 5-6 sections with item counts
  2. For 3 items, write the formal form (with all the fields)
  3. Define the weighting or the critical/non-critical classification
  4. Design the output template
See the solution (skeleton)

Sections:

  1. Bias and Fairness — 6 items
  2. Privacy and Data Protection — 7 items
  3. Transparency and Explainability — 5 items
  4. Compliance (EU AI Act + GDPR) — 7 items
  5. Documentation and Governance — 4 items
  6. Meta: Application and Review — 3 items

Total: 32 items.

Sample formal items: (use the template above)

Weighting:

  • Bias: 25%
  • Privacy: 30%
  • Transparency: 15%
  • Compliance: 25%
  • Documentation: 5%

Critical items (any "No" blocks deployment):

  • All the items in Privacy
  • Item 4.1 (EU AI Act classification)
  • Item 4.3 (GDPR Art. 22 compliance)
  • Item 4.5 (Data subject rights)

Output template: as shown in the module.

Summary

You learned:

  • ✅ The structure: 5-6 sections + meta, 25-30 items total
  • ✅ The item format: question, why it matters, evidence, status, date, verifier
  • ✅ Specific vs. vague items
  • ✅ Pass/fail/N-A criteria
  • ✅ Section ordering, weighting, critical items
  • ✅ The output dashboard
  • ✅ Versioning

Checkpoint: if your structure is manageable + has a clear format + is actionable, you're ready.

Next capsule

03 — Checklist content: bias + fairness. We fill in the first section with specific items derived from M2.

Resources

  1. Microsoft Responsible AI Standard PDF.
  2. Google AI Principles.
  3. Aequitas — bias audit.
  4. SecurityScorecard's RAFT framework — an example of an operationalized checklist.
← PreviousIntroduction to the Responsible AI FrameworkNext →Checklist content: bias + fairness