Module 8: Capstone Project — Ethics Audit of an AI System
Findings: the Bias + Privacy sections
Description
How to write the Findings: Bias and Fairness and Findings: Privacy and Data Protection sections of the Ethics Audit Report. The two most technically critical sections.
By the end you'll be able to:
- Structure a findings section professionally
- Communicate technical findings without losing rigor
- Quantify the evidence with real metrics
- Identify patterns and root causes
The structure of every findings section
# Section X: [Section Name]
## X.1 Overview
[1 paragraph: the scope of this section, the items audited]
## X.2 Status summary
| Item | Description | Status | Critical |
|------|-------------|--------|----------|
| 1.1 | ... | ✅ Yes | Yes |
| 1.2 | ... | ❌ No | Yes |
| ... |
## X.3 Detailed findings
### Compliant items (briefly)
[List what's working]
### Gaps identified (detailed)
For each "No" item:
#### Item X.Y: [Title]
**Current state**: [What's happening now]
**Required state**: [What the checklist requires]
**Gap**: [The specific difference]
**Evidence reviewed**:
- [What you looked at]
**Root cause**: [Why the gap exists]
**Impact**: [If it isn't addressed]
**Recommendation**: [A concrete action]
**Priority**: Critical/High/Medium/Low
**Owner**: [Suggested]
**Estimated effort**: [Hours/days]
## X.4 Patterns and themes
[Cross-cutting observations]
## X.5 Recommendations summary
[The list of recommendations from this section]Section: Bias and Fairness (an example)
# Section 4: Bias and Fairness Findings
## 4.1 Overview
This section presents the findings from the bias and fairness assessment of
the AI Knowledge Assistant. Six checklist items were reviewed, covering
the identification of protected attributes, demographic parity, equalized
odds (where applicable), mitigation, and monitoring.
## 4.2 Status Summary
| Item | Description | Status | Critical |
|------|-------------|--------|----------|
| 1.1 | Protected attributes identified | ✅ Yes | Yes |
| 1.2 | Demographic parity measured | ❌ No | Yes |
| 1.3 | Equalized odds tested | N/A | No |
| 1.4 | Disparities mitigated | ✅ Yes (partial) | Yes |
| 1.5 | Mitigation effectiveness verified | ⚠️ Partial | Yes |
| 1.6 | Production bias monitoring | ❌ No | Yes |
Compliant: 1, partial: 2, gaps: 2 (both Critical), N/A: 1.
## 4.3 Detailed Findings
### Compliant items
- **Item 1.1**: The protected attributes are documented (gender, age bucket,
language preference, hierarchical role). A justification is provided.
Recently reviewed by the Tech Lead. ✅
- **Item 1.4**: Disparities were found in the last audit (December 2025) for
language preference (Spanish vs. English DPR 0.78). A mitigation was
implemented: retraining with a balanced data sample. Post-mitigation,
the language DPR rose to 0.91. ✅ However, the mitigation was only applied to
language; the other protected attributes weren't audited or mitigated.
### Item 1.2: Demographic parity measured — GAP
**Current state**: The last comprehensive bias audit was conducted on December 15,
2025 — five months ago. The audit covered gender and language preference only.
Not covered: age buckets, hierarchical role.
**Required state**: A bias audit within the last 90 days covering all
identified protected attributes.
**Gap**:
- 60 days overdue
- 2 of 4 protected attributes never audited
**Evidence reviewed**:
- ethics-audit/bias-2025-12-15.md (the last full audit)
- Code in the bias_audit/ folder (the toolkit is implemented)
- The production bias dashboard: not implemented
**Root cause**: The bias audit wasn't added to the quarterly cadence; the team has
been firefighting other priorities.
**Impact**:
- We can't detect bias drift since December 2025
- Untested protected attributes may have severe undetected disparities
- Risk: discriminatory outcomes affecting users; legal liability
under anti-discrimination laws
**Recommendation**:
1. Conduct a fresh bias audit covering ALL 4 protected attributes within
2 weeks (by 2026-05-25)
2. Add the bias audit to the quarterly cadence permanently
3. Schedule the next bias audit for 2026-08-15
**Priority**: HIGH
**Owner**: ML Engineer (Sarah)
**Estimated effort**: 8-10 hours over 2 weeks
**Target date for "Yes"**: 2026-05-25
### Item 1.6: Production bias monitoring — GAP
**Current state**: No production monitoring of bias metrics. System
behavior post-deployment isn't segmented by protected attribute.
**Required state**: Real-time monitoring with alerts when bias metrics
exceed the thresholds.
**Gap**: The production system has no bias visibility post-deployment.
**Evidence reviewed**:
- Datadog dashboards: aggregate metrics only, no segmentation
- Code: no protected_attr tagging in metrics emission
**Root cause**: Bias was treated as a "training-time concern," not extended
to production observability. An implicit assumption that mitigation at
training time is sufficient.
**Impact**:
- We can't detect bias drift
- We can't detect performance degradation per group
- We can't trigger a response when something goes wrong
**Recommendation**:
1. Add demographic tags to the metrics emission (1 week of effort)
2. Build a Datadog dashboard with per-group metrics (3 days)
3. Configure alerts: DPR < 0.80, latency disparity > 50% (1 day)
4. Validation period: 2 weeks of monitoring before declaring it complete
**Priority**: HIGH
**Owner**: ML Engineer + DevOps
**Estimated effort**: 2-3 weeks
**Target date for "Yes"**: 2026-06-15
### Item 1.5: Mitigation effectiveness verified — PARTIAL
**Current state**: The language bias mitigation has pre/post metrics.
The gender mitigation never had a pre/post measurement (the mitigation was applied
but not measured).
**Required state**: For each mitigation, a demonstrated improvement with
a pre/post measurement.
**Gap**: 1 mitigation lacks verification.
**Recommendation**:
- Apply the original eval set to the current model (post-gender-mitigation)
- Compare it to the pre-mitigation baseline if available
- Document the findings or accept the gap with reasoning
**Priority**: MEDIUM
**Owner**: ML Engineer
**Estimated effort**: 2-3 days
**Target date**: 2026-06-30
## 4.4 Patterns and themes
**Pattern 1**: Bias was treated as a one-time concern during development;
it wasn't extended to ongoing operations. This shows up in:
- The outdated comprehensive audit
- The lack of production monitoring
- Incomplete mitigation verification
**Root cause**: Cultural — bias work is seen as "done" after the first audit.
**Recommendation pattern**: Embed bias work into the operational rhythms
(a quarterly cadence, production observability, monitoring as code).
## 4.5 Recommendations summary
| ID | Recommendation | Priority | Effort | Target |
|----|----------------|----------|--------|--------|
| BF-1 | Refresh the bias audit (all attributes) | HIGH | 8-10h | 2026-05-25 |
| BF-2 | Implement production monitoring | HIGH | 15-20h | 2026-06-15 |
| BF-3 | A quarterly bias cadence | MEDIUM | 1h/sprint | Ongoing |
| BF-4 | Verify the gender mitigation | MEDIUM | 2-3 days | 2026-06-30 |Section: Privacy and Data Protection (a similar pattern)
# Section 5: Privacy and Data Protection Findings
## 5.1 Overview
[A similar 1-paragraph overview]
## 5.2 Status Summary
| Item | Description | Status | Critical |
|------|-------------|--------|----------|
| 2.1 | Data inventory | ✅ Yes | Yes |
| 2.2 | Data minimization | ⚠️ Partial | Yes |
| 2.3 | Anonymization | ✅ Yes | No |
| 2.4 | Tenant isolation | ✅ Yes | Yes |
| 2.5 | Retention policy | ❌ No | Yes |
| 2.6 | Encryption | ✅ Yes | Yes |
Compliant: 4, partial: 1, gap: 1, all Critical.
## 5.3 Detailed Findings
### Item 2.5: Retention policy — GAP
**Current state**: A retention policy is documented in the privacy policy but
it isn't automatically enforced. A manual deletion process exists but is
ad-hoc.
**Required state**: Documented retention periods automatically
enforced via TTL or scheduled deletion.
**Gap**: Retention isn't enforced systematically.
**Evidence reviewed**:
- The privacy policy: it states "retention 24 months"
- The database: no TTL on the records
- No scheduled deletion jobs
**Root cause**: The retention policy was created for legal compliance but
the implementation was deferred.
**Impact**:
- A GDPR Art. 5(1)(e) violation (storage limitation)
- The risk of data accumulating indefinitely
- Increased breach impact (more data exposed)
**Recommendation**:
1. Implement a scheduled deletion job (1 week)
2. Add TTL where applicable (1-2 days)
3. Test the retention enforcement (1 week of validation)
**Priority**: HIGH (regulatory)
**Owner**: Backend Engineer + DPO
**Estimated effort**: 2-3 weeks
**Target date**: 2026-06-15Common traps in the findings sections
Trap 1 — Vague descriptions. "Bias somewhat addressed" → useless. Be specific: "DPR 0.92 for gender, 0.71 for age 65+."
Trap 2 — No actionable recommendations. "Need better monitoring" → vague. Actionable: "Implement a Datadog dashboard with X metrics, alerts at Y thresholds."
Trap 3 — Lists, no analysis. Mechanically listing items without synthesizing patterns or root causes. The analysis is the value.
Trap 4 — Hiding compliance gaps. "The item passes 'with caveats'" instead of a clear "No." Be honest. Real auditors see through it.
Trap 5 — No effort estimates. "Critical, needs fixing" with no effort estimate. The stakeholder can't plan. Always estimate (even roughly).
Exercise
For your system:
- Take your M2 deliverables and the Bias section of the checklist
- For each item in your Bias section, write the finding (status + detailed if it's a No)
- Do the same for the Privacy section
- Identify the patterns
Summary
You learned:
- ✅ The standard structure of a findings section
- ✅ How to write a detailed finding for a gap (the item template)
- ✅ How to identify patterns and root causes
- ✅ How to provide actionable recommendations with effort estimates
- ✅ The traps: vagueness, list-only, hiding gaps
Checkpoint: if your findings are specific, traceable, and actionable, you're set.
Next capsule
04 — Findings: Transparency + Compliance. We continue with the next sections of the report.