Módulo 8: Proyecto Integrador — Ethics Audit of an AI System

Audit Methodology

Siguiente →

Descripción

Cómo conducir el audit step-by-step. Esta es la metodología que vas a aplicar al sistema elegido. Estructurada, replicable, defendible.

Al terminar vas a poder:

  • Planificar el audit (scope, timeline, resources)
  • Ejecutar items del checklist sistemáticamente
  • Documentar findings con evidencia
  • Validar que audit es completo y defendible

Las 5 fases del audit

1. Preparation (1-3 days)
   - Define scope
   - Gather artifacts
   - Schedule reviewers

2. Execution (3-7 days)
   - Apply checklist item by item
   - Gather evidence
   - Document findings

3. Analysis (1-3 days)
   - Synthesize findings
   - Identify gaps
   - Prioritize

4. Reporting (2-5 days)
   - Write report
   - Generate recommendations
   - Prepare presentations

5. Sign-off and Communication (1-2 days)
   - Stakeholder review
   - Action plan finalization
   - Schedule follow-up

Total: 1-3 weeks for thorough audit. Plus future follow-ups.

Phase 1: Preparation

Define scope

# Audit Scope Definition

## System under audit
- **Name**: AI-Powered Knowledge Assistant v1.5
- **Description**: B2B SaaS multi-tenant AI assistant
- **Deployed**: Production (5 tenants)
- **Last audited**: Never (this is first comprehensive audit)

## Scope inclusions
- All AI components (RAG, agent, LLM integration)
- All data processing (training + inference)
- Multi-tenant architecture
- User-facing features

## Scope exclusions (with rationale)
- Underlying infrastructure (handled by separate IT audit)
- Standard CI/CD pipeline (covered by SOC2 work)
- Marketing/sales tools (not AI)

## Audit period
- **Reference timeframe**: Last 12 months of operation
- **Audit duration**: 2 weeks (this audit)
- **Next audit**: 6 months

## Auditor team
- **Lead auditor**: Tech Lead (Mike)
- **Supporting**: ML Engineer, DPO consult
- **External advisor**: None (this iteration)

## Standards reference
- Responsible AI Framework v1.0 (M7 deliverable)
- EU AI Act
- GDPR
- NIST AI RMF

Gather artifacts

Collect documents you'll need:

# Required artifacts (collect before starting)

## From path deliverables
- [ ] M2: Bias Audit Toolkit + last audit results
- [ ] M3: Privacy Assessment document
- [ ] M4: EU AI Act Risk Classification
- [ ] M5: GDPR Compliance Checklist
- [ ] M6: NIST Standards Mapping
- [ ] M7: Responsible AI Framework v1.0

## From system documentation
- [ ] Architecture Design Document (M8 system design)
- [ ] RoPA (Records of Processing Activities)
- [ ] DPIA (Data Protection Impact Assessment)
- [ ] Privacy Policy (current version)
- [ ] User-facing documentation
- [ ] Internal runbooks
- [ ] Incident reports last 12 months

## From operations
- [ ] Monitoring dashboards (last 90 days metrics)
- [ ] Bias metrics last 90 days
- [ ] Error logs + incident logs
- [ ] User feedback aggregated

## From legal/compliance
- [ ] Contracts with processors (OpenAI, Pinecone, etc.)
- [ ] SCCs for international transfers
- [ ] Sign-off records

Schedule reviewers

Block calendar for auditor + interviewees:

  • Tech Lead: lead auditor (10-15 hours)
  • ML Engineer: subject matter expert (4-6 hours)
  • DPO/Legal: privacy + compliance items (2-3 hours)
  • Product: use case impact (2 hours)

Phase 2: Execution

Approach to each item

For each checklist item:

1. Read item carefully
2. Identify what evidence is required
3. Locate evidence in gathered artifacts
4. Verify evidence is:
   - Current (within applicable timeframe)
   - Accurate (matches reality)
   - Complete (all required components)
5. Document:
   - Status: Yes / No / N-A
   - Evidence link/reference
   - Verifier: who confirmed
   - Date: when verified
   - Notes: contextual details

Example: Item 2.4 (Multi-tenant isolation)

## Item 2.4: Multi-tenant data isolation verified

**Required evidence**: Test results showing tenant isolation;
penetration test report.

**Investigation steps**:
1. Search repo for "tenant_id" filter usage
   - Found in: vector_search.py, db_queries.py, api_router.py
   - Code review: filters present in all query paths ✓
2. Run test suite focusing on multi-tenant tests
   - tests/test_multi_tenant.py: 45 tests, all pass ✓
3. Check if penetration test conducted
   - Last pen test: 2026-02-15 (3 months ago)
   - Report: ethics-audit/2026-pen-test-report.pdf
   - Findings: 0 cross-tenant vulnerabilities found
4. Verify isolation in production
   - Sample queries from 5 tenants, all return only own data ✓

**Status**: ✅ YES
**Evidence**: 
- Code review notes: ethics-audit/code-review-2.4.md
- Test results: tests/test_multi_tenant.py reports
- Pen test report: ethics-audit/2026-pen-test-report.pdf
**Verifier**: Tech Lead (Mike), Engineering manager (Ana)
**Date**: 2026-05-11
**Notes**: Pen test was 3 months ago; well within typical
6-month freshness window. Plan next pen test for 2026-08-15.

Example: Item 1.2 (Demographic parity) — finding a gap

## Item 1.2: Demographic parity measured

**Required evidence**: Bias audit report within 90 days; methodology;
results per group.

**Investigation steps**:
1. Locate bias audit reports
   - Most recent: 2025-12-15 (5 months ago — OUTDATED)
   - Audit covered: gender, language
   - NOT covered: age buckets, hierarchical role
2. Verify whether monitoring continues
   - Production monitoring of bias: NO
   - No alerts configured for bias metrics

**Status**: ❌ NO
**Evidence**: 
- Last bias audit: ethics-audit/bias-2025-12-15.md (outdated)
- No production monitoring evidence
**Verifier**: Tech Lead (Mike)
**Date**: 2026-05-11
**Notes**: 
- Last comprehensive audit 5 months ago — outside acceptable window
- Age buckets and role have never been tested
- Production monitoring not implemented
**Classification**: HIGH priority gap
**Recommended action**: Conduct fresh bias audit covering all
protected attributes within next 2 weeks. Implement production
monitoring within sprint.
**Target date for "Yes" status**: 2026-06-15

Honest gap documentation. This is the value of audit.

Phase 3: Analysis

Synthesize findings

# Audit Findings Synthesis

## Status Summary
- Total items: 28
- Yes (compliant): 22 (79%)
- No (gap identified): 4 (14%)
- N/A (justified): 2 (7%)

## Findings by Severity
- Critical (block production): 0
- High (urgent fix): 3
  - Item 1.2 (bias measurement outdated)
  - Item 1.6 (production bias monitoring absent)
  - Item 4.6 (international transfer safeguards)
- Medium (next quarter): 4
- Low (planned improvement): 2

## Findings by Theme

### Theme 1: Bias monitoring gaps
- Items 1.2, 1.6 show pattern of insufficient ongoing measurement
- Root cause: implemented baseline but no production monitoring
- Recommendation: Phase-2 monitoring implementation

### Theme 2: Privacy compliance partially addressed
- Items 4.4-4.6 show GDPR articles partially implemented
- Root cause: focused on most visible (Art. 22), neglected internal
- Recommendation: complete GDPR compliance sweep

### Theme 3: Documentation lag
- Items in Section 5 show docs not kept current
- Root cause: documentation considered overhead, not value
- Recommendation: docs-as-code culture, CI checks

Identify root causes

Don't just list gaps — analyze why:

  • Resource constraints?
  • Knowledge gaps?
  • Process gaps?
  • Tooling gaps?
  • Culture gaps?

Understanding root cause makes recommendations more effective.

Phase 4: Reporting

Will cover detail en M8-03 a M8-07. Mental model:

  • Technical version: deep detail, evidence links
  • Executive version: high-level, decision-oriented

Phase 5: Sign-off

# Audit Sign-off

## Audit completion confirmation
- All applicable items reviewed: ✅
- Evidence collected and documented: ✅
- Findings synthesized: ✅
- Recommendations prioritized: ✅

## Stakeholder sign-offs

### Tech Lead (Mike)
"I have conducted this audit with the methodology described.
Findings accurately reflect current state."
Signature: __________ Date: __________

### CTO (or designate)
"I have reviewed findings and approved action plan."
Signature: __________ Date: __________

### DPO (if applicable)
"Privacy and compliance findings are accurate."
Signature: __________ Date: __________

## Next steps
- Action plan implementation begins: [date]
- Next periodic audit: [6 months]
- Critical items: tracked weekly

Trampas comunes en methodology

Trampa 1 — Rushing through items. "All Yes" para finish quickly. Not real audit. Take time.

Trampa 2 — Sin evidence trail. Status "Yes" pero no documentás evidence. Can't defend in re-audit.

Trampa 3 — Audit by single person sin verification. One person checks everything. Hidden bias. Have someone else verify random samples.

Trampa 4 — No tracking gaps to closure. Identified gaps in audit, but action plan dies. Tracking system needed.

Trampa 5 — Treat outdated as N/A. "Bias audit 6 months ago" → "N/A" instead of "outdated". Be honest.

Ejercicio

Para tu sistema:

  1. Drafteá scope document
  2. Lista artifacts needed (with checkboxes)
  3. Schedule reviewers (calendar block)
  4. Plan timeline (1-3 weeks)

Resumen

Aprendiste:

  • ✅ 5 phases: prep, execution, analysis, reporting, sign-off
  • ✅ Each phase has specific outputs and time investment
  • ✅ Investigation > checking (gather real evidence)
  • ✅ Honest gap documentation expected and valuable
  • ✅ Root cause analysis beyond surface findings
  • ✅ Trampas: rushing, no evidence, hidden bias, untracked gaps

Checkpoint: si podés ejecutar audit methodically siguiendo phases, listo para next.

Siguiente cápsula

03 — Findings: Bias + Privacy sections. Cómo structurar findings para these two key sections del report.

Recursos

  1. Algorithmic Audit (Upturn) — methodology.
  2. SAIF — Secure AI Framework — Google's approach.
  3. Anthropic's RSP audits — frontier AI auditing.
  4. Internal Auditing Standards (IIA) — general audit principles.
← AnteriorIntroducción al Ethics Audit ProjectSiguiente →Findings: Bias + Privacy sections