Contenido del checklist: bias + fairness

Descripción

Llenamos la primera section del checklist con items específicos derivados de M2 (Bias and Fairness). Esta section típicamente tiene 5-6 items, organizados en sub-categories: detection, measurement, mitigation, and ongoing monitoring.

Al terminar vas a poder:

• Specificar 5-6 items de bias/fairness con full format
• Derivar items directamente de M2 deliverables
• Calibrar evidence requirements

---

Section completa: Bias and Fairness

# Section 1: Bias and Fairness

## Item 1.1: Protected attributes identified

**Question**: Have the protected demographic attributes relevant to this
system been explicitly identified and documented?

**Why this matters**: Cannot test bias for groups you haven't defined.
Common protected attributes vary by jurisdiction (US: race, gender,
age 40+; EU: also includes religion, sexual orientation).

**Evidence required**:
- Document listing protected attributes for this system
- Justification for selection
- Legal review if applicable

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**:
**Verified by**:
**Critical**: Yes

---

## Item 1.2: Demographic parity measured

**Question**: Has demographic parity been calculated for all protected
attributes within the last 90 days, with results documented?

**Why this matters**: Demographic parity disparities indicate potential
discriminatory impact. Untested = unknown risk.

**Evidence required**:
- Bias audit report (from M2 deliverable) date < 90 days
- Calculation methodology documented (parity ratio per group)
- Results per group with ratios

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**:
**Verified by**:
**Critical**: Yes

---

## Item 1.3: Equalized odds tested

**Question**: Has equalized odds (true positive rate parity, false positive
rate parity) been tested for protected groups?

**Why this matters**: Demographic parity alone insufficient. Some
applications (criminal justice, lending) require equalized odds. Knowing
both gives full picture.

**Evidence required**:
- TPR per protected group calculated
- FPR per protected group calculated
- Gaps documented; thresholds defined

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**:
**Verified by**:
**Critical**: Yes for high-impact decisions, No otherwise

---

## Item 1.4: Disparities mitigated or documented

**Question**: For any disparity exceeding threshold (e.g., DPR < 0.85 or
EOR gap > 0.1), is mitigation implemented OR justified acceptance
documented with stakeholder review?

**Why this matters**: Discovering bias is half the work. Acting on it is
the other half. Untreated bias = liability.

**Evidence required**:
- Mitigation actions taken (with before/after metrics)
- OR justified acceptance with reasoning
- Stakeholder sign-off if disparity accepted

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**:
**Verified by**:
**Critical**: Yes

---

## Item 1.5: Bias mitigation effectiveness verified

**Question**: For each mitigation applied, is its effectiveness verified
with pre/post measurement showing improvement?

**Why this matters**: Mitigation that doesn't work is theater. Must
demonstrate actual reduction in disparity.

**Evidence required**:
- Pre-mitigation metrics
- Post-mitigation metrics
- Quantified improvement

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**:
**Verified by**:
**Critical**: Yes if mitigations applied

---

## Item 1.6: Ongoing bias monitoring in production

**Question**: Are bias metrics monitored continuously in production with
alerts when thresholds exceeded?

**Why this matters**: Models drift. Inputs change. Bias can emerge
post-deployment. Need ongoing detection, not one-time test.

**Evidence required**:
- Production dashboard showing bias metrics
- Alert configuration (thresholds, recipients)
- Last 90 days of metric history

**Status**: [ ] Yes [ ] No [ ] N/A
**Date verified**:
**Verified by**:
**Critical**: Yes for systems in production

---

Adaptaciones por tipo de sistema

Si tu sistema no toma decisiones (e.g., generative chat sin recommendations)

• Item 1.1 still applies (groups exist regardless)
• Item 1.2 demographic parity: less directly applicable. Adapt to "response quality parity"
• Item 1.3 equalized odds: typically N/A
• Item 1.4 disparities: applies if quality varies
• Item 1.5: applies if mitigations implemented
• Item 1.6 monitoring: monitor response quality per group

Si tu sistema es clasificación binaria (e.g., fraud detection)

• All items apply standard
• Item 1.3 critical (TPR/FPR para fairness in security context)

Si tu sistema genera ranked outputs (e.g., search, recommendations)

• Item 1.2 adapted: representation of groups in top-K results
• Item 1.3 adapted: differential exposure rates
• Item 1.6: monitor representation over time

Principle: items deben ser interpretados en contexto de tu sistema, no aplicados rígidamente.

---

Trampas comunes

Trampa 1 — Asumir que "no protected attributes" elimina bias risk. "My system doesn't use race or gender." But it may use proxies (zipcode, name, etc.) that correlate. Identifying explicit attrs is start, not end.

Trampa 2 — Bias testing una vez al deployment. Testing in dev, never in production. Production data different, model drifts. Re-test periodically.

Trampa 3 — Aceptar disparities sin justification. "DPR 0.7, acceptable because business reasons" → unjustified. Either improve or document explicit rationale with stakeholder buy-in.

Trampa 4 — Olvidar mitigation effectiveness. "Mitigated by adding fairness constraints." Did it work? Need before/after measurement.

Trampa 5 — Bias-blind monitoring. Production monitoring of accuracy but not bias. Performance can be great average + terrible per group. Monitor segmented.

---

Ejercicio

Para tu sistema:

1. Identificá los protected attributes relevantes (los 3-5 más importantes)
2. Para each item arriba, status (Yes/No/N/A) actual
3. Para cada "No" o "N/A", explain why
4. Action plan para los "No"

Ver solución (ejemplo Knowledge Assistant)

Protected attrs relevantes:

• Gender (binary + non-binary categorization)
• Age bucket (18-30, 31-50, 51+)
• Language preference (Spanish, English, others)
• Hierarchical role (executive, mid-level, individual contributor — may correlate with privilege)

Item status:

• 1.1: ✅ Identified above
• 1.2: ⚠️ Partially — language parity tested, others pending
• 1.3: N/A — system doesn't binary classify
• 1.4: N/A pending 1.2 completion
• 1.5: N/A
• 1.6: ❌ Production monitoring of quality NOT segmented by group yet

Action plan:

• Q3: Complete 1.2 for all attrs (4-week effort)
• Q3: Implement 1.6 segmented dashboard (2-week effort)
• Q4: Re-evaluate based on Q3 findings

---

Resumen

Aprendiste:

• ✅ 6 items específicos de bias/fairness section
• ✅ Each item: question, why, evidence, status, critical flag
• ✅ Adaptations por tipo de sistema
• ✅ Trampas: proxies, one-time testing, unjustified acceptance

Checkpoint: si tu sistema tiene bias section custom-tailored con 5-6 items, estás listo para next section.

---

Siguiente cápsula

04 — Privacy + GDPR + EU AI Act sections. Continuamos llenando sections derived from M3-M5.

---

Recursos

1. Bias Audit Toolkit (M2 deliverable).
2. Fairness Indicators (Google).
3. Aequitas reports.
4. What-If Tool — interactive bias exploration.