Module 2: Bias and Fairness — Detection, Measurement, Mitigation

8. Mini-project: Bias Audit Toolkit

Next →

You close the module by building a Bias Audit Toolkit — a reusable Python repo with metrics, tests, mitigations, and a dashboard. You'll apply it to future AI systems, refining it as you learn more.

It's the equivalent of the Ethics Impact Analysis (M1/08) but operational: instead of a document, it's code that runs and produces reports.

Deliverable spec

Minimum viable

  • ✅ A Python library with modules for:
    • Fairness metrics (demographic parity, equalized odds, calibration).
    • Bias detection (slicing, counterfactual).
    • Mitigation (a pre-processing baseline).
  • ✅ Unit tests verifying each metric with synthetic datasets.
  • ✅ An audit script that takes a model + dataset and generates a complete report.
  • ✅ A CLI: runnable from the command line.
  • ✅ A README documenting the audit process.

Stretch goals

  • ⭐ Wire it into the CI of a real system: a fairness gate as a pytest test.
  • ⭐ A dashboard in Streamlit/Gradio with visualizations.
  • ⭐ More mitigations: implement adversarial debiasing.
  • ⭐ Automated multi-attribute intersectional slicing.
  • ⭐ Integration with AIF360 or Fairlearn for benchmarking.

Repo structure

bias-audit-toolkit/
├── README.md
├── pyproject.toml
├── src/
│   ├── __init__.py
│   ├── metrics/
│   │   ├── __init__.py
│   │   ├── demographic_parity.py
│   │   ├── equalized_odds.py
│   │   └── calibration.py
│   ├── detection/
│   │   ├── __init__.py
│   │   ├── slicing.py
│   │   └── counterfactual.py
│   ├── mitigation/
│   │   ├── __init__.py
│   │   ├── reweighting.py
│   │   └── threshold_tuning.py
│   ├── audit.py        ← orchestrator
│   └── cli.py          ← entry point
├── tests/
│   ├── test_demographic_parity.py
│   ├── test_equalized_odds.py
│   ├── test_calibration.py
│   ├── test_slicing.py
│   └── test_counterfactual.py
└── examples/
    ├── audit_example.py
    └── data/
        └── synthetic_loan_data.csv

Step-by-step implementation

Step 1: repo setup

mkdir bias-audit-toolkit
cd bias-audit-toolkit
git init
python -m venv .venv
source .venv/bin/activate
pip install pandas numpy scikit-learn matplotlib seaborn pytest

pyproject.toml:

[project]
name = "bias-audit-toolkit"
version = "0.1.0"
description = "Toolkit for auditing bias in AI systems"
dependencies = [
    "pandas",
    "numpy",
    "scikit-learn",
    "matplotlib",
    "seaborn",
]

[project.optional-dependencies]
dev = ["pytest", "ruff", "mypy"]

Step 2: implement the metrics

src/metrics/demographic_parity.py:

import pandas as pd

def demographic_parity(predictions, groups, favorable_outcome=1):
    """
    Compute the demographic parity ratio.
    
    Returns:
        dict with rates per group, the ratio, and meets_4_5_rule.
    """
    df = pd.DataFrame({
        'prediction': predictions,
        'group': groups,
    })
    
    rates = df.groupby('group')['prediction'].apply(
        lambda x: (x == favorable_outcome).mean()
    )
    
    if len(rates) < 2:
        return {'error': 'Need at least 2 groups'}
    
    parity_ratio = rates.min() / rates.max() if rates.max() > 0 else 0
    
    return {
        'rates_by_group': rates.to_dict(),
        'parity_ratio': parity_ratio,
        'meets_4_5_rule': parity_ratio >= 0.80,
    }

src/metrics/equalized_odds.py:

from sklearn.metrics import confusion_matrix
import pandas as pd

def equalized_odds(predictions, ground_truth, groups, favorable_outcome=1):
    """
    Compute TPR and FPR per group.
    """
    df = pd.DataFrame({
        'pred': predictions, 
        'true': ground_truth, 
        'group': groups
    })
    
    metrics_per_group = {}
    for group_name, group_df in df.groupby('group'):
        y_true = (group_df['true'] == favorable_outcome).astype(int)
        y_pred = (group_df['pred'] == favorable_outcome).astype(int)
        
        cm = confusion_matrix(y_true, y_pred, labels=[0, 1])
        tn, fp, fn, tp = cm.ravel()
        
        tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
        fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
        
        metrics_per_group[group_name] = {
            'TPR': tpr,
            'FPR': fpr,
            'n': len(group_df),
        }
    
    tprs = [m['TPR'] for m in metrics_per_group.values()]
    fprs = [m['FPR'] for m in metrics_per_group.values()]
    
    return {
        'metrics_per_group': metrics_per_group,
        'tpr_difference': max(tprs) - min(tprs) if tprs else 0,
        'fpr_difference': max(fprs) - min(fprs) if fprs else 0,
        'meets_threshold_5pct': (
            (max(tprs) - min(tprs) < 0.05 if tprs else True) and
            (max(fprs) - min(fprs) < 0.05 if fprs else True)
        ),
    }

(continue with a similar calibration.py)

Step 3: implement detection

src/detection/slicing.py:

import pandas as pd
from sklearn.metrics import accuracy_score, precision_score, recall_score

def slice_by_subgroup(predictions, ground_truth, groups, attribute_name='',
                      min_samples_per_group=30):
    """
    Slice by demographic subgroup.
    """
    df = pd.DataFrame({
        'pred': predictions,
        'true': ground_truth,
        'group': groups,
    })
    
    results = {}
    for group_name, group_df in df.groupby('group'):
        if len(group_df) < min_samples_per_group:
            continue
        
        results[group_name] = {
            'n': len(group_df),
            'accuracy': accuracy_score(group_df['true'], group_df['pred']),
            'precision': precision_score(
                group_df['true'], group_df['pred'], zero_division=0
            ),
            'recall': recall_score(
                group_df['true'], group_df['pred'], zero_division=0
            ),
            'positive_rate': (group_df['pred'] == 1).mean(),
        }
    
    accuracies = [r['accuracy'] for r in results.values()]
    
    return {
        'attribute': attribute_name,
        'metrics_per_group': results,
        'accuracy_difference': max(accuracies) - min(accuracies) if accuracies else 0,
    }

src/detection/counterfactual.py:

def counterfactual_test(model, inputs, swap_function, threshold=0.05):
    """
    A generic counterfactual test. swap_function defines how to transform inputs.
    """
    results = []
    for input_original in inputs:
        score_original = model(input_original)
        input_swapped = swap_function(input_original)
        score_swapped = model(input_swapped)
        
        delta = abs(score_original - score_swapped)
        
        results.append({
            'original': str(input_original)[:80],
            'swapped': str(input_swapped)[:80],
            'score_original': score_original,
            'score_swapped': score_swapped,
            'delta': delta,
            'fair': delta < threshold,
        })
    
    df = pd.DataFrame(results)
    
    return {
        'mean_delta': df['delta'].mean(),
        'max_delta': df['delta'].max(),
        'pct_fair': df['fair'].mean(),
        'examples_unfair': df[~df['fair']].head(5).to_dict('records'),
    }

Step 4: implement mitigation

src/mitigation/reweighting.py:

import pandas as pd

def compute_sample_weights(df, group_column):
    """
    Compute weights inversely proportional to the group's frequency.
    """
    group_counts = df[group_column].value_counts()
    total = len(df)
    n_groups = len(group_counts)
    
    weights = df[group_column].apply(
        lambda g: total / (n_groups * group_counts[g])
    )
    
    return weights.values

src/mitigation/threshold_tuning.py:

def find_thresholds_for_demographic_parity(scores, groups, target_rate=0.5):
    """
    Find per-group thresholds that satisfy DP.
    """
    df = pd.DataFrame({'score': scores, 'group': groups})
    
    thresholds = {}
    for group_name, group_df in df.groupby('group'):
        sorted_scores = sorted(group_df['score'].values, reverse=True)
        idx = int(target_rate * len(sorted_scores))
        thresholds[group_name] = sorted_scores[idx] if idx < len(sorted_scores) else 0
    
    return thresholds


def apply_per_group_threshold(scores, groups, thresholds, default=0.5):
    """Apply per-group thresholds."""
    return [
        1 if s > thresholds.get(g, default) else 0
        for s, g in zip(scores, groups)
    ]

Step 5: the orchestrator

src/audit.py:

import pandas as pd
from .metrics.demographic_parity import demographic_parity
from .metrics.equalized_odds import equalized_odds
from .detection.slicing import slice_by_subgroup

def audit_model(predictions, ground_truth, groups_dict, output_path=None):
    """
    A complete audit of a model.
    
    Args:
        predictions: array of the model's predictions.
        ground_truth: the real labels.
        groups_dict: dict {attribute_name: array of group labels}.
        output_path: path to save the report (optional).
    
    Returns:
        dict with all the results.
    """
    report = {'attributes': {}}
    
    for attr_name, groups in groups_dict.items():
        attr_report = {
            'demographic_parity': demographic_parity(predictions, groups),
            'equalized_odds': equalized_odds(predictions, ground_truth, groups),
            'slicing': slice_by_subgroup(predictions, ground_truth, groups,
                                          attribute_name=attr_name),
        }
        report['attributes'][attr_name] = attr_report
    
    report['summary'] = generate_summary(report)
    
    if output_path:
        save_report(report, output_path)
    
    return report


def generate_summary(report):
    """Generate an executive summary."""
    issues = []
    for attr_name, attr_report in report['attributes'].items():
        dp = attr_report['demographic_parity']
        if not dp.get('meets_4_5_rule', True):
            issues.append(
                f"{attr_name}: DP ratio {dp['parity_ratio']:.3f} fails 4/5 rule"
            )
        
        eo = attr_report['equalized_odds']
        if not eo.get('meets_threshold_5pct', True):
            issues.append(
                f"{attr_name}: EO violation (TPR diff {eo['tpr_difference']:.3f}, "
                f"FPR diff {eo['fpr_difference']:.3f})"
            )
    
    return {
        'total_issues': len(issues),
        'issues': issues,
        'overall_pass': len(issues) == 0,
    }


def save_report(report, output_path):
    """Save the report as JSON or markdown."""
    import json
    with open(output_path, 'w') as f:
        json.dump(report, f, indent=2, default=str)

Step 6: the CLI

src/cli.py:

import argparse
import pandas as pd
import joblib
from .audit import audit_model

def main():
    parser = argparse.ArgumentParser(description='Bias audit toolkit')
    parser.add_argument('--model', required=True, help='Path to pickled model')
    parser.add_argument('--data', required=True, help='Path to CSV data')
    parser.add_argument('--target', required=True, help='Target column')
    parser.add_argument('--protected', required=True, nargs='+',
                        help='Protected attribute columns')
    parser.add_argument('--output', default='audit_report.json',
                        help='Output path')
    
    args = parser.parse_args()
    
    # Load
    model = joblib.load(args.model)
    df = pd.read_csv(args.data)
    
    # Predictions
    features = df.drop([args.target] + args.protected, axis=1)
    predictions = model.predict(features)
    
    # Audit
    groups_dict = {attr: df[attr].values for attr in args.protected}
    report = audit_model(
        predictions=predictions,
        ground_truth=df[args.target].values,
        groups_dict=groups_dict,
        output_path=args.output,
    )
    
    print(f"Audit complete. {report['summary']['total_issues']} issues found.")
    print(f"Report saved to {args.output}")
    
    if not report['summary']['overall_pass']:
        for issue in report['summary']['issues']:
            print(f"  - {issue}")


if __name__ == '__main__':
    main()

Usage from the command line:

python -m src.cli \
    --model models/loan_model.pkl \
    --data data/test.csv \
    --target approved \
    --protected gender race \
    --output audit_report.json

Step 7: tests

tests/test_demographic_parity.py:

from src.metrics.demographic_parity import demographic_parity

def test_perfect_parity():
    """Same rate per group → ratio = 1.0."""
    predictions = [1, 0, 1, 0, 1, 0, 1, 0]
    groups = ['A', 'A', 'A', 'A', 'B', 'B', 'B', 'B']
    
    result = demographic_parity(predictions, groups)
    
    assert result['parity_ratio'] == 1.0
    assert result['meets_4_5_rule']

def test_disparate_impact():
    """Different rates → ratio < 1.0."""
    predictions = [1, 1, 1, 0] + [0, 0, 0, 1]  # A: 75%, B: 25%
    groups = ['A']*4 + ['B']*4
    
    result = demographic_parity(predictions, groups)
    
    assert result['rates_by_group']['A'] == 0.75
    assert result['rates_by_group']['B'] == 0.25
    assert result['parity_ratio'] == 0.25 / 0.75  # ~0.333
    assert not result['meets_4_5_rule']

def test_single_group():
    """Edge case: only one group."""
    predictions = [1, 0, 1, 0]
    groups = ['A', 'A', 'A', 'A']
    
    result = demographic_parity(predictions, groups)
    
    assert 'error' in result

Run:

pytest tests/ -v

Step 8: the README

README.md:

# Bias Audit Toolkit

A Python toolkit for auditing bias in AI systems.

## Quick start

```bash
pip install -e .

python -m src.cli \
    --model your_model.pkl \
    --data test_data.csv \
    --target outcome \
    --protected gender race

Components

Metrics

  • demographic_parity: 4/5 rule compliance check.
  • equalized_odds: TPR/FPR equality.
  • calibration: score-to-probability mapping.

Detection

  • slicing: performance per subgroup.
  • counterfactual: causal bias testing.

Mitigation

  • reweighting: pre-processing weights.
  • threshold_tuning: post-processing per-group thresholds.

Audit process

  1. Train your model on training data.
  2. Generate predictions on test set.
  3. Run audit_model() with predictions + ground truth + protected attributes.
  4. Review report. Apply mitigations as needed.
  5. Re-audit. Iterate until acceptable.

Integration with CI

Add to .github/workflows/ci.yml:

- name: Bias audit gate
  run: |
    python -m src.cli \
        --model artifacts/model.pkl \
        --data data/holdout.csv \
        --target outcome \
        --protected gender \
        --output reports/audit.json
    python scripts/check_audit_gate.py reports/audit.json

check_audit_gate.py exits 1 if any issues, blocking deploy.


---

## How to apply the toolkit to a real system

### Step 1: identify the protected attributes

Go back to the Ethics Impact Analysis (M1/08). The stakeholders you identified there are the groups to evaluate.

Examples:
- Hiring system: gender, race, age, disability.
- Loan approval: gender, race, age, ZIP-inferred location.
- Healthcare: gender, race, age, insurance status.

### Step 2: prepare the test set with the attributes

Ideally, your test set includes the protected attributes annotated. If not:

- Infer them where legally permissible (ZIP → race with documented error rates).
- Run a demographic sampling study on a subset.
- Counterfactual testing if you can't get them at all.

### Step 3: run the audit

```python
from src.audit import audit_model

predictions = model.predict(test_features)

groups_dict = {
    'gender': test_set['gender'].values,
    'race': test_set['race'].values,
    'age_bracket': test_set['age_bracket'].values,
}

report = audit_model(
    predictions=predictions,
    ground_truth=test_set['target'].values,
    groups_dict=groups_dict,
    output_path='audit_initial.json',
)

Step 4: interpret the results

If report['summary']['overall_pass'] is True: deploy with monitoring.

If not:

  1. Identify the main issues.
  2. Apply the appropriate mitigations.
  3. Retrain the model.
  4. Re-audit.
  5. Iterate until acceptable.

Step 5: document the trade-offs

## Bias Audit Report

**Initial state**:
- Demographic parity (gender): 0.65 (fails 4/5 rule)
- Equalized odds (gender): TPR diff 12%, FPR diff 8%

**Mitigations applied**:
1. Re-weighted training data by gender.
2. Calibration adjustment per group.

**Final state**:
- Demographic parity (gender): 0.85 ✅
- Equalized odds (gender): TPR diff 4%, FPR diff 3% ✅
- Accuracy: 91% (was 93% pre-mitigation, 2% trade-off)

**Decision**: deploy with monitoring. Trade-off acceptable per stakeholder review.

Closing the module

8 capsules

  1. Module introduction.
  2. Demographic parity.
  3. Equalized odds and calibration.
  4. Bias detection: slicing, A/B, counterfactual.
  5. Impossibility theorems.
  6. Pre-processing mitigation.
  7. In/post-processing mitigation.
  8. Mini-project: Bias Audit Toolkit (this capsule).

What you have now

  • Technical vocabulary: 4 fairness metrics, 3 categories of mitigation.
  • From-scratch implementations of every concept.
  • A reusable toolkit you apply to any future system.
  • An understanding of the trade-offs: impossibility theorems, fairness vs. accuracy.
  • CI integration: bias testing as a deploy gate.

What changed in you

Before:

  • "Bias is important."

After:

  • "Bias is measurable with [4 metrics]. My system currently has [X], the legal threshold is [Y], I apply [Z mitigation], I validate with a CI gate, I monitor in production."

Concreteness. Metrics. Informed decisions with a paper trail.

We start in the next module

Module 3: Privacy and Data Protection Fundamentals. The second dimension of impact on people: data protection.

Bias affects groups. Privacy affects individuals. The transition is: "You know how to measure whether your system discriminates → now learn to protect the data of the people your system processes."

Topics: PII, anonymization, k-anonymity, differential privacy, data minimization, retention policies. The compliance fundamentals for GDPR (M4) and CCPA/etc. (M5-6).

Resources

  1. AIF360 (IBM) — full toolkit — comprehensive open source.
  2. Fairlearn (Microsoft) — an alternative.
  3. Aequitas — the Carnegie Mellon toolkit.
  4. Fairness Indicators (Google) — TF integration.
  5. Algorithmic Justice League — research and advocacy.

Capsule 08 of 08 — Module 2 — AI Ethics & Compliance Guide

End of module 2. Continue with module 3 (Privacy and Data Protection Fundamentals).

← Previous7. Mitigation: In-processing and Post-processingNext →1. Module introduction: Privacy and Data Protection Fundamentals