4. Bias Detection: Slicing, A/B Testing and Counterfactual Testing

Capsule description

Capsules 02 and 03 gave you the metrics: demographic parity, equalized odds, calibration. Now comes the practical question: how do you apply them to your system in the real world?

This capsule covers three concrete techniques for detecting bias in practice:

1. Subgroup slicing — breaking predictions down by demographic attribute and comparing performance.
2. Demographic A/B testing — comparing model versions from each group's perspective.
3. Counterfactual testing — generating pairs of inputs that differ only in the protected attribute and measuring the output delta.

The three are complementary. Slicing is the simplest when you have demographic data. Counterfactual is your lifeline when you do NOT have demographic data (common in production). Demographic A/B testing is for validating mitigations.

By the end, you'll be able to wire bias detection in as a CI gate in your pipeline, not just as a one-time audit.

Technique 1: Subgroup slicing

The concept

Take your test set, split it by protected attribute, and compute the metrics per subset.

Full test set → 95% accuracy

Split by gender:
  Group M → 97% accuracy
  Group F → 92% accuracy
  Difference: 5 points.

A 5-point difference across groups is a bias signal.

Why slicing detects what global accuracy hides

Global accuracy is weighted: if group M is 80% of the dataset and scores 97%, while group F is 20% and scores 92%, global accuracy = 0.96 (sounds fine).

But the individual experience of a group F user is 92%, not 96%. Slicing exposes this.

A basic implementation

import pandas as pd
from sklearn.metrics import accuracy_score, precision_score, recall_score

def slice_by_subgroup(predictions, ground_truth, groups, attribute_name=''):
    """
    Slice performance by subgroup.
    """
    df = pd.DataFrame({
        'pred': predictions,
        'true': ground_truth,
        'group': groups,
    })
    
    results = {}
    for group_name, group_df in df.groupby('group'):
        results[group_name] = {
            'n': len(group_df),
            'accuracy': accuracy_score(group_df['true'], group_df['pred']),
            'precision': precision_score(group_df['true'], group_df['pred'], zero_division=0),
            'recall': recall_score(group_df['true'], group_df['pred'], zero_division=0),
            'positive_rate': (group_df['pred'] == 1).mean(),
        }
    
    # Differences
    accuracies = [r['accuracy'] for r in results.values()]
    
    return {
        'attribute': attribute_name,
        'metrics_per_group': results,
        'accuracy_difference': max(accuracies) - min(accuracies),
        'concerning': max(accuracies) - min(accuracies) > 0.05,
    }

Intersectional slicing

Single-attribute slicing can hide intersectional problems:

# Metrics by gender and by race separately:
# - Men: 95% accuracy
# - Women: 92% accuracy  
# - White people: 95% accuracy
# - Black people: 92% accuracy

# But...
# - White men: 97%
# - Black women: 84%   ← HIDDEN problem

Solution: intersectional slicing by combinations of attributes:

def slice_intersectional(df, attributes):
    """
    Slice by combinations of attributes.
    
    Args:
        df: DataFrame with 'pred', 'true', and the attributes.
        attributes: list of columns to combine (e.g., ['gender', 'race']).
    """
    grouped = df.groupby(attributes)
    
    results = {}
    for combo, group_df in grouped:
        if len(group_df) < 30:
            continue  # Skip subgroups with an insufficient sample
        
        results[combo] = {
            'n': len(group_df),
            'accuracy': accuracy_score(group_df['true'], group_df['pred']),
            'positive_rate': (group_df['pred'] == 1).mean(),
        }
    
    return results

Trade-off: more dimensions = more subgroups = a smaller sample size per subgroup. If your dataset is small, intersectional slicing may not be viable.

Visualization: a heatmap

To present intersectional results, a heatmap works well:

import seaborn as sns
import matplotlib.pyplot as plt

def heatmap_subgroup_metric(df, row_attr, col_attr, metric='accuracy'):
    """
    Heatmap of a metric by attribute combination.
    """
    pivot = df.groupby([row_attr, col_attr]).apply(
        lambda x: accuracy_score(x['true'], x['pred'])
    ).unstack()
    
    fig, ax = plt.subplots(figsize=(10, 6))
    sns.heatmap(pivot, annot=True, fmt='.2%', cmap='RdYlGn', vmin=0.8, vmax=1.0)
    ax.set_title(f'{metric} by {row_attr} × {col_attr}')
    return fig

A visual output where red cells = under-served subgroups. Useful for presenting to non-technical stakeholders.

Technique 2: Demographic A/B testing

The concept

Once you make a change to the model (retrain, mitigation), you want to know: did fairness improve without destroying accuracy?

Demographic A/B testing:

1. Take the old model's (A) and the new model's (B) predictions on the same test set.
2. Compute the fairness metrics for each.
3. Compare: is B better on fairness? With no significant accuracy loss?

Implementation

def ab_test_fairness(predictions_a, predictions_b, ground_truth, groups,
                    fairness_metric_func):
    """
    Compare two model versions on fairness and accuracy.
    
    Args:
        predictions_a: the old model's predictions.
        predictions_b: the new model's predictions.
        ground_truth: the true labels.
        groups: the protected attributes.
        fairness_metric_func: a function taking (pred, ground_truth, groups) and returning a dict with the metric.
    """
    metric_a = fairness_metric_func(predictions_a, ground_truth, groups)
    metric_b = fairness_metric_func(predictions_b, ground_truth, groups)
    
    accuracy_a = accuracy_score(ground_truth, predictions_a)
    accuracy_b = accuracy_score(ground_truth, predictions_b)
    
    return {
        'model_a': {'metric': metric_a, 'accuracy': accuracy_a},
        'model_b': {'metric': metric_b, 'accuracy': accuracy_b},
        'fairness_improvement': (
            metric_b.get('parity_ratio', 0) - metric_a.get('parity_ratio', 0)
        ),
        'accuracy_change': accuracy_b - accuracy_a,
    }

The decision matrix

Important: testing in production can be a problem

Traditional A/B testing means giving the new model to a % of real users. If the new model is biased, that % of real users suffers the bias.

For fairness specifically, A/B testing should be done on an offline test set, NOT in real production with users. Afterward, if the offline test passes, deploy gradually with strict monitoring.

Technique 3: Counterfactual testing

The problem it solves

Slicing requires demographic data (gender, race, etc.) in your test set. In many contexts:

• You don't collect it (doesn't apply).
• Collecting it is illegal (Europe).
• Users don't want to provide it.
• It's only inferable with error.

What do you do with no demographic data? Counterfactual testing.

The concept

Generate pairs of inputs that differ only in markers associated with the protected attribute. Run both through the model. If the outputs differ, there's causal bias.

Input A: "John Smith, captain of chess club"
Input B: "Mary Smith, captain of women's chess club"

Difference: gender markers.

If the model gives a very different score to A vs. B, that's causal bias.

Implementation

def counterfactual_test(model, inputs, gender_swap_function, 
                        threshold=0.05):
    """
    A counterfactual gender test (generalizable to other attributes).
    
    Args:
        model: a callable that takes an input and returns a score.
        inputs: a list of original inputs.
        gender_swap_function: a function taking an input and returning a gender-swapped version.
        threshold: the max acceptable delta.
    
    Returns:
        dict with results.
    """
    results = []
    
    for input_original in inputs:
        score_original = model(input_original)
        
        input_swapped = gender_swap_function(input_original)
        score_swapped = model(input_swapped)
        
        delta = abs(score_original - score_swapped)
        
        results.append({
            'original': input_original[:80],
            'swapped': input_swapped[:80],
            'score_original': score_original,
            'score_swapped': score_swapped,
            'delta': delta,
            'fair': delta < threshold,
        })
    
    df = pd.DataFrame(results)
    
    return {
        'all_results': results,
        'mean_delta': df['delta'].mean(),
        'max_delta': df['delta'].max(),
        'pct_fair': df['fair'].mean(),
        'examples_unfair': df[~df['fair']].head(5).to_dict('records'),
    }

Implementing gender_swap_function

This is the hard part. For English text:

def swap_gender_markers(text):
    """
    Simple swap of gender markers in English text.
    Imperfect but useful for detection.
    """
    swaps = {
        'he': 'she', 'she': 'he',
        'his': 'her', 'her': 'his',
        'him': 'her', 'mr.': 'ms.',
        'mr ': 'ms ', 'man': 'woman', 'woman': 'man',
        'men': 'women', 'women': 'men',
        'boy': 'girl', 'girl': 'boy',
        'father': 'mother', 'mother': 'father',
        'son': 'daughter', 'daughter': 'son',
        'husband': 'wife', 'wife': 'husband',
    }
    
    # Simple substitution; in production use an NLP library
    result = text
    for original, replacement in swaps.items():
        result = result.replace(original, replacement)
    
    # More sophisticated: for names, use a list of gendered names
    # ...
    
    return result

Limitations: a simple string swap is imperfect. For production:

• Use an NLP library (spaCy, NLTK) for correct token handling.
• A curated list of gendered names.
• Pronoun handling across different languages.
• Consider context (a title like "captain of chess team" shouldn't swap "team").

Counterfactual for other attributes

Same principle:

• Race: swap names associated with racial groups (careful: error-prone, do it thoughtfully).
• Age: change the dates on a résumé ("graduated 1985" → "graduated 2015").
• Disability: add/remove a mention of disability.

Each case requires domain knowledge to swap correctly.

Why counterfactual is powerful

• It doesn't require demographic data in the test set.
• It's causal: if the delta is high, it's a direct cause of the swapped marker (not a proxy via another feature).
• It's CI-friendly: runs fast, needs no humans.
• It works with any model: black-box agnostic.

Wiring it into CI/CD

Combine the three techniques into one automated fairness test:

# tests/test_fairness.py
import pytest
from your_model import model
from your_test_data import test_set, counterfactual_inputs

def test_demographic_parity():
    """Blocking: the deploy fails if demographic parity < 0.80."""
    predictions = model.predict(test_set['features'])
    
    for attribute in ['gender', 'race', 'age_bracket']:
        if attribute not in test_set.columns:
            continue
        
        result = demographic_parity(predictions, test_set[attribute])
        
        assert result['meets_4_5_rule'], (
            f"DP violation for {attribute}: ratio={result['parity_ratio']:.3f}"
        )

def test_equalized_odds():
    """Warning if the difference is > 5%."""
    predictions = model.predict(test_set['features'])
    truth = test_set['ground_truth']
    
    for attribute in ['gender', 'race']:
        if attribute not in test_set.columns:
            continue
        
        result = equalized_odds(predictions, truth, test_set[attribute])
        
        assert result['meets_threshold_5pct'], (
            f"EO violation for {attribute}: "
            f"TPR diff={result['tpr_difference']:.3f}, "
            f"FPR diff={result['fpr_difference']:.3f}"
        )

def test_counterfactual_fairness():
    """Counterfactual: the max delta must be <= 0.10."""
    result = counterfactual_test(
        model=model,
        inputs=counterfactual_inputs,
        gender_swap_function=swap_gender_markers,
    )
    
    assert result['max_delta'] < 0.10, (
        f"Counterfactual delta too high: {result['max_delta']:.3f}. "
        f"Examples: {result['examples_unfair'][:3]}"
    )

In your CI:

# .github/workflows/ci.yml
- name: Fairness tests
  run: pytest tests/test_fairness.py -v

If fairness regresses, the deploy fails. If it passes, it continues.

Monitoring in production

Offline tests are necessary but not sufficient. A model in production can drift. Continuous monitoring:

# monitoring/fairness_monitor.py

def monitor_fairness_in_production(predictions_prod, demographic_data_prod):
    """
    Compute fairness metrics on recent production data.
    Alert if the metric degrades.
    """
    metric = demographic_parity(predictions_prod, demographic_data_prod)
    
    log_to_dashboard({
        'timestamp': now(),
        'demographic_parity_ratio': metric['parity_ratio'],
        'rates_by_group': metric['rates_by_group'],
    })
    
    # Alert if it drops below the threshold
    if metric['parity_ratio'] < 0.80:
        send_alert(
            severity='HIGH',
            message=f"Demographic parity dropped to {metric['parity_ratio']:.3f}",
        )
    
    # Alert also if it decays from the baseline
    baseline = get_baseline_parity_ratio()
    if metric['parity_ratio'] < baseline - 0.05:
        send_alert(
            severity='MEDIUM',
            message=f"Parity ratio decayed: {baseline:.3f} → {metric['parity_ratio']:.3f}",
        )

Run it weekly or more often. Make the dashboard accessible to stakeholders.

Traps and common mistakes

1. Slicing by a single attribute only

It hid intersectional problems (Black women in facial recognition). Do intersectional slicing whenever the data allows.

2. Insufficient sample sizes

A subgroup with 5 samples = an unstable metric. Filter out subgroups with n < 30.

3. Incomplete counterfactual swaps

Swapping only "he/she" misses names and other references. Use an NLP library for completeness.

4. Fairness tests not in CI

If bias testing is a manual one-time thing, regression is certain. CI is non-negotiable.

5. Forgetting post-deploy monitoring

Offline tests ≠ the reality of production. Drift requires continuous monitoring.

Self-check

# tests/test_fairness.py
def test_counterfactual_gender_fairness():
    inputs = load_test_inputs()
    
    for input_original in inputs:
        score_original = model.predict(input_original)
        
        input_swapped = swap_gender(input_original)
        score_swapped = model.predict(input_swapped)
        
        delta = abs(score_original - score_swapped)
        assert delta < 0.05, f"Delta {delta} too high"

Summary and next step

• Subgroup slicing breaks performance down and exposes gaps that global accuracy hides. Use intersectional slicing when the data allows it.
• Demographic A/B testing validates that mitigations improve fairness without destroying accuracy. Offline before production.
• Counterfactual testing detects causal bias without needing demographic data in the test set. A lifesaver in production.
• Wire it into CI: blocking tests + warnings.
• Monitor in production: drift detection with dashboards + alerts.

Checkpoint: you should be able to choose and apply the right technique for your situation (with/without demographic data, offline/online, single/intersectional).

Bridge to the next capsule: capsule 05 covers the impossibility theorems — the uncomfortable truth that you can't satisfy every metric simultaneously. You'll learn Chouldechova (2017) and Kleinberg-Mullainathan-Raghavan (2017), understand why COMPAS exposed this conflict, and how to consciously choose which metric to prioritize based on context.

Resources

1. What-If Tool — Google — interactive visual slicing.
2. Aequitas — Bias audit toolkit — open source.
3. Counterfactual Fairness (Kusner et al., 2017) — the foundational paper.
4. Slice Finder (Chung et al., 2019) — automating slicing.

Next: 05-impossibility-theorems-tradeoffs.md — Why you can't satisfy every metric simultaneously.

Capsule 04 of 08 — Module 2 — AI Ethics & Compliance Guide