Module 2: Bias and Fairness — Detection, Measurement, Mitigation

7. Mitigation: In-processing and Post-processing

Next →

Capsule description

Pre-processing (capsule 06) modifies the training data. It works when the solution is "more balanced data." But sometimes that isn't enough:

  • Structural bias lives in patterns that persist even with balanced data.
  • You can't modify the training data (you're using an external API, or the dataset is legally fixed).
  • The accuracy trade-off is too high with pre-processing alone.

This capsule covers the two remaining categories:

  1. In-processing: modify the training algorithm. Adversarial debiasing, fairness constraints in the loss function. More complex but often more effective.

  2. Post-processing: modify the trained model's outputs. Threshold tuning per group, calibration adjustment, a reject option. Applicable to any model (including black-box APIs).

You'll learn when to choose each category, how to implement them (with code), and how to combine them with pre-processing for defense in depth.

In-processing: modifying the training

The concept

Instead of balancing the data up front, add fairness to the loss function during training.

Traditional:

loss = error(predictions, ground_truth)

With a fairness constraint:

loss = error(predictions, ground_truth) + λ * fairness_violation(predictions, groups)

λ controls the trade-off: λ=0 = accuracy only. A high λ = more fairness, with a possible cost in accuracy.

Technique 1: Fairness regularization

The simplest idea: add a term to the loss that penalizes disparate predictions across groups.

import torch
import torch.nn as nn

class FairnessRegularizedLoss(nn.Module):
    """
    Traditional loss + fairness regularization.
    """
    def __init__(self, base_loss=nn.BCELoss(), lambda_fair=0.5):
        super().__init__()
        self.base_loss = base_loss
        self.lambda_fair = lambda_fair
    
    def forward(self, predictions, targets, group_labels):
        # Traditional loss
        accuracy_loss = self.base_loss(predictions, targets)
        
        # Fairness term: difference in mean predictions across groups
        unique_groups = torch.unique(group_labels)
        group_means = []
        for g in unique_groups:
            mask = (group_labels == g)
            group_means.append(predictions[mask].mean())
        
        # Penalty: max difference between group means
        fairness_loss = max(group_means) - min(group_means)
        
        return accuracy_loss + self.lambda_fair * fairness_loss

Technique 2: Adversarial Debiasing

A more sophisticated idea: train two models simultaneously:

  • A predictor: predicts the target (e.g., loan approve).
  • An adversary: tries to predict the protected attribute from the predictor's predictions.

The goal: the predictor must achieve high accuracy while the adversary can't recover the protected attribute. If the adversary can't predict gender from the predictor's output, then the output contains no information about gender (= unbiased).

class AdversarialDebiasing:
    """
    A simplified scheme for adversarial debiasing.
    """
    def __init__(self, predictor, adversary, alpha=1.0):
        self.predictor = predictor   # Predicts target
        self.adversary = adversary   # Predicts protected attribute
        self.alpha = alpha           # Trade-off
    
    def train_step(self, X, y, protected, optimizer_pred, optimizer_adv):
        # 1. Predictor forward
        y_pred = self.predictor(X)
        
        # 2. Adversary tries to predict protected from y_pred
        protected_pred = self.adversary(y_pred)
        
        # 3. Predictor loss = task_loss - alpha * adversary_loss
        # (predictor wants to MAX adversary's loss = MIN its accuracy)
        task_loss = nn.BCELoss()(y_pred, y)
        adv_loss = nn.BCELoss()(protected_pred, protected)
        
        predictor_loss = task_loss - self.alpha * adv_loss
        
        # 4. Update predictor
        optimizer_pred.zero_grad()
        predictor_loss.backward(retain_graph=True)
        optimizer_pred.step()
        
        # 5. Update adversary (just minimize its own loss)
        optimizer_adv.zero_grad()
        adv_loss.backward()
        optimizer_adv.step()
        
        return {
            'task_loss': task_loss.item(),
            'adv_loss': adv_loss.item(),
        }

In-processing trade-offs

Pros:

  • More effective than pre-processing in many cases.
  • Direct control over the fairness/accuracy trade-off via λ.
  • The model learns not to use proxy variables during training.

Cons:

  • It requires a customizable model (doesn't work with black-box APIs).
  • Training is more complex (especially adversarial: two networks, tricky balancing).
  • Less interpretable: stakeholders ask "what did you do?" → the answer is "fairness regularization in the loss."
  • Harder to debug if it doesn't converge.

When to choose in-processing

  • You have complete control over the training pipeline.
  • A deep learning model you can modify.
  • Pre-processing alone isn't enough.
  • You have the expertise/time to tune hyperparameters.

Post-processing: modifying the outputs

The concept

The model is already trained (it can be a black box). You modify the outputs before returning them as a decision.

The most popular: threshold tuning per group. Instead of a single threshold (e.g., approve if score > 0.5), use a different threshold for each group:

Group A: approve if score > 0.50
Group B: approve if score > 0.43

A lower threshold for group B compensates for miscalibration or disparate impact.

Technique 1: Threshold tuning

def find_fair_thresholds(scores, ground_truth, groups, target_metric='dp',
                        target_value=0.95):
    """
    Find per-group thresholds that satisfy a target metric.
    
    target_metric: 'dp' (demographic parity), 'eo' (equal opportunity).
    target_value: e.g., 0.95 = a 95% match on the metric.
    """
    df = pd.DataFrame({
        'score': scores,
        'true': ground_truth,
        'group': groups,
    })
    
    # For each group, find the threshold that produces a similar TPR (for EO)
    # or a similar positive rate (for DP).
    
    if target_metric == 'dp':
        # Demographic parity: equal positive rate
        # Find the per-group threshold that gives the target positive rate
        thresholds = {}
        for group_name, group_df in df.groupby('group'):
            # Default: the majority group's positive rate
            target_pos_rate = (df['score'] > 0.5).mean()
            
            sorted_scores = sorted(group_df['score'].values, reverse=True)
            threshold_idx = int(target_pos_rate * len(sorted_scores))
            threshold = sorted_scores[threshold_idx]
            thresholds[group_name] = threshold
        
        return thresholds
    
    elif target_metric == 'eo':
        # Equal opportunity: equal TPR
        target_tpr = 0.80  # configurable
        
        thresholds = {}
        for group_name, group_df in df.groupby('group'):
            positives = group_df[group_df['true'] == 1]
            
            # A threshold such that TPR = target
            sorted_pos_scores = sorted(positives['score'].values, reverse=True)
            tpr_idx = int(target_tpr * len(sorted_pos_scores))
            threshold = sorted_pos_scores[tpr_idx]
            thresholds[group_name] = threshold
        
        return thresholds


def apply_per_group_threshold(scores, groups, thresholds):
    """Apply per-group thresholds."""
    decisions = []
    for score, group in zip(scores, groups):
        threshold = thresholds.get(group, 0.5)  # default if group not in thresholds
        decisions.append(1 if score > threshold else 0)
    return decisions

Threshold tuning trade-offs

Pros:

  • It applies to any model (black-box compatible).
  • Simple to implement.
  • Reversible: if it doesn't work, go back to the uniform threshold.

Cons:

  • Legally questionable in some contexts: applying different thresholds by group can be "disparate treatment" (treating individuals differently based on a protected attribute), which is illegal in US hiring/lending.
  • It requires demographic data in production (to apply the right threshold).
  • It doesn't address the root cause: the model is still biased, it's just mitigated at decision time.

An important legal caveat

In the US, applying different thresholds by group can be illegal (disparate treatment). The exception: explicitly affirmative action programs that have been court-approved.

In the EU, the EU AI Act includes provisions on equal treatment. Check the specific regulation before implementing.

Solution: sometimes tuned thresholds are used internally for analysis, but the deploy uses a single threshold + accepts that demographic parity won't be perfect.

Technique 2: Calibration adjustment

If your model is miscalibrated across groups (capsule 03), adjust the scores post-hoc:

from sklearn.isotonic import IsotonicRegression

def calibrate_per_group(scores, ground_truth, groups):
    """
    Apply isotonic regression per group for calibration.
    """
    df = pd.DataFrame({
        'score': scores,
        'true': ground_truth,
        'group': groups,
    })
    
    calibrators = {}
    for group_name, group_df in df.groupby('group'):
        ir = IsotonicRegression(out_of_bounds='clip')
        ir.fit(group_df['score'].values, group_df['true'].values)
        calibrators[group_name] = ir
    
    return calibrators


def apply_calibration(scores, groups, calibrators):
    """Apply per-group calibration to new predictions."""
    calibrated = []
    for score, group in zip(scores, groups):
        if group in calibrators:
            calibrated.append(calibrators[group].predict([score])[0])
        else:
            calibrated.append(score)
    return calibrated

Output: each group has separately calibrated scores. "Score = 0.7" means a real 70% probability for every group.

Technique 3: Reject option

In marginal zones (scores near the threshold), reject and require human review instead of auto-deciding.

def reject_option(scores, low_threshold=0.4, high_threshold=0.6):
    """
    Approve/deny only if the score is very high/low. Otherwise, manual review.
    """
    decisions = []
    for score in scores:
        if score >= high_threshold:
            decisions.append('approve')
        elif score <= low_threshold:
            decisions.append('deny')
        else:
            decisions.append('manual_review')
    return decisions

Useful when errors carry a high cost. Marginal cases get human review = more accurate decisions in those cases.

Cost: 20-30% of cases require human review. Operational overhead.

Combining the categories

In production-grade systems, you combine all three:

DATA  →  MODEL  →  OUTPUT  →  DECISION
  ↓        ↓         ↓
Pre-      In-      Post-
proc     proc      proc

Defense in depth

  • Pre-processing: reduces the initial bias in the data.
  • In-processing: the model learns not to use proxies.
  • Post-processing: corrects residual bias in the outputs.

Each one covers what the others don't. Combined, they reduce bias significantly more than any one alone.

An example pipeline

def fair_pipeline(X_train, y_train, X_test, y_test, groups_train, groups_test):
    """
    A complete pipeline with all the categories.
    """
    # 1. Pre-processing: re-weighting
    weights = compute_sample_weights_for_groups(y_train, groups_train)
    
    # 2. In-processing: train with fairness regularization
    model = FairModel(lambda_fair=0.3)
    model.fit(X_train, y_train, sample_weight=weights, groups=groups_train)
    
    # 3. Post-processing: calibrate per group
    train_scores = model.predict_proba(X_train)
    calibrators = calibrate_per_group(train_scores, y_train, groups_train)
    
    # 4. Inference with all the corrections
    test_scores = model.predict_proba(X_test)
    calibrated_scores = apply_calibration(test_scores, groups_test, calibrators)
    
    return calibrated_scores

How to choose between categories

The decision tree

Do you have access to the training data?
├── No → POST-PROCESSING ONLY (threshold tuning, calibration)
│
└── Yes → Do you have expertise in customizing training?
          ├── No → PRE-PROCESSING (simplest)
          │
          └── Yes → Is pre-processing alone enough?
                    ├── Yes → PRE-PROCESSING
                    │
                    └── No → Does the model allow it (deep learning, custom training)?
                             ├── Yes → COMBINE PRE + IN-PROCESSING
                             │
                             └── No → COMBINE PRE + POST-PROCESSING

By practical scenario

Scenario 1: Using an OpenAI/Anthropic API (black-box):

  • You can't modify the training.
  • Pre-processing of inputs (prompt engineering): limited.
  • Post-processing only: filter the outputs, threshold tuning if you're scoring, a reject option.

Scenario 2: A custom sklearn model:

  • Pre-processing: re-weighting or re-sampling.
  • Post-processing: threshold tuning if legally tolerable.

Scenario 3: Custom deep learning:

  • Combine everything: pre-processing + adversarial debiasing + post-hoc calibration.

Scenario 4: Enterprise constraints (legal sensitivity):

  • Pre-processing only (more defensible).
  • Post-processing only if court-approved.

Validating the mitigation

For any technique:

  1. Apply it.
  2. Re-measure the metrics (DP, EO, calibration).
  3. Compare against the baseline (no mitigation).
  4. Evaluate the accuracy impact.
  5. Document the trade-offs.
def evaluate_mitigation(predictions_baseline, predictions_mitigated,
                        ground_truth, groups):
    """
    Compare baseline vs mitigated across all the metrics.
    """
    return {
        'baseline': {
            'accuracy': accuracy_score(ground_truth, predictions_baseline),
            'demographic_parity': demographic_parity(predictions_baseline, groups),
            'equalized_odds': equalized_odds(predictions_baseline, ground_truth, groups),
        },
        'mitigated': {
            'accuracy': accuracy_score(ground_truth, predictions_mitigated),
            'demographic_parity': demographic_parity(predictions_mitigated, groups),
            'equalized_odds': equalized_odds(predictions_mitigated, ground_truth, groups),
        },
    }

Report the full table in your Bias Audit. Stakeholders decide whether the trade-off is acceptable.

Traps and common mistakes

1. Applying a mitigation without measuring

"I applied adversarial debiasing, so now it's fair." Without re-measuring, you don't know.

2. Superficial λ tuning

Adversarial debiasing and fairness regularization have a λ hyperparameter. Tuning it correctly requires a sweep + validation. The default λ=1 is rarely optimal.

3. Threshold tuning without considering legality

US hiring + per-group threshold tuning = a potential disparate treatment claim. Check with legal first.

4. Calibration alone, expecting demographic parity

Calibration does not guarantee demographic parity (impossibility theorems). Each metric requires its own mitigation.

5. Not combining techniques

Pre alone is sometimes insufficient. In alone is sometimes insufficient. Combinations tend to be more robust.

Self-check

1. When is post-processing your only option?

When you do NOT have access to the training pipeline. Cases:

  1. The model is an external API (OpenAI, Anthropic, Google): you can't modify the training or access the weights.

  2. The model is a vendor product: you buy a trained model, you can't retrain it.

  3. Compliance restrictions: some regulated industries don't allow modifying the training data or algorithms.

  4. The cost of retraining is prohibitive: foundation models that cost millions to train.

In these cases, post-processing (threshold tuning, calibration adjustment, filtering, a reject option) is all you have.

Limitations: post-processing doesn't address the root cause. The model is still biased; you're only filtering the outputs. For serious systems, ideally migrate to a model you can modify.

2. Why can adversarial debiasing be more effective than pre-processing?

Pre-processing balances the data, but the model can still learn subtle proxies for the protected attribute (remember Apple Card).

Adversarial debiasing explicitly forces the model to produce outputs that contain no information about the protected attribute. If the model learns a proxy, the adversary can recover the attribute, and the predictor is penalized.

Result: the model learns representations that are fairness-aware, not just decisions. More robust against proxy discrimination.

Trade-off: adversarial training is technically harder. It requires balancing two networks, tuning λ, debugging convergence. For teams without the expertise, pre-processing may be more practical even if less effective.

3. Why can per-group threshold tuning be legally problematic?

Because applying different thresholds based on a protected attribute = "disparate treatment" — treating individuals differently because of their group, which is illegal in the US under the Civil Rights Act, ECOA, FHA, etc.

The typical case:

  • A hiring system with a 0.5 threshold for men, 0.4 for women.
  • Even if the motivation is achieving demographic parity, in court → "applied a lower bar to female applicants" = disparate treatment.

Exceptions:

  • Court-approved affirmative action programs: they require legal precedent.
  • Disparate impact remediation: if a court determines a single threshold causes disparate impact, court-ordered differentiated thresholds may be allowed.

But by default, in the US: uniform thresholds are more defensible. The EU has similar logic under the equal treatment principle.

A common practice: use tuned thresholds for internal analysis (to understand the trade-off), but deploy with a single threshold + accept that demographic parity won't be perfect.

Exceptions by industry:

  • Marketing/advertising: per-group thresholds are more acceptable (not high-stakes).
  • Healthcare: clinical thresholds per population are legitimate (biologically different base rates).
4. Why is combining pre + in + post processing defense in depth?

Each category addresses bias at a different stage:

Pre-processing mitigates bias in the input data: it balances representation before training.

In-processing mitigates bias during learning: the model doesn't learn subtle proxies.

Post-processing mitigates bias in the decisions: it corrects residual disparities in the outputs.

Combined:

  • Pre-processing reduces input bias.
  • In-processing prevents the model from learning the remaining bias.
  • Post-processing cleans up any residual bias.

The total bias reduction > any single category alone.

Defense in depth is a term from security: if one layer fails, another covers. The same concept applies to fairness.

Trade-off: complexity. Combining three is more complex than one. For small systems, one may be enough. For critical systems (loans, hiring at scale), combining is prudent.

Summary and next step

  • In-processing: modify the training. Fairness regularization, adversarial debiasing. More effective but requires control over the model.
  • Post-processing: modify the outputs. Threshold tuning, calibration, reject option. Applicable to black boxes, but legally questionable in some cases.
  • Combining all three categories (pre + in + post) = defense in depth.
  • Always validate: re-measure the metrics post-mitigation, compare the accuracy impact, document the trade-offs.
  • The decision depends on constraints: if it's a black box, post only. If it's custom DL, combine.

Checkpoint: you should be able to choose the right technique based on your constraints (data access, model access, legal context).

Bridge to the next capsule: capsule 08 is the mini-project: the Bias Audit Toolkit. You'll integrate everything you learned — metrics, detection, mitigation — into a reusable toolkit you can apply to future AI systems. The deliverable is a Python repo with tests, scripts, and a README documenting the process.

Resources

  1. Adversarial Learning for Fair Classification (Zhang et al., 2018) — the foundational adversarial debiasing paper.
  2. Fairlearn — ExponentiatedGradient — an in-processing implementation.
  3. AIF360 — Calibrated Equalized Odds — post-processing.
  4. Equality of Opportunity Calibrator (Pleiss et al., 2017) — the technical paper.

Next: 08-mini-project-bias-audit-toolkit.md — Build your reusable Bias Audit Toolkit.

Capsule 07 of 08 — Module 2 — AI Ethics & Compliance Guide

← Previous6. Mitigation: Pre-processing TechniquesNext →8. Mini-project: Bias Audit Toolkit