6. Mitigation: Pre-processing Techniques

Capsule description

You detected bias (capsules 02-04). You understood the trade-offs (capsule 05). Now the actionable question: what do you do to reduce it?

Mitigation techniques are organized into three categories based on where they intervene:

1. Pre-processing: modify the training data before training.
2. In-processing: modify the training algorithm (capsule 07).
3. Post-processing: modify the trained model's outputs (capsule 07).

This capsule covers pre-processing — the simplest category, the most interpretable, and often the most effective. If your data is biased, the first fix is the data.

You'll learn 4 concrete techniques:

1. Re-balancing: equalize subgroup sizes in the training set.
2. Re-sampling: oversample minority subgroups.
3. Re-weighting: give more weight to under-represented samples.
4. Data augmentation: generate synthetic samples for under-represented groups.

All of them implemented from scratch, with runnable Python code.

---

Why start with pre-processing

Technical reasons

1. It's simpler: changes to the data, not the model.
2. Compatible with any model: it applies before you choose an architecture.
3. Interpretable: you can inspect the resulting data. "It was 80/20, now it's 50/50."
4. Reusable: balanced data gets reused across experiments.

Ethical reasons

1. It treats the problem at its root: the bias came in through the data, so fix the data.
2. It's transparent: stakeholders understand "we balanced the dataset."
3. It requires no claims about the model: the model is the same, the data changed.

Limitations

1. It only works if you have access to the training data: if you use an external API (OpenAI), you can't modify their training data.
2. It isn't always enough: structural bias may require in-processing/post-processing too.
3. It can reduce predictive signal: if you oversample minorities that have different patterns, the model learns mixed patterns.

---

Technique 1: Re-balancing

The concept

If your dataset has 10,000 samples with 90% group A and 10% group B, subsample group A down to 50/50.

Before:
  Group A: 9,000 samples
  Group B: 1,000 samples
  Total: 10,000

After:
  Group A: 1,000 samples (random subsample)
  Group B: 1,000 samples
  Total: 2,000

Implementation

import pandas as pd
import numpy as np

def rebalance_dataset(df, group_column, target_size_per_group=None,
                      random_state=42):
    """
    Re-balance a dataset by subsampling.
    
    Args:
        df: DataFrame with features + the group column.
        group_column: name of the protected attribute column.
        target_size_per_group: desired size per group. If None, use min(group_sizes).
    
    Returns:
        A balanced DataFrame.
    """
    group_sizes = df.groupby(group_column).size()
    
    if target_size_per_group is None:
        target_size_per_group = group_sizes.min()
    
    balanced_dfs = []
    for group_name, group_df in df.groupby(group_column):
        if len(group_df) > target_size_per_group:
            # Subsample
            sampled = group_df.sample(target_size_per_group, random_state=random_state)
        else:
            # Use all
            sampled = group_df
        balanced_dfs.append(sampled)
    
    balanced = pd.concat(balanced_dfs, ignore_index=True)
    
    return balanced.sample(frac=1, random_state=random_state).reset_index(drop=True)

Usage

# Original dataset
print(df['gender'].value_counts())
# M    9000
# F    1000

balanced = rebalance_dataset(df, group_column='gender')

print(balanced['gender'].value_counts())
# M    1000
# F    1000

Trade-offs

Pros:

• Simple.
• Guarantees equal representation.

Cons:

• You lose data: 8,000 samples discarded. You lose signal.
• It doesn't work if the minority group has < N samples for meaningful training.

When to use it: when you have lots of data and the difference between groups is very large. If you have 1M samples with an 80/20 split, you can balance to 50/50 without losing too much in total.

---

Technique 2: Re-sampling (Oversampling)

The concept

Instead of discarding the majority, replicate the minority. If group B has 1,000 samples, multiply them until you have 9,000 (matching A).

Before:
  Group A: 9,000
  Group B: 1,000

After (oversampling):
  Group A: 9,000 (unchanged)
  Group B: 9,000 (each one duplicated 9x, or sampled with replacement)
  Total: 18,000

Implementation: simple oversampling with replacement

def oversample_minority(df, group_column, random_state=42):
    """
    Oversample minorities up to the majority's size.
    """
    group_sizes = df.groupby(group_column).size()
    target_size = group_sizes.max()
    
    sampled_dfs = []
    for group_name, group_df in df.groupby(group_column):
        if len(group_df) < target_size:
            # Oversample with replacement
            extra_needed = target_size - len(group_df)
            extra_samples = group_df.sample(extra_needed, replace=True, 
                                             random_state=random_state)
            sampled = pd.concat([group_df, extra_samples])
        else:
            sampled = group_df
        sampled_dfs.append(sampled)
    
    result = pd.concat(sampled_dfs, ignore_index=True)
    return result.sample(frac=1, random_state=random_state).reset_index(drop=True)

Trade-offs

Pros:

• No data lost (it keeps the majority).
• Simple.

Cons:

• Exact repetition of samples can cause overfitting to the minority group's pattern.
• If the minority group is very small (< 50), repetition gives you little diversity.

SMOTE: Synthetic Minority Over-sampling Technique

A more sophisticated version. Instead of repeating, interpolate between nearby samples to generate synthetic ones:

from imblearn.over_sampling import SMOTE

def smote_oversample(X, y, group_labels, random_state=42):
    """
    SMOTE oversampling.
    Generates synthetic samples instead of repeating.
    
    Note: the original SMOTE is for class imbalance.
    For bias, you can use it on group_labels or combine them.
    """
    smote = SMOTE(random_state=random_state)
    X_resampled, y_resampled = smote.fit_resample(X, group_labels)
    return X_resampled, y_resampled

SMOTE generates samples in the "space" between real samples of the minority group. More diverse than pure repetition.

SMOTE's limitations:

• It only works for numeric features (not text directly).
• Synthetic samples may not be realistic.
• For bias correction (as opposed to class imbalance), the application is indirect.

---

Technique 3: Re-weighting

The concept

Instead of duplicating samples, give more weight to under-represented samples during training.

If group A is 90% and group B is 10%, the weight for B's samples = 9x the weight for A's.

Models like sklearn's accept a sample_weight parameter. The loss is computed weighted, so samples with weight 9 count 9 times more in the gradient.

Implementation

def compute_sample_weights(df, group_column):
    """
    Compute weights inversely proportional to the group's frequency.
    """
    group_counts = df[group_column].value_counts()
    total = len(df)
    n_groups = len(group_counts)
    
    # Weight = total / (n_groups * count_group)
    # This gives weight ~1 if groups are balanced, > 1 if under-represented
    weights = df[group_column].apply(
        lambda g: total / (n_groups * group_counts[g])
    )
    
    return weights.values

# Usage with sklearn
from sklearn.linear_model import LogisticRegression

weights = compute_sample_weights(df, 'gender')

model = LogisticRegression()
model.fit(X_train, y_train, sample_weight=weights)

Trade-offs

Pros:

• It doesn't physically modify the data.
• More efficient than duplicating.
• Compatible with most ML libraries (sklearn, xgboost, lightgbm).

Cons:

• Not every model accepts sample_weights (some NN frameworks require custom code).
• Outlier samples with a high weight can dominate training.

When to choose re-weighting over oversampling

• Very large data: re-weighting is more memory-efficient.
• Models that accept weights: sklearn, xgboost, lightgbm.
• You want to keep real diversity without synthetic samples.

---

Technique 4: Data Augmentation

The concept

For under-represented groups, generate synthetic samples that are more realistic than plain SMOTE.

The approach depends on the data type:

For tabular data

Generate variations of real samples by slightly modifying non-protected features:

def augment_tabular(df, group_column, minority_label, n_augment=1000,
                    noise_scale=0.05):
    """
    Augment tabular data by adding Gaussian noise to numeric features.
    """
    minority = df[df[group_column] == minority_label]
    numeric_cols = minority.select_dtypes(include=[np.number]).columns
    
    augmented_samples = []
    for _ in range(n_augment):
        sample = minority.sample(1).copy()
        # Add gaussian noise to numeric features
        for col in numeric_cols:
            std = minority[col].std()
            noise = np.random.normal(0, std * noise_scale)
            sample[col] += noise
        augmented_samples.append(sample)
    
    augmented_df = pd.concat(augmented_samples + [df], ignore_index=True)
    return augmented_df

For text data

Augmentation by paraphrasing:

• Synonym replacement.
• Back-translation (English → French → English).
• Style transfer.

# Pseudo-code (requires an NLP library)
def augment_text(text):
    """Augment text via back-translation."""
    intermediate = translate(text, source='en', target='fr')
    augmented = translate(intermediate, source='fr', target='en')
    return augmented

With LLMs:

# Generate paraphrases with GPT
def llm_paraphrase(text, n=5):
    prompt = f"""Generate {n} paraphrases of this text, preserving meaning:
    
    {text}
    
    Return as JSON array."""
    
    response = llm.complete(prompt)
    return json.loads(response)

For image data

Standard augmentations (rotation, flip, brightness) + more sophisticated ones:

• Generative augmentation with StyleGAN or similar.
• For demographic balance: generate samples of under-represented demographics.

An ethical caveat: if you generate synthetic faces of minorities for training, make sure the generator did NOT inherit bias from the original dataset.

Trade-offs

Pros:

• More diversity than pure repetition.
• It can expand the distribution covered.

Cons:

• Quality depends on the augmentation method.
• Synthetic data can introduce new artifacts.
• Computationally expensive.

---

Combining techniques

In practice, you rarely use just one. Typical combinations:

Combo 1: Re-weighting + Augmentation

1. Augment the minority group with paraphrasing/SMOTE.
2. Apply re-weighting on top of the augmented dataset.

Result: more diversity + balance.

Combo 2: Stratified rebalancing by intersectional groups

def rebalance_intersectional(df, attributes, target_per_combo=None):
    """
    Re-balance by attribute combination (intersectional).
    """
    combo_sizes = df.groupby(attributes).size()
    
    if target_per_combo is None:
        target_per_combo = combo_sizes.min()
    
    balanced = df.groupby(attributes).apply(
        lambda x: x.sample(min(len(x), target_per_combo))
    ).reset_index(drop=True)
    
    return balanced

Useful for solving intersectional bias (Black women under-represented), not just single-attribute bias.

---

Validating the mitigation

After applying pre-processing, measure the metrics again:

def validate_mitigation(df_original, df_mitigated, model, test_set, 
                        group_column):
    """
    Retrain the model on the mitigated data and compare the metrics.
    """
    # Train on the original data
    model_original = clone(model)
    model_original.fit(df_original.drop('label', axis=1), df_original['label'])
    pred_original = model_original.predict(test_set.drop('label', axis=1))
    
    # Train on the mitigated data
    model_mitigated = clone(model)
    model_mitigated.fit(df_mitigated.drop('label', axis=1), df_mitigated['label'])
    pred_mitigated = model_mitigated.predict(test_set.drop('label', axis=1))
    
    # Compare
    dp_original = demographic_parity(pred_original, test_set[group_column])
    dp_mitigated = demographic_parity(pred_mitigated, test_set[group_column])
    
    acc_original = accuracy_score(test_set['label'], pred_original)
    acc_mitigated = accuracy_score(test_set['label'], pred_mitigated)
    
    return {
        'parity_improvement': dp_mitigated['parity_ratio'] - dp_original['parity_ratio'],
        'accuracy_change': acc_mitigated - acc_original,
        'recommend_deploy': (
            dp_mitigated['parity_ratio'] >= 0.80 and
            acc_mitigated >= acc_original - 0.05
        ),
    }

If fairness improves but accuracy drops > 5%: investigate other techniques. Pre-processing may not be enough; move to in-processing (capsule 07).

If fairness improves and accuracy is similar: deploy the mitigated version.

If fairness doesn't improve: the technique you chose isn't the right one. Try another.

---

Traps and common mistakes

1. Applying a mitigation without measuring post-hoc

You applied re-balancing, so "now it's fair." Did you verify? Without re-measuring, you don't know.

2. Re-balancing loses too much data

If the minority group is 1% (rare), perfect re-balancing discards 99% of the data. Useless.

Solution: use oversampling or re-weighting when the minority is very small.

3. Synthetic samples that aren't realistic

SMOTE on tabular data can generate impossible samples (e.g., a person with negative income). Validate synthetic samples before training.

4. Augmentation that changes the label

If paraphrasing text changes its meaning, the label is invalidated. Validate.

5. Not considering the downstream cost

Re-balancing can improve fairness on train but degrade accuracy in a production deployment with a different distribution. Validate on a realistic test set.

---

Self-check

1. When would you choose oversampling over rebalancing?

When your minority group is very small in absolute terms. Re-balancing by subsampling the majority would leave you with a dataset too small for meaningful training.

Example:

• Group A: 100,000 samples
• Group B: 500 samples

Re-balancing → 500 + 500 = 1,000 samples total. Probably not enough to train a decent model.

Oversampling → 100,000 + 100,000 (B replicated 200x) = 200,000 samples. The model trains with the full majority + an oversampled B. Risk: overfitting to B's patterns because each sample appears many times. Mitigated with SMOTE (synthetic) instead of duplication.

Alternatively, re-weighting is viable: without modifying the data, give B's samples a 200x weight during training.

The decision matrix:

Minority size	Recommended
Very small (<100)	Oversampling with SMOTE/augmentation
Small (100-1000)	Oversampling or re-weighting
Medium (1000-10000)	Re-weighting or re-balancing
Large but skewed	Re-balancing or re-weighting

2. Why is re-weighting often preferable to duplication?

Three reasons:

1. Memory efficiency: not duplicating samples = less RAM/storage.

2. It preserves real diversity: with duplication, the model sees the exact same samples multiple times, introducing overfitting risk. With re-weighting, the model sees each sample once with an adjusted weight.

3. Faster training: fewer samples = fewer gradient computations.

Caveat: it requires the model to support sample_weights. Sklearn, XGBoost, and LightGBM do. Some custom NN frameworks may not support it natively — that would require a custom loss function.

In the modern stack, re-weighting is the default option when the model supports it.

3. When is data augmentation NOT viable?

Cases:

1. An extremely sensitive domain: medical data, legal records — synthetic samples can be legally problematic or factually misleading.

2. The augmentation method introduces bias: if your augmentation tool (LLM, GAN) has its own bias, you propagate it into the dataset.

3. Quality verification is hard: for text augmentation, verifying that synthetic samples preserve meaning requires human review (doesn't scale).

4. Regulatory restrictions: some jurisdictions require the training data to be "real" — synthetic samples don't qualify.

5. Domain-specific complexity: augmenting medical images requires expert validation that the synthetic data doesn't introduce false features.

When augmentation isn't viable, the alternatives are: re-weighting, in-processing techniques (capsule 07), or more balanced data collection upstream.

4. How do you decide whether pre-processing is enough or you also need in/post-processing?

The procedure:

1. Apply pre-processing (re-balance/oversample/re-weight).

2. Retrain the model on the mitigated data.

3. Measure the fairness metrics on the test set.

4. Decide:

   • If fairness passes the thresholds and accuracy ~ the original: pre-processing is sufficient. Deploy.
   • If fairness passes but accuracy drops > 5%: an explicit trade-off. Consider other techniques (in/post-processing can have a lower accuracy cost).
   • If fairness doesn't pass the thresholds: pre-processing is insufficient. Combine it with in/post-processing.

General rule: start with pre-processing because it's the simplest. If it's not enough, escalate the complexity with in/post-processing. Combinations (pre + post) are common in production systems.

---

Summary and next step

• Pre-processing modifies the training data. Simplest, interpretable, model-agnostic.
• Techniques:
  • Re-balancing: subsample the majority (loses data).
  • Oversampling: replicate or SMOTE the minority.
  • Re-weighting: weights at training time (most efficient if the model supports it).
  • Augmentation: synthetic samples (depends on the data type).
• Combine techniques for better balance + diversity.
• Validate post-mitigation: measure the metrics again, do NOT assume that applying a mitigation = fair.
• Pre-processing may not be enough: if it isn't, combine it with in/post-processing.

Checkpoint: you should be able to choose and apply the right pre-processing technique based on your data's characteristics.

Bridge to the next capsule: capsule 07 covers in-processing (modifying the training algorithm: fairness constraints in the loss, adversarial debiasing) and post-processing (modifying predictions: threshold tuning per group, calibration adjustment, a reject option). These are more sophisticated techniques you apply when pre-processing alone isn't enough, or when you can't modify the training data.

---

Resources

1. imbalanced-learn library — SMOTE and other techniques.
2. Fairlearn — Reweighting preprocessor — implementations.
3. AIF360 — Reweighing — reference implementation.
4. Synthetic Data Vault — tabular augmentation.

---

Next: 07-mitigation-in-post-processing.md — In-processing and post-processing mitigations.

Capsule 06 of 08 — Module 2 — AI Ethics & Compliance Guide