4. Case Study #3: Apple Card and Credit Scoring

Capsule description

November 2019. David Heinemeier Hansson (creator of Ruby on Rails) posts on Twitter that Apple Card granted him a credit limit 20x higher than his wife's, even though they share finances, file joint tax returns, and she has a better credit score than he does.

The thread went viral. Steve Wozniak (Apple co-founder) replied that he had the same experience: 10x more credit than his wife, same finances. Dozens of other couples reported the same thing.

In under 60 days, the New York Department of Financial Services opened a formal investigation. Goldman Sachs (Apple Card's issuer) had to defend its algorithm publicly. And although the official investigation concluded (2021) that it found no legal evidence of intentional discrimination, the case became the canonical example of proxy discrimination: how a model can discriminate without using explicit gender features.

This capsule covers the case. The technical payoff is understanding proxy variables — seemingly neutral features that act as proxies for protected attributes, producing disparate impact. And learning how to detect them before you deploy.

---

What happened

Timeline

• August 2019: Apple Card launches, issued by Goldman Sachs. Marketing: "the most transparent, customer-friendly credit card". Credit scoring algorithm: proprietary, undocumented.
• November 7, 2019: David Heinemeier Hansson posts:

  "The Apple Card is such a fucking sexist program. My wife and I filed joint tax returns, live in a community-property state, and have been married for a long time. Yet Apple's black box algorithm thinks I deserve 20x the credit limit she does."

• November 10, 2019: Steve Wozniak replies:

  "The same thing happened to us. I got 10x the credit limit. We have no separate bank or credit card accounts or any separate assets. Hard to get to a human for a correction though. It's big tech in 2019."

• November 11, 2019: the NY Department of Financial Services announces a formal investigation.
• November-December 2019: hundreds of similar reports on Twitter. Some women report being rejected outright when applying as individuals but approved when applying with their husbands.
• March 2021: NYDFS publishes its report. Official conclusion: no violation of anti-discrimination law found. But the report recommends: better transparency, a better appeal process, better fairness testing.

What Goldman Sachs said

Goldman's defense was that:

• They did not use gender as a feature in the model.
• They had no access to marital status or to the partner's data.
• Limits were decided solely on the applicant's individual credit history, individual income, and other individual financial factors.

That defense is probably factually correct. And even so, the system's outputs showed a clear gender pattern.

How is that possible? The answer: proxies.

---

Why it happened: the mechanism of proxy discrimination

What a proxy variable is

A proxy variable is a feature that looks neutral but is statistically correlated with a protected attribute. When the model learns to use it, it indirectly learns to use the protected attribute.

Classic examples:

Apparent feature	Proxy for	How
ZIP code	Race	Historic residential segregation → ZIP correlates with race
University	Race, gender, class	Universities have historically filtered demographically
Name	Gender, ethnicity	Names have a clear demographic distribution
Continuous employment history	Gender (women)	Maternity breaks fragment the history
Type of employment	Gender	Uneven gender distribution across industries
Purchase of specific products	Gender, class	Consumption categories differ demographically
Time of purchase / spending pattern	Age, gender	Life patterns correlate with demographics

Why Apple Card could discriminate without explicit features

Even though Goldman didn't use "gender" as a feature, it used many features that correlate with gender:

Individual credit history: in traditional heterosexual couples (especially older couples), credit has historically been issued more often in the husband's name. Joint cards defaulted to the husband as "primary cardholder." That leaves the woman with a shorter or thinner credit history of her own, even if they shared the exact same cards and paid together.

Applied to the Wozniaks: Steve was probably "primary" on cards and loans for 40+ years. His wife, technically, had less credit history of her own, even though the family finances were exactly the same.

Individual income: in couples where one spouse earns significantly more than the other, individual income is lower for the lower-earning spouse, even when expenses are shared.

Credit usage patterns: purchase types correlate with gender in aggregate datasets. The model learns that.

Employment industry: if the applicant is an engineer (mostly male) vs. a yoga teacher (mostly female), the model associates industry with risk score. Indirectly, it discriminates by gender.

The full process

Seemingly neutral inputs:
  - individual credit history
  - individual income
  - usage patterns
  - industry
  - ZIP code

        ↓

Model learns statistical correlations

        ↓

Some correlations are proxies for gender/race/age

        ↓

Output: credit limit with disparate impact
        by gender (without gender being in the model)

Gender never goes in; it comes out anyway.

Why the model works "fine" individually but fails in aggregate

At the individual level, the model can defend itself: "this person has credit history X, income Y, patterns Z; therefore they get credit limit W." Each variable is justifiable.

At the aggregate level, the composition of variables produces bias: women as a group have certain distributions of X, Y, Z that the model penalizes. The bias isn't in any single feature — it's in how the set of features systematically disadvantages the group.

It's like a casino where every individual game has "fair odds" by some metric, but the mix of available games favors the house. Piece by piece, "fair." In aggregate, biased.

---

The measurable cost

Legal and regulatory cost to Goldman/Apple

• NYDFS investigation: ~$1M+ in legal and compliance costs during the investigation.
• System changes: Goldman had to implement a re-evaluation process. Operational and development cost.
• Appeal process: Goldman created an explicit process for applicants to request human review of algorithmic decisions.
• Ongoing regulatory cooperation: Goldman now operates under heightened scrutiny. Permanently higher compliance costs.

Reputational cost

• Weeks of coverage in the NYT, WSJ, FT, BBC, Bloomberg, dozens of outlets.
• Apple Card, marketed as "transparent and customer-friendly," was tainted with "algorithmic discrimination."
• Dozens of blog posts and academic case studies. The case is taught in MBA programs.

Personal cost to applicants

The women reported:

• Receiving dramatically lower credit limits than their partners with identical finances.
• Feeling like they had to "explain their worth" while their husbands did not.
• No way to get a clear answer about why the algorithm decided X.
• Frustration with customer service ("we can't explain the decision, it's an algorithm").

Regulatory cost to the fintech industry

• Accelerated regulation of "explainable AI" in credit decisions.
• Reinforced FCRA (Fair Credit Reporting Act) application to ML systems.
• Catalyzed proposals like the Algorithmic Accountability Act (US).
• EU: the case is cited in the credit scoring section of the EU AI Act.

---

How it could have been prevented: the missing techniques

Technique 1: Disparate impact testing

Missing: testing whether the model's outputs had significantly unequal approval rates or credit limits by demographic subgroup.

Even though gender wasn't an input, the model could have been tested to see whether the output differentiated by gender. Goldman could have (should have) run an internal analysis where they:

1. Take a sample of historical applicants.
2. Infer or join with demographic data wherever legal.
3. Compute the approval rate / average credit limit by gender.
4. Apply the 4/5 rule.

If the ratio came out < 80%, the model had disparate impact and should have been revised before going to production.

Implementation:

def disparate_impact_test(model_decisions, demographics):
    """
    model_decisions: DataFrame with scores/approvals
    demographics: DataFrame with gender, ethnicity, etc.
    """
    merged = model_decisions.merge(demographics, on='applicant_id')
    
    by_gender = merged.groupby('gender').agg({
        'approved': 'mean',
        'credit_limit': 'mean',
    })
    
    approval_ratio = (
        by_gender.loc['F', 'approved'] / 
        by_gender.loc['M', 'approved']
    )
    
    limit_ratio = (
        by_gender.loc['F', 'credit_limit'] / 
        by_gender.loc['M', 'credit_limit']
    )
    
    return {
        'approval_ratio_F_to_M': approval_ratio,
        'limit_ratio_F_to_M': limit_ratio,
        'passes_4_5_rule_approval': approval_ratio >= 0.80,
        'passes_4_5_rule_limit': limit_ratio >= 0.80,
    }

Technique 2: Proxy variable detection

Missing: an explicit analysis of which features act as proxies for protected attributes.

Implementation: for each feature in the model, compute its correlation with gender (wherever inferable), race (via ZIP, name), age. Features with a high correlation are suspect proxies.

def detect_proxies(features_df, protected_attribute):
    """
    For each feature, compute how well it predicts protected_attribute.
    If a feature predicts protected_attribute well, it's a proxy.
    """
    from sklearn.linear_model import LogisticRegression
    
    proxies = {}
    for feature in features_df.columns:
        if feature == protected_attribute:
            continue
        
        X = features_df[[feature]]
        y = features_df[protected_attribute]
        
        model = LogisticRegression()
        model.fit(X, y)
        accuracy = model.score(X, y)
        
        # If a single feature predicts protected with >60% accuracy
        # (better than a 50/50 coin flip), it's a strong proxy
        proxies[feature] = {
            'predicts_protected_accuracy': accuracy,
            'is_strong_proxy': accuracy > 0.60,
        }
    
    return proxies

If Goldman had run this test over its features, it would have seen that features like "industry of employment," "credit history length," and "average transaction patterns" predict gender with > 70-80% accuracy. That's a clear proxy signal.

Technique 3: Mitigation after detection

Detecting proxies isn't enough. You have to mitigate.

Options:

Option A: Drop the feature (drastic, loses predictive signal).

Option B: Re-balance the training data so the correlation dilutes.

Option C: Adversarial debiasing — train the model with an extra loss term that penalizes predictions correlating with protected attributes. Technically complex but effective.

Option D: Post-processing — adjust outputs after prediction to equalize approval rates across groups. Controversial but used in some regulated industries.

Goldman could have applied any of these. It didn't (at least not before launch).

Technique 4: Explainability for customers

Missing: when an applicant got a low credit limit, there was no explanation. "The algorithm decided" is not an explanation.

Implementation: for each decision, generate an explanation using techniques like SHAP or LIME that show which features most affected the decision. That lets you:

• The applicant understands why.
• The applicant can request a review if the explanation looks like it rests on a problematic feature.
• External auditors can see the reasoning.

Goldman, post-controversy, implemented something like this. Before, no.

Technique 5: Joint application option

Missing: accepting joint applications where finances are shared (married couples, in community property states).

Apple Card didn't accept joint applications initially. Each spouse applied as an individual, which is what triggered the bias.

After the controversy, Apple/Goldman added a joint accounts option. But the initial system didn't have it — a sub-optimal design from launch.

---

The structural lesson

Removing explicit features for protected attributes does NOT eliminate discrimination when proxies are available. Disparate impact testing of the output is what detects the problem, not inspecting the input features.

This is the principle of proxy discrimination, central to modern compliance:

• The US Fair Housing Act and Equal Credit Opportunity Act prohibit discrimination by explicit proxy.
• Recent legal cases (in the US and EU) are applying this logic to ML.
• The EU AI Act includes disparate impact as an evaluation criterion.

For your work: always test the model's output by demographic subgroup, not just inspect the inputs.

---

Traps and common mistakes in interpreting this case

1. "If I don't include gender/race, I can't discriminate"

False, as Apple Card demonstrated. Proxies make the problem invisible when you inspect inputs. Output testing is the way to detect it.

2. "This only applies to credit scoring"

False. Proxy discrimination shows up in:

• Hiring (Amazon — the word "women's" as a proxy).
• Insurance pricing.
• Healthcare resource allocation.
• Recommendation systems.
• Content moderation.
• Any system whose decisions affect people.

3. "The model was technically correct"

Possibly, yes. And that's not a defense. The model captured real patterns in the world (including its historical bias). The problem isn't that the model is technically wrong — the problem is that reproducing biased historical patterns in future decisions perpetuates the bias. The system's ethical job is to not replicate the pattern, not to copy it faithfully.

4. "NYDFS found no violation, so everything's fine"

NYDFS found no legal violation. That means the system didn't violate the specific laws in force in NY. It does not mean the system was ethical, fair, or well designed.

Compliance ≠ ethics (back to capsule 01). Passing the legal minimum is not the same as being right.

---

Self-check

1. How can a model discriminate by gender if "gender" is not one of its features?

Via proxies: seemingly neutral features that are statistically correlated with gender.

Examples:

• Credit history length: older women typically have shorter histories of their own because of historical "primary cardholder" patterns.
• Industry of employment: industries have an uneven gender distribution.
• Transaction patterns: purchase types correlate with demographics.

The model learns that these features predict "a good candidate for a high credit limit." Indirectly, it discriminates by gender.

Solution: measure disparate impact of the output, not just inspect the inputs.

2. What's the difference between disparate treatment and disparate impact?

Disparate treatment: explicitly different treatment based on a protected attribute. "If she's a woman, subtract 10 points" — plainly illegal.

Disparate impact: seemingly neutral treatment that results in unequal outcomes by group. For example, a requirement of "10 years of continuous employment history" — neutral on the surface, but it disproportionately excludes women because of maternity breaks.

ML produces far more disparate impact than disparate treatment. Modern regulation (US, EU) covers both. But detecting disparate impact requires statistical analysis — you can't see it by inspecting code.

Practical implication: your system can have zero intentional discrimination and still produce illegal disparate impact. Output testing is what detects it.

3. Why is "the model captured real patterns" not a valid defense?

Because the real patterns of the past include systemic biases (historic residential segregation, historic gender discrimination in employment, etc.). A model that faithfully captures those patterns perpetuates the bias in future decisions.

The purpose of a new AI system isn't to reproduce the past — it's to make decisions for the future. If the past was biased, copying it faithfully perpetuates it.

Analogy: an insurance pricing system that learns "ZIP code X has more claims" technically captures a real pattern. But if ZIP code X correlates with a racial minority because of historic segregation, the system perpetuates redlining.

Ethical decision: sometimes you have to break the historical pattern, not replicate it.

4. How would you detect proxy variables in your own system?

Procedure:

1. Identify protected attributes: gender, race, age, ethnicity, religion, national origin, disability.
2. For each feature in the model, compute how well it predicts each protected attribute.
3. If a single feature predicts a protected attribute with > 60% accuracy (significantly better than random), it's a strong proxy.
4. Decide: drop the feature, mitigate via re-weighting, apply adversarial debiasing, or document it and monitor it closely.

Additionally: measure disparate impact of the complete model's output, not just individual features. Combinations of features can be a proxy even when no single one is.

Tools: SHAP, LIME, Fairlearn (Microsoft), AIF360 (IBM), WhatIfTool (Google).

---

Summary and next step

• Apple Card granted dramatically lower credit limits to women vs. their partners with identical finances (DHH 20x, Wozniak 10x).
• Goldman Sachs did not use gender as a feature — the bias came in via proxies: individual credit history, individual income, industry of employment, etc.
• NYDFS investigated but found no legal violation. Compliance ≠ ethics.
• Cost: regulatory investigation, operational changes, reputational damage, accelerated regulation.
• Structural lesson: removing explicit features does not eliminate discrimination. Disparate impact testing of the output is what detects proxy discrimination.

Checkpoint: you should be able to distinguish disparate treatment from disparate impact, and explain why a "technically correct" model can be discriminatory in its outcomes.

Bridge to the next capsule: capsules 02-04 were the case studies. Capsule 05 synthesizes them: the 4 dimensions of the cost of ignoring ethics in AI — legal, reputational, technical, and human. You'll walk away with a framework for arguing (to a PM, a C-level exec, a client) why investing in an ethics process is positive ROI, not overhead.

---

Resources

1. DHH Twitter thread (Nov 2019) — where it went public.
2. NYDFS Apple Card investigation report (2021) — the official conclusions.
3. Fairlearn (Microsoft) — fairness toolkit — open source.
4. AI Fairness 360 (IBM) — comprehensive toolkit.
5. Equal Credit Opportunity Act (ECOA) — US legal reference.

---

Next: 05-the-four-dimensions-of-cost.md — The 4 dimensions of cost: legal, reputational, technical, human.

Capsule 04 of 08 — Module 1 — AI Ethics & Compliance Guide