The Right to Explanation for LLMs

Description

GDPR Art. 22 requires "meaningful information about the logic involved" in automated decisions. For classical models (random forests, regressions, simpler ML), explaining is manageable. For LLMs it's fundamentally harder.

An LLM has billions of parameters. Its "reasoning" emerges from gradient descent over massive data. There are no obvious feature importances, no decision tree path, no linear coefficients to show.

This capsule gives you the practical strategies for meeting the right to explanation with LLMs, while being honest about the limitations.

By the end you'll be able to:

• Recognize what kind of explanation is feasible with LLMs
• Implement logging and observability to reconstruct decisions
• Design human-in-the-loop as a complement
• Communicate meaningful, if imperfect, explanations to users

The fundamental problem

A classical model (Random Forest):
  Input → Feature extraction → Tree traversal → Decision
  Explanation: "Used these features with these importances"

An LLM (RAG system):
  Input → Tokenize → Embed → Retrieve chunks → Prompt construction →
  → Transformer layers (hundreds of millions of parameters) →
  → Token-by-token generation → Output
  Explanation: ???

The LLM has no separable components you can point at. Its "decision" emerges holistically.

Explanation strategies for LLMs

Strategy 1: Document the process (not the internal reasoning)

You can't explain why the LLM decided X. But you can explain the process that led to that decision:

Your question: "Can I expand my coverage to Brazil?"

The process:
1. We searched our knowledge base and found 4 relevant documents
2. The documents indicate that international coverage is available
   to customers on the premium plan after 12 months
3. Your current plan: standard, tenure 6 months
4. Decision: not currently eligible

Documents consulted:
- International coverage policy (2024)
- Plan upgrade requirements
- General terms of service

Would you like a human agent to review this?

You didn't explain the LLM — you explained the process and the inputs. That's defensible under GDPR.

Strategy 2: Exhaustive logging

To be able to explain later, you record everything during:

# When we process a query
log = {
    "session_id": session.id,
    "user_id": user.id,
    "timestamp": now(),
    "query": original_query,
    "retrieved_chunks": [
        {"id": c.id, "score": c.score, "source": c.source}
        for c in chunks
    ],
    "prompt_template_version": "v3.2",
    "full_prompt": prompt,
    "model_used": "gpt-4o-mini",
    "model_version": "2024-07-18",
    "model_parameters": {"temperature": 0.1, "max_tokens": 500},
    "raw_response": response,
    "extracted_decision": decision,
    "decision_confidence": confidence,
    "tools_called": [tool_calls],
}
db.log_interaction(log)

When a user asks for an explanation 2 months later, you have complete traceability.

Strategy 3: Constrained outputs with reasoning fields

Structure the LLM's output to include explicit reasoning:

prompt = """
Evaluate the user's request and return JSON with:
- "decision": "approve" | "reject" | "needs_review"
- "key_factors": a list of the 3-5 factors that most affected the decision
- "supporting_documents": IDs of the consulted documents that support it
- "confidence": 0.0-1.0
- "uncertainty_areas": what was ambiguous

Request: {user_query}
Context: {retrieved_chunks}
"""

response = call_llm(prompt, response_format="json")
# response.key_factors is what we show the user

The LLM explains itself in its output. It isn't ground truth about its internal reasoning, but it is defensible.

Strategy 4: Pre-written decision templates

For critical, common decisions, use templates instead of free-form LLM generation:

# For a credit denial, use a fixed template
def credit_denial_explanation(scoring_result):
    primary_factor = identify_main_factor(scoring_result)
    return f"""
Your application was not approved at this time.

Main factor: {primary_factor['name']}
- Your value: {primary_factor['user_value']}
- Typical threshold: {primary_factor['typical_threshold']}

Other factors considered:
{format_factors(scoring_result.factors)}

You can improve your profile for future applications:
{recommendations(scoring_result)}

[Request human review]
"""

Templates give you consistency and defensibility. The LLM can help generate the explanation, but it isn't the only path.

Strategy 5: Human-in-the-loop as a complement

For critical decisions, a human reviews and signs off:

1. The LLM generates an initial decision + a draft explanation
2. It's routed to a human reviewer for significant decisions
3. The human approves/modifies/rejects
4. The final explanation includes: the AI suggestion + the human review notes
5. It's logged as a "human-supervised" decision

This transforms a "solely automated" decision into a "human-supervised" one, taking it out of Art. 22's strict scope.

What you CANNOT do

Don't invent explanations

❌ Bad: "The system decided X because it considered Y" — but the system never exposed Y.

Why it's a problem: when the regulator inspects and sees that your explanation doesn't match the real logs, they'll sanction you.

Don't use attribution methods without understanding their limitations

LIME, SHAP, attention visualization — tools that promise to explain LLMs. Significant limitations:

• Attention weights ≠ causal importance
• LIME's local approximations are fragile
• SHAP for LLMs is computationally expensive and approximate

If you use them, document the limitations explicitly.

Don't claim "interpretable AI" without being accurate

"Our AI is completely interpretable" → you get sued, your LLM turns out to be complex, reputational damage.

UX design for explanations

Layers of information

Different users want different levels:

Level 1 (default): "Your application was rejected because of [main factor]."

Level 2 (click "more info"): 
"Other factors considered: [list]. Compared to typical thresholds: [comparison]."

Level 3 (click "technical details"):
"Documents consulted: [list with IDs]. Model version: X. 
Decision timestamp: Y. Confidence: Z."

Level 4 (admin/audit): full logs

Each layer is progressively more technical. A normal user sees L1; a lawyer/auditor accesses L3-4.

Common traps

Trap 1 — Assuming "we have AI" → "we have interpretability." Cars and planes work; you can't explain every bolt. Accept the opacity while you document the process.

Trap 2 — Logging with no retention policy. You log everything, and then your DB fills up. Define a TTL: detailed logs 90 days, summaries indefinitely.

Trap 3 — Rigid templates that don't cover edge cases. "If scoring < threshold" but there are 50 different reasons. The template must be dynamic or have a general fallback.

Trap 4 — A human reviewer with no SLA. The user asks for human review, and it arrives 60 days later. Maybe legally defensible, but terrible UX.

Trap 5 — Privacy violations in the explanations. Your explanation includes other users' data: "compared to similar users, X." If it leaks personal info, that's a violation.

Exercise

Your AI assistant rejects a plan upgrade request. Design the explanation:

1. Level 1 (default, 2-3 sentences)
2. Level 2 (more info)
3. Exhaustive logging (what you store)
4. The procedure if the user requests human review

{
    "request_id": uuid,
    "user_id": user.id,
    "timestamp": now,
    "request_type": "upgrade_plan",
    "input_data": {
        "current_plan": "Standard",
        "requested_plan": "Premium",
        "tenure_months": 6,
        "payment_status": "current",
        "usage_active": True,
    },
    "evaluation_rules_version": "v2.3",
    "decision": "reject",
    "key_factor": "insufficient_tenure",
    "all_factors": [...],
    "explanation_template_used": "tenure_insufficient_v1",
    "ai_assistance": False,  # rule-based, no LLM
    "next_eligible_date": "2026-XX-XX",
}

Summary

You learned:

• ✅ Why LLMs are fundamentally less explainable
• ✅ 5 practical strategies: process, logging, constrained outputs, templates, human-in-the-loop
• ✅ What you CANNOT do (invent, misuse attribution methods, overclaim)
• ✅ Layered UX (levels 1-4)
• ✅ The traps: logging without retention, rigid templates, privacy violations in explanations

Checkpoint: if you can design an explanation for a real LLM system that meets Art. 22 without inventing anything, you're ready.

Next capsule

04 — Data Minimization applied to AI. Another GDPR principle with real tensions: ML wants more data, GDPR wants the minimum. Let's resolve the conflict.

Resources

1. Edwards & Veale "Slave to the algorithm" — an analysis of the right to explanation.
2. Wachter et al. "Counterfactual Explanations" — an explanation method.
3. GDPR Recital 71 — context for Art. 22.
4. Anthropic's Claude — Constitutional AI — an example of design for explainability.