Hallucination as a failure mode

Overview

Of all the failures an AI component can have, there's one that resembles nothing you knew before integrating an LLM: the model invents a datum with total confidence. It doesn't fail loudly, it doesn't throw an exception, it doesn't return null or an error code. It generates a fluent, well-worded response, with the same poise as always, and that datum is false. In Mercado, the support agent tells a customer "your order ships with tracking TRK-4521" and that tracking doesn't exist; the model produced it because it sounded like a tracking number, not because it looked it up. This lesson installs the principle for treating it: hallucination is a failure mode, not a quirk to tolerate, and it's contained like any other failure —with an architectural defense—: verify every factual claim against a deterministic source of truth before serving it, and degrade when it can't be verified.

In lesson 2 you classified hallucination as the silent failure par excellence: the one no try/except catches. Here we go down to the concrete defense. You're going to see, executed, Mercado's support agent answering questions about orders —sometimes citing the correct datum, sometimes inventing it— and a verification against the source of truth (the real order record) that separates the real from the invented: what matches is served, what the model invented is blocked and degraded to an "I can't confirm it, let me connect you with an agent."

Connection with the module. Lesson 2 gave you the taxonomy; this one takes the content family —the silent failure— and builds its defense. It's the counterpart of lessons 4, 5, and 6, which defend the noisy failures (availability): here, since the failure doesn't announce itself, the defense can't be "catch an exception," it has to be "actively verify." The boundary with AI Engineering is hard and here it matters a lot: we are not going to teach you to reduce hallucinations by building a good RAG, tuning the prompt, or fine-tuning —that's the AI Engineering ecosystem—; we're going to treat them as a failure mode the architecture must contain, assuming the model will hallucinate and designing so that failure doesn't reach the customer.

An analogy: the employee who, when they don't know, escalates instead of inventing

Go back to the customer-service employee of lesson 1, but look closely at their best virtue. A customer comes and asks: "when exactly does my order arrive and with what carrier?". The good employee does something very specific before answering: they look it up in the system. If the system shows them the status and the tracking, they give them confidently. If the system doesn't have the datum, or the order doesn't appear, the employee doesn't invent it to look good: they say "let me check with a colleague" or escalate to the supervisor. The difference between this employee and a bad one isn't that they know more; it's that they know when not to trust their own memory and have the discipline to verify before asserting, and the humility to escalate when they can't.

The bad employee, by contrast, always answers. Asked about an order they can't find, they respond "it arrives Thursday, it's with express shipping" —a datum that sounded reasonable, said with confidence, completely invented—. The customer believes them, stops waiting for Thursday, and when the package doesn't arrive, the problem is bigger than admitting "I don't have it handy, let me confirm it." The bad employee preferred to seem useful over being correct.

Here's the point: an LLM, by default, is the bad employee —it always answers, even when it doesn't know—, and the architecture has to turn it into the good employee. The model doesn't have, on its own, the instinct to "verify before asserting" nor to "escalate when I can't": it generates a plausible response no matter what. That instinct is given to it by the surrounding system: a layer that looks up the source of truth to confirm every factual datum the model asserts, and that degrades —escalates to a human, says "I can't confirm it"— when the datum can't be verified. The model proposes the response; the system confronts it with reality before letting it out. In Mercado, the source of truth is the order record; the verification is the sensible boss reviewing before the employee opens their mouth.

Worked example: verify the claim against the source of truth

We're going to build the defense. We have a deterministic source of truth, ORDERS: the real status of three Mercado orders (their status and their tracking number). The support agent (a stub) answers questions about orders, and in its response it cites a status and a tracking. Sometimes it cites well; sometimes it hallucinates —it invents an order that doesn't exist, or a status or a tracking that don't match reality—. All the responses sound equally confident; that's the trap.

The is_grounded function is the defense: it takes the model's claim (the order, the status, the tracking it asserts) and confronts it with ORDERS. If the order doesn't exist, or the status doesn't match, or the tracking doesn't match, the claim is unanchored from reality —it's a hallucination— and it's blocked. What can't be verified isn't served: it's degraded to a safe response that escalates to a human.

# Lesson 3: HALLUCINATION as a failure mode. The model invents with
# confidence a datum that doesn't exist. The architectural containment: verify
# the model's claim against a deterministic SOURCE OF TRUTH before
# serving it; if it can't be verified, DEGRADE (escalate to a human / don't assert).
# LLM simulated; no network.

# Mercado's source of truth (deterministic): the real status of the orders.
ORDERS = {
    "A-1001": {"status": "shipped",    "tracking": "TRK-77"},
    "A-1002": {"status": "processing", "tracking": None},
    "A-1003": {"status": "delivered",  "tracking": "TRK-88"},
}

# The support agent (stub) answers a question about an order. Sometimes
# it CITES correctly; sometimes it HALLUCINATES: it invents an order, a status, or a
# tracking number that do NOT match the source of truth. Notice: all the
# responses SOUND equally confident.
AGENT_REPLIES = [
    # (order_id, claim_status, claim_tracking, text for the customer)
    ("A-1001", "shipped",    "TRK-77", "Your order is on its way, tracking TRK-77."),
    ("A-1002", "shipped",    "TRK-99", "Your order has shipped, tracking TRK-99."),
    ("A-1003", "delivered",  "TRK-88", "Your order was delivered."),
    ("A-9999", "processing", None,     "Your order A-9999 is being prepared."),
    ("A-1002", "processing", None,     "Your order is being prepared."),
    ("A-1001", "delivered",  "TRK-77", "Your order has already been delivered."),
]

SAFE_FALLBACK = "I can't confirm that right now; let me connect you with an agent."


def is_grounded(order_id, claim_status, claim_tracking):
    # Deterministic verification: the model's claim MUST match the
    # source of truth. A datum the model can't back up isn't served.
    order = ORDERS.get(order_id)
    if order is None:
        return (False, f"order {order_id} does not exist")
    if claim_status != order["status"]:
        return (False, f"cited status '{claim_status}' != real '{order['status']}'")
    if claim_tracking != order["tracking"]:
        return (False, f"cited tracking '{claim_tracking}' != real '{order['tracking']}'")
    return (True, "verified against the source of truth")


served = blocked = 0
print(f"{'order':<9}{'verdict':<10}{'reason':<47}served to the customer")
print("-" * 100)
for oid, cs, ct, text in AGENT_REPLIES:
    ok, reason = is_grounded(oid, cs, ct)
    if ok:
        served += 1
        out = text
        verdict = "SERVE"
    else:
        blocked += 1
        out = SAFE_FALLBACK
        verdict = "BLOCK"
    print(f"{oid:<9}{verdict:<10}{reason:<47}{out}")

print("-" * 100)
print(f"{served} responses served (verified), "
      f"{blocked} hallucinations blocked before reaching the customer.")

What to expect. When you run the file, the output is exactly this:

order    verdict   reason                                         served to the customer
----------------------------------------------------------------------------------------------------
A-1001   SERVE     verified against the source of truth            Your order is on its way, tracking TRK-77.
A-1002   BLOCK     cited status 'shipped' != real 'processing'    I can't confirm that right now; let me connect you with an agent.
A-1003   SERVE     verified against the source of truth            Your order was delivered.
A-9999   BLOCK     order A-9999 does not exist                     I can't confirm that right now; let me connect you with an agent.
A-1002   SERVE     verified against the source of truth            Your order is being prepared.
A-1001   BLOCK     cited status 'delivered' != real 'shipped'     I can't confirm that right now; let me connect you with an agent.
----------------------------------------------------------------------------------------------------
3 responses served (verified), 3 hallucinations blocked before reaching the customer.

Read the table calmly, because each row is a different way of hallucinating and of containing it.

The three served responses are the ones the model could back up: A-1001 (it said "shipped, TRK-77" and that's right), A-1003 (it said "delivered" and that's right), and the second A-1002 (it said "processing" and that's right). In these three, the model's claim matched the source of truth, so the real text reached the customer. Notice: we didn't block the model out of blind distrust; when the model is right and we can prove it, its response passes. The verification isn't a lock that turns off the feature; it's a gate that lets the verifiable through.

The three blocked are the three ways of hallucinating, each different:

• A-1002 "shipped, TRK-99": the model invented both the status (it said "shipped" when it's "processing") and a tracking (TRK-99) that doesn't exist. The verification caught it by the status. Without it, the customer would have believed their order had already shipped with a tracking that tracks nothing.
• A-9999: the model invented a whole order. It doesn't exist in ORDERS. This is the most telling case: the customer maybe didn't even ask about A-9999, or the model mixed up a number, but the result is a datum about a phantom order. The verification blocks it with "order A-9999 does not exist."
• A-1001 "delivered": the model said an order on its way had already been delivered. It's the most expensive error in support: a customer told "it already arrived" when it hasn't stops waiting and doesn't complain. The verification catches it by comparing the cited status with the real one.

In all three, the response that reached the customer was not the hallucination: it was the SAFE_FALLBACK, "I can't confirm that right now; let me connect you with an agent." That's the degradation: when the model asserts something that can't be verified, the system doesn't serve the assertion —it replaces it with an honest response that escalates to a human—. It's exactly the good employee: when they can't confirm, they escalate instead of inventing.

The architectural implication: 3 of 6 model responses were hallucinations, and none reached the customer, not because the model improved —it's the same model, it hallucinates the same—, but because every factual datum it asserted was confronted with the source of truth before going out. The trust in the support agent doesn't come from the model not hallucinating; it comes from the system verifying what the model says.

Going deeper: contain the hallucination without pretending to eliminate it

The distinction that marks the boundary with AI Engineering. There are two very different questions about hallucination, and this guide answers only one. The first is "how do I make the model hallucinate less?" —better prompt, RAG that gives it the right context, fine-tuning, a more capable model—; that's the AI Engineering question, and it's real and important. The second is "how do I design the system so that, when the model hallucinates —because it will hallucinate—, that failure doesn't reach the user?"; that's the architectural question, this module's. The difference is the same as between "make the elevator fail less" (elevator engineering) and "put stairs for when it fails" (building architecture). You can and should do both, but they're different jobs, and confusing them is an expensive mistake: a team that only works on "make it hallucinate less" never reaches zero, and without the architectural containment, the residue —there will always be residue— reaches the customer.

What can be verified and what can't. This lesson's defense —verify against the source of truth— works wonderfully for factual claims about data you have: an order's status, a product's existence, a price, a balance. All those live in a deterministic database you can query. The verification is the confrontation of the model's claim against that datum. But not every claim is verifiable this way: if the model writes an apology text, there's no "source of truth" against which to compare the wording. The design rule that follows: structure the model's response to separate the verifiable from the non-verifiable, and verify the verifiable. In the support agent, instead of letting the model write freely "your order A-1001 ships with tracking TRK-77," you have the model propose the fields (order_id, status, tracking) and the system verifies them and fills in the template with the real data. The model contributes the tone and the intent; the system contributes the facts. This pattern —the model proposes, the system disposes— is exactly what module 6 develops in depth.

Degrading is part of the response, not an error. When the verification blocks a claim, the system doesn't go mute nor throw a raw error at the customer: it degrades to a safe response. And there's a hierarchy of degradations, from best to worst: (1) serve the verified datum directly from the source of truth, skipping the model's text ("your order is in status: processing"); (2) say honestly "I can't confirm that right now"; (3) escalate to a human who can resolve it. The example's SAFE_FALLBACK combines (2) and (3). What's never done is serve the unverified claim. Note the connection with lesson 1: this is the employee who escalates to the supervisor, and with lesson 5: it's a graceful degradation —a worse but valid response instead of a fall or a lie—.

The asymmetric cost: why you always verify. A colleague might say "the model gets 90% of the trackings right, do we really have to verify them all?". The answer is in the asymmetric cost. Verifying a claim against the database costs one query —microseconds, an index—. Serving a hallucination costs the whole chain of decisions the customer makes on a false datum: they wait for a package that doesn't arrive, don't complain in time, lose their return window, go to the competitor. The verification is cheap and the silent failure is expensive, so the verification always goes, not just when "you suspect." The model's hit rate decides how much you're going to degrade (if it hallucinates 10%, you'll degrade 10%), not whether you verify.

Anatomy of the containment. It's worth fixing the shape, because you'll apply it to every factual datum an AI component asserts:

    customer question
            │
            ▼
   ┌─────────────────┐
   │  agent (LLM)    │   PROPOSES a response with factual claims
   │  (stub)         │   (order_id, status, tracking). Can HALLUCINATE.
   └─────────────────┘
            │  proposed claim (NOT verified)
            ▼
   ┌─────────────────┐        ┌───────────────────────┐
   │   is_grounded   │◄──────►│  ORDERS (deterministic│
   │  (verification) │        │   source of truth)    │
   └─────────────────┘        └───────────────────────┘
        │            │
     MATCHES     NO MATCH
        │            │
        ▼            ▼
  serve the datum  degrade (escalate to a human / "I can't confirm it")

Common mistakes

Trusting the hallucinated output because it sounds confident. What happens: the agent responds "your order was delivered yesterday, tracking TRK-4521" with impeccable wording, and the system serves it as is because it threw no exception and "looks good." The datum was invented. Why it happens: the model's fluency is confused with correctness, and since the failure is silent, nothing stops it. How to spot it: your system serves data the model asserts without confronting them against a source of truth. How to fix it: verify every factual claim against the real datum before serving it, and degrade if it doesn't match. The example executes it: 3 hallucinations blocked of 6 responses.

Pretending to eliminate the hallucination instead of containing it. What happens: the team spends months on a perfect prompt and a tuned RAG to lower the hallucination rate, lowers it from 10% to 2%, and declares the problem solved —it removes the verification because "it barely hallucinates now"—. The remaining 2% now reaches the customer with no barrier. Why it happens: reducing the frequency (AI Engineering work, valuable) is confused with containing the failure (architecture work, indispensable). How to spot it: your only defense against hallucination is "we made it hallucinate little"; there's no verification that catches the residue. How to fix it: reduce the frequency and contain the residue with verification —both, because no model reaches zero—. The containment always goes, no matter how good the model is.

Verifying some claims and not others. What happens: the system verifies the order status against the database, but lets the model freely invent the estimated delivery date or the carrier's name, because "those are details." The customer gets an invented date. Why it happens: what's easy to verify is verified and the rest is let through, without realizing that a single invented datum in an otherwise correct response is already a served hallucination. How to spot it: in your response there are factual fields the model fills in and the system doesn't confront against any source. How to fix it: structure the response so that all the factual fields come from the source of truth (the system fills them in), and the model contributes only the tone and the intent. What the system can't verify, it doesn't assert. It's the "the model proposes, the system disposes" pattern of module 6.

Exercises

Exercise 1 — The three ways of hallucinating. In the example, the three blocked responses hallucinated in different ways. For each, say what the model invented and why the verification caught it: (a) A-1002 "shipped, TRK-99"; (b) A-9999; (c) A-1001 "delivered". Then propose a fourth type of hallucination the example's verification wouldn't catch, and how you'd extend is_grounded to cover it.

Exercise 2 — The model proposes, the system fills in. The example's design lets the model cite the status and tracking, and then verifies them. An alternative design has the model propose only the order_id and the intent (answer about the status), and the system fills in the status and tracking from ORDERS. Write (in pseudocode or Python) that alternative design, and explain why it eliminates at the root the possibility of an invented status or tracking reaching the customer.

def answer_about_order(order_id):
    # The model NO LONGER cites factual data; it only identifies the order.
    # The system fills the facts from the source of truth.
    order = ORDERS.get(order_id)
    if order is None:
        return SAFE_FALLBACK   # the order doesn't exist: degrade
    status = order["status"]        # REAL datum, not from the model
    tracking = order["tracking"]    # REAL datum, not from the model
    if tracking:
        return f"Your order {order_id} is in status: {status}, tracking {tracking}."
    return f"Your order {order_id} is in status: {status}."

Exercise 3 — When verifying isn't enough. This lesson's defense assumes there's a deterministic source of truth against which to confront the claim. Give an example of a Mercado AI feature where the model's claim can't be verified against a structured source of truth, and describe what other architectural defense you'd use in its place (hint: think about degradation and human review).

Summary and next step

In this lesson you installed the principle for treating hallucination: it's a failure mode contained with an architectural defense —verify every factual claim against a deterministic source of truth and degrade when it can't be verified—, not a quirk that's tolerated. You saw it with the good employee who looks it up in the system and escalates when they don't know, versus the bad one who invents to look good, and you measured it: Mercado's support agent produced 3 hallucinations of 6 responses —an invented status, a nonexistent order, a falsely advanced status— and none reached the customer, because every factual datum was confronted with the real order record and the non-verifiable was degraded to "I can't confirm it, let me connect you with an agent." And you saw the hard boundary with AI Engineering: here we don't reduce hallucination (that's RAG/prompt/fine-tuning, another ecosystem); we contain it, assuming the model will hallucinate.

Before moving on you should be able to: explain why hallucination is a silent failure no try/except catches; verify a claim against a source of truth and degrade when it doesn't match; distinguish "reducing hallucination" (AI Eng) from "containing hallucination" (architecture); and recognize when a claim isn't verifiable and what defense to use in its place.

Lesson 4 changes families: from the silent failures (content) to the noisy failures (availability). You're going to see, executed, what happens when the model is down, slow, or rate-limited —depending on an external API with quotas— and that family's first defense: the timeout, so as not to wait forever for an API that hung. You're going to measure how an 800 ms timeout cuts the wait of a 30-second hang, and how the system, instead of blocking, falls to the fallback and responds. The mechanics of the timeout in depth live in the resilience guide; here we apply it to the model.

Resources

• Chip Huyen, AI Engineering (O'Reilly, 2024). The chapters on reliability and on RAG treat hallucination as a failure mode of foundation models and the strategies to detect and contain it (grounding, verification) —the conceptual frame of this lesson—. In English.
• Anthropic, Claude documentation — docs.anthropic.com. The guides on reducing hallucinations and on asking for verifiable outputs (with citations, with structured output) show, at a conceptual level, how to structure the model's response so you can confront it with a source —without fixing a model version—. In English.
• Martin Fowler and Bharani Subramaniam, "Emerging Patterns in Building GenAI Apps" — martinfowler.com/articles/gen-ai-patterns. The guardrails and output-verification patterns around an AI component frame the containment of hallucination as design. In English.
• To reduce (not just contain) hallucination with RAG, prompting, or fine-tuning, the destination is the AI Engineering ecosystem —outside the scope of this guide, which treats hallucination as an architectural failure mode—.