Module 7: Agent Security and Reliability

6. Hallucination and output verification

Description

By the end of this lesson you'll be able to tell apart the three types of hallucination happening in an agent with tools — which aren't the same problem and don't get defended against the same way — you'll know how to structure the agent's output so the fact travels separately from the wording, and you'll have set up the pattern that verifies every factual data point against what the tool genuinely returned, with the exact comparison done in a deterministic node and not in another model's judgment.

This matters because it changes the enemy. Lessons 2 through 5 dealt with someone trying to get your agent to do something improper. Here there's nobody: the filter is in place, the content is isolated, permissions are trimmed, approval works — and the agent still tells the customer their order arrives Thursday the 12th, a fact lookup_order never returned. To that customer, the difference between an attack and a model slip-up is exactly zero: you promised them something false, and Thursday the 12th they're going to be waiting. An agent that makes up data isn't insecure in lesson 2's sense; it's unreliable, and that pulls it out of production just as fast.

Connection to the module: lessons 4 and 5 put limits on what the agent can do. This one puts limits on what the agent can say, which is the surface no read-only credential covers. It reuses two pieces you already have: structured output from Module 2, lesson 6, which up to now you used to make the response easy to process and today becomes the mechanism that makes a data point verifiable; and lesson 2's Guardrails node, now applied to output instead of input. Lesson 7 closes the module with the ability to reconstruct what happened when any of these defenses fell short.

The salesperson who thinks they remember

Think of someone who's staffed a store counter for years. They know the catalog, they know the customers, and they answer fast — which is their virtue. A customer arrives and asks how much a model that just got added costs. This person doesn't have it memorized, but they know the line, they know what similar models are worth, and they answer with total naturalness: "that one's at $890."

They didn't lie. They're not being negligent in the sense we usually give that word. Their brain did what it always does: complete the pattern. Models in that line are worth between $850 and $920, so $890 is a perfectly reasonable answer. The problem is the real price is $1,150, and the customer just left with a number in their head the store's going to have to honor or take back, and neither is good.

Notice something important: that person answered with the same confidence as when they actually know the fact. They didn't say "I think," they didn't hesitate. The confidence something's said with has no relation to whether it's true. That disconnect is exactly what makes a model's hallucination dangerous.

A hallucination is a plausible output not backed by the data. It isn't a random error or a system failure: it's the model's normal mechanism — completing the most probable thing given the context — applied to a gap where there was no information. And that's why hallucinations are never absurd. They're always believable, with the right format, with the right tone. A model doesn't make up a reference number like XZ%%2; it makes up TT-2026-4482, which looks exactly like yours.

In an agent with tools, this takes three distinct forms, and it's worth separating them because they get defended against differently:

Type 1 — The made-up fact in the response. The tool returned partial information and the agent filled in the rest while drafting. It's the most common case and the one in the example above: lookup_order returned status: in_transit with no date, and the agent wrote "arrives Thursday." It gets defended against by verifying the response against the tool's output, which is the bulk of this lesson.

Type 2 — The made-up parameter in a call. The agent calls a tool with a value nobody provided: an orderId the customer never said, an amount appearing nowhere. You already saw the case in Module 4, lesson 5, with $fromAI("refundAmount") with no description. It gets defended against with the tool's contract, with fixed parameters (lesson 4), and, for sensitive stuff, with human approval (lesson 5), where the made-up reason jumps out.

Type 3 — The claim with no tool behind it. The agent answers a question that required checking a system without checking anything, because the model "knows" how those things usually go. It's the most treacherous one, because the trace shows no wrong call — it shows no call at all. It gets detected by comparing what the response claims against what tools got called that turn.

All three share a common origin, worth naming because it orients the defense: the model prefers responding over admitting it doesn't know. Not because it "feels embarrassed," but because in the material it was trained on, a complete answer is overwhelmingly more frequent than an "I don't have that data." Asking it in the prompt to say "I don't know" helps — and you should do it — but you're asking it to go against its statistical tendency, and that works most of the time, not all of it. Once more the module's distinction: layer, not guarantee.

Worked example

The concrete case, with the complete trace, to see where the false fact slips in.

# System: order_specialist with a trimmed lookup_order
#          (lesson 4's hardened version)

Customer's message:

Hi, my order 4521, when does it arrive?

Execution trace:

1. order_specialist → calls lookup_order { orderId: "4521" }

2. lookup_order → returns:
   {
     "order_id": "4521",
     "customer_id": "CUS-8842",
     "status": "in_transit",
     "created_at": "2026-07-14",
     "shipped_at": "2026-07-19",
     "carrier_tracking_code": "TT-99381042"
   }

3. order_specialist → responds:
   "Hi there! Your order 4521 shipped on July 19th and is on the
    way. Based on the carrier's estimated time it should arrive
    Thursday the 23rd. You can track it with code TT-99381042."

Read step 3's response carefully and separate what's true from what isn't:

ClaimWhere did it come from?
"order 4521"tool's order_id
"shipped on July 19th"tool's shipped_at
"is on the way"tool's status: in_transit
"code TT-99381042"tool's carrier_tracking_code
"should arrive Thursday the 23rd"Nowhere

Four of five claims are correct and verifiable. The fifth is made up, and it's — of course — the only one the customer cares about. It's the answer to their question.

And look at how well-built the lie is. "Thursday the 23rd" is coherent: the order shipped on the 19th, four days in transit is plausible, and July 23rd, 2026 does in fact fall on a Thursday. The model didn't throw out a random number; it reasoned about what it knows about shipping in general and produced the most reasonable estimate. If you, as a person, had to guess, you'd say something similar.

The problem isn't that the estimate is bad. The problem is the customer isn't going to read it as an estimate. They're going to read "Thursday the 23rd" as a TuTienda commitment, and Friday the 24th they're going to write in angry. And if order_specialist also has create_ticket connected, there's going to be a ticket documenting that the store promised a date nobody promised.

Now, the question that orders the defense: how does a workflow detect that phrase was extra? There's no error. The execution's green, the tool responded correctly, the agent used the data correctly in four of five cases. The only way to detect it is comparing, field by field, what the response claims against what the tool returned. And to be able to compare field by field, you first need fields — which is exactly the problem with a free-text paragraph.

Separating the fact from the wording

Here's the design change making everything else possible, and it's simpler than it looks: the agent's response stops being a paragraph and becomes an object with two zones. A facts zone, with one field per factual data point, and a wording zone, with the text going to the customer.

Think of it as an invoice. An invoice has amounts in their line, each in its own box, and separately has a notes field where you can write prose. Nobody audits an invoice by reading the notes: the boxes get audited. If the amounts only existed inside a paragraph — "we charged you approximately twelve hundred for the three items" — there'd be nothing to reconcile.

The same thing happens with the agent's output. A paragraph can't be verified; a field can.

# Node: Structured Output Parser connected to order_specialist
#
# JSON Schema:
{
  "type": "object",
  "properties": {
    "order_id":       { "type": "string" },
    "status":         { "type": "string",
                        "enum": ["pending", "in_transit",
                                 "delivered", "canceled"] },
    "shipped_at":     { "type": ["string", "null"] },
    "eta_date":       { "type": ["string", "null"],
                        "description": "Delivery date ONLY if the
                          lookup_order tool returned one. If the tool
                          did not return a delivery date, this MUST
                          be null. Never estimate it." },
    "tracking_code":  { "type": ["string", "null"] },
    "message_to_customer": { "type": "string",
                        "description": "The reply text. It must not
                          state any fact that is not present in the
                          fields above." },
    "facts_source":   { "type": "array", "items": { "type": "string" },
                        "description": "Names of the tools whose
                          output supports the facts above." }
  },
  "required": ["order_id", "status", "message_to_customer",
               "facts_source"]
}

Three decisions in this schema deserve explanation, because they're what make it work:

eta_date explicitly accepts null. If the field were required and of type string, you'd be forcing the model to make up a date — the schema doesn't leave it any other way out. A field that can be null gives the model a correct way to say "I don't have that data," which is exactly what you need it to be able to do.

status is an enum. It isn't free text. The model can't write "probably delivered" or "almost there"; it has four possible values and none more. Every time you can close a vocabulary, close it: it's the cheapest way to eliminate an entire class of inventions.

facts_source forces declaring the origin. It's a field that often gets skipped and that does double duty. It's useful for verifying type 3 — if the response claims things and facts_source comes back empty, the agent answered without checking anything — and, on top of that, the mere fact of having to declare the source changes the model's behavior: while writing the response it already knows it's going to have to say where it got each thing.

And now the verification. Another model doesn't do this. It's tempting to connect a second agent whose system prompt says "check if this response is backed by this data and answer valid or invalid," and it's a design mistake: you'd be verifying a probabilistic output with another probabilistic output, and when both get it wrong at the same time — which happens, because they tend to get it wrong in the same gray areas — you have nothing. The comparison is arithmetic, and it goes in a Code node.

# Node: Code — Name: validate_agent_output
# (after the agent, before responding to the channel)

// We compare, field by field, what the agent CLAIMS against what
// the tool RETURNED. There's no judgment or interpretation here:
// either the values match exactly, or they don't.

const claimed = $input.first().json;                 // agent's output
const actual  = $('lookup_order').first().json;      // tool's real output

const violations = [];

// 1. The identifier has to be the same. If the agent talks about
//    another order, everything else is meaningless.
if (String(claimed.order_id) !== String(actual.order_id)) {
  violations.push(
    `order_id doesn't match: agent="${claimed.order_id}" ` +
    `tool="${actual.order_id}"`
  );
}

// 2. The status has to be literally what the tool returned.
if (claimed.status !== actual.status) {
  violations.push(
    `status doesn't match: agent="${claimed.status}" ` +
    `tool="${actual.status}"`
  );
}

// 3. Shipping date: if the agent claims one, it has to be the
//    same. If the tool didn't return one, the agent can't have one.
if (claimed.shipped_at && claimed.shipped_at !== actual.shipped_at) {
  violations.push(`shipped_at made up or altered: "${claimed.shipped_at}"`);
}

// 4. THE EXAMPLE'S RULE. TuTienda's tool doesn't return an
//    estimated delivery date — that field doesn't exist in our view.
//    Therefore, any non-null value here is an invention.
if (claimed.eta_date !== null && claimed.eta_date !== undefined) {
  violations.push(
    `eta_date made up: the agent claims "${claimed.eta_date}" ` +
    `and lookup_order returns no delivery date`
  );
}

// 5. The tracking code: exact or nothing. A code with one digit
//    changed is worse than none, because the customer is going to
//    paste it into the carrier's website.
if (claimed.tracking_code &&
    claimed.tracking_code !== actual.carrier_tracking_code) {
  violations.push(`tracking_code doesn't match: "${claimed.tracking_code}"`);
}

// 6. Type 3: claiming facts without having checked anything.
if (!Array.isArray(claimed.facts_source) ||
    claimed.facts_source.length === 0) {
  violations.push("the agent claims facts with no declared source");
}

// 7. A text check, deliberately simple: that the message to the
//    customer contains no dates that weren't validated. It doesn't
//    try to understand the text — it looks for a date pattern and
//    verifies it corresponds to a field we did verify.
const datePattern = /\b(January|February|March|April|May|June|July|August|September|October|November|December) (\d{1,2})\b/gi;
const datesInText = claimed.message_to_customer.match(datePattern) || [];
if (datesInText.length > 0 && !claimed.shipped_at) {
  violations.push(
    `the message mentions dates (${datesInText.join(", ")}) ` +
    `with no validated date field`
  );
}

return [{
  json: {
    ...claimed,
    validation_passed: violations.length === 0,
    violations,
  }
}];

And the branch deciding what to do with the result:

# Node: IF — Name: output_is_valid
#   Condition: {{ $json.validation_passed }} is true
#
#   [true]  ─► respond to the customer with message_to_customer
#
#   [false] ─► Switch by severity:
#                 · retry with feedback   (1 time)
#                 · safe degraded response
#                 · escalate to a person
#              and ALWAYS: Google Sheets → hallucination_log

What to expect. Run the order 4521 example against this validator. The agent produces:

{
  "order_id": "4521",
  "status": "in_transit",
  "shipped_at": "2026-07-19",
  "eta_date": "2026-07-23",
  "tracking_code": "TT-99381042",
  "message_to_customer": "Hi there! Your order 4521 shipped on July 19th...",
  "facts_source": ["lookup_order"]
}

Check 4 trips: eta_date comes in with "2026-07-23" and lookup_order doesn't return that field. validation_passed comes out false, with the violation logged. The response never reaches the customer. And notice the detection didn't depend on any model noticing anything: it depended on an if comparing a value against null.

What to do when validation fails

Three paths, and picking wrong here ruins the layer.

Retry with feedback — exactly once. The agent gets called again, adding to the context specifically what failed:

# Fragment added to the retry's input
Your previous response was rejected by validation:
- eta_date made up: you claimed "2026-07-23" and lookup_order
  returns no estimated delivery date.

Answer again. If you don't have a piece of data, the field goes to
null and the message to the customer shouldn't mention it. It's
correct and expected to tell the customer we don't have that data.

What to expect. In most cases the retry produces a clean response: eta_date: null and a message like "Your order 4521 shipped on July 19th and is on the way. I don't have an exact delivery date, but you can track it in real time with code TT-99381042." That response is worse commercially and better in every other way — and, above all, it's true.

The single-retry limit matters: if the second response also fails, don't push it. Retrying in a loop costs calls to the model, adds latency the customer is waiting through in the chat, and if the problem is systematic — your schema asks for something the tool can never give — retrying is never going to fix it.

Safe degraded response. The agent's text gets discarded and the response gets built with a template, using only the fields that did get validated:

# Node: Set — Name: safe_degraded_response
#
# message =
#   "Your order {{ $('lookup_order').item.json.order_id }} is in
#    status: {{ $('lookup_order').item.json.status }}.
#    {{ $('lookup_order').item.json.carrier_tracking_code
#       ? 'Tracking code: ' +
#         $('lookup_order').item.json.carrier_tracking_code + '.'
#       : '' }}
#    If you need more detail, I'm happy to connect you with a
#    team member."

It's cold, it's flat, and it can't be wrong — because the model didn't write it, a template built it from the tool's data. For low-risk actions, it's a perfectly acceptable output.

Escalate. For cases where neither the retry nor the template works: it gets logged, the customer's told a person is going to pick up the case, and the team gets notified. It's the right thing when the field that failed is one of the kind that costs money — an amount, a warranty condition, a legal deadline.

A simple criterion for choosing: retry if the violation looks like a wording slip; degraded if the missing data isn't critical to the customer; escalate if the field that failed is the kind that, wrong, generates a costly complaint.

Second layer: Guardrails on the output

Field-by-field validation covers the data you can compare. There's a stretch it doesn't cover: things the agent shouldn't say, regardless of whether they're true.

For that, lesson 2's Guardrails node comes back, now applied to the final text before it goes out to the channel.

# Node: Guardrails — Name: output_guardrail
#
# Operation:     Check Text for Violations
# Text To Check: {{ $json.message_to_customer }}
#
# Guardrails:
#
#   PII
#     Entities: CREDIT_CARD, EMAIL_ADDRESS, PHONE_NUMBER
#     # Prevents another person's data, dragged in from a tool's
#     # result, from showing up in the response.
#
#   Keywords
#     Keywords: refund approved, lifetime warranty,
#       we'll give you your money back, no cost at all, 100% guaranteed,
#       special discount
#     # Commitments no TuTienda agent can make in writing.
#     # Deterministic and cheap: doesn't need a model.
#
#   Secret Keys
#     # In case a key slipped into an API's result and the
#     # agent repeated it.
#
# Pass branch -> send the response to the channel
# Fail branch -> degraded response + notify the team

This filter is different in nature from the input one, and it's worth being clear about it: at the input you were looking for attacks; here you're looking for commitments and leaks. A perfectly honest agent, with no injection involved at all, can write "we'll give you your money back at no cost at all" simply because it sounds like good customer service. Keywords cuts that off without calling any model.

And about order: field validation goes first, because it's the one that can trigger a useful retry with feedback. The output guardrail goes at the end, over text that already passed validation, as the last gate before the channel.

What validation can't verify

Three honest limits, so you don't oversell this layer for more than it delivers.

It can't verify what it has nothing to compare against. This whole lesson rests on there being a tool output that's the truth. If the customer asks "is this sweater warm?" and the agent responds "yes, it's ideal for winter," there's no field to compare that against. It's a qualitative claim. For that stretch, the only defense is design: if the agent shouldn't opine about product properties, you say so in the System Message and accept it's a weak layer; or you give it a tool that fetches the catalog's official description and validate the response doesn't stray from it.

It can't verify the wording's nuance. The validator confirms status is in_transit. It doesn't confirm the message says "it's on the way" instead of "it's almost there," which is the same fact with a different implicit promise. The example's text patterns — check 7 — scratch at this, but don't solve it; understanding nuance in free text requires a model, and we're back to the problem of verifying probability with probability. What you can do, and it's effective, is closing vocabularies: if message_to_customer gets built from a template with gaps instead of drafted freely, the nuance stops being in the model's hands.

It doesn't cover document-based AI. When the source of truth isn't a database row but a document — a policy PDF, a manual — verifying the response is backed by the retrieved text is a different problem, with its own set of techniques (mandatory citations, verification against the retrieved fragment, faithfulness measurement). That belongs to the ecosystem's document-AI guide, not this one. Here the source of truth is always structured: a JSON a tool returned.

And an observation summing up the lesson's spirit: the best defense against hallucination isn't detecting it, it's not letting the model be the one writing the data. Every time a number, a date, or a status comes from a template fed by the tool instead of from the agent's drafting, you eliminated the possibility of that data point being false. The model is still excellent for tone, empathy, and the response's structure. The data, when you can, put it there yourself.

Common mistakes

Verifying the agent's output with another agent (conceptual). What happens: someone connects a second AI Agent whose system prompt says "check if this response is backed by this data and answer valid or invalid." It works well in testing and fails in production, because both models share the same blind spots: where the first one filled in a plausible fact, the second one finds it plausible too. Why it happens: it's the solution anyone thinks of first, it gets built in five minutes, and it gets the obvious cases right — which builds confidence. How to spot it: feed the verifier a response with a made-up but coherent date, not an absurd one; if it approves it, you have two models agreeing on a lie. How to fix it: field comparison goes in a Code node or in IF nodes, where "2026-07-23" !== null allows no interpretation; save the model for what only a model can do, and keep the facts in deterministic territory.

Making fields required in the schema that the tool doesn't always return (practical). What happens: someone sets eta_date as required and of type string, and discovers the agent always sends a date, even when the tool returned none. It gets concluded the model hallucinates a lot, when actually the schema left it no other choice: a required field of type string can't stay empty. Why it happens: marking everything as required feels like rigor, and the side effect — forcing invention — isn't obvious until you look at the schema with that question in mind. How to spot it: for every required field in your schema, ask whether there's a real case where the tool doesn't have that data; if there is, the field can't be required. How to fix it: every field whose data can be missing gets declared as ["string", "null"], with a description explicitly saying when it must go to null and forbidding estimating it.

Asking for "don't make up data" in the System Message and considering it solved (conceptual). What happens: "respond only with information obtained from the tools, never make up data" gets added to the prompt, ten cases get tested, none hallucinates, and no validation gets set up. Weeks later an angry customer shows up over a date nobody promised. Why it happens: that instruction genuinely works and lowers the frequency a lot, and an improvement visible in testing feels like a solution. How to spot it: test twenty cases where the tool returns partial information — no date, no amount, a null field — which are the ones that trigger invention, instead of cases with complete data where the model doesn't need to fill in anything. How to fix it: the instruction stays, because it helps; but the structured output with allowed-null fields and the deterministic comparison is what turns "almost never hallucinates" into "if it hallucinates, it never reaches the customer."

Exercises

Exercise 1 — Design the schema. billing_specialist answers charge questions with lookup_charge, which returns charge_id, customer_id, order_id, amount, currency, charged_at, and status. Design the JSON Schema for its structured output separating facts from wording, and mark which fields can be null and why.

See solution
{
  "type": "object",
  "properties": {
    "charge_id":   { "type": "string" },
    "amount":      { "type": ["number", "null"],
                     "description": "The charge amount exactly as
                       returned by lookup_charge. Never rounded,
                       never estimated. Null if the tool returned
                       no matching charge." },
    "currency":    { "type": ["string", "null"], "enum": ["COP", "USD", null] },
    "charged_at":  { "type": ["string", "null"] },
    "status":      { "type": "string",
                     "enum": ["settled", "pending", "disputed",
                              "refunded", "not_found"] },
    "related_order_id": { "type": ["string", "null"],
                     "description": "Null if the charge is not
                       linked to any order. Do NOT guess an order
                       from the conversation." },
    "dispute_id":  { "type": ["string", "null"],
                     "description": "Only if open_dispute was called
                       in this turn and returned an id. Never invent
                       a reference number." },
    "message_to_customer": { "type": "string" },
    "facts_source": { "type": "array", "items": { "type": "string" } }
  },
  "required": ["charge_id", "status", "message_to_customer",
               "facts_source"]
}

What can be null and why:

  • amount, currency, charged_at — null when the query found no matching charge. If they were required, the agent would have to make up an amount for a charge that doesn't exist, which is the worst possible case.
  • related_order_id — some charges don't correspond to an order (an adjustment, a subscription). Requiring it would lead the agent to associate the charge with whatever order shows up in the conversation.
  • dispute_id — only exists if a dispute got opened that turn. It's the field most prone to invention, because reference numbers have a very predictable format and the model fills them in effortlessly. The description explicitly forbids it.

And one design decision: status includes "not_found" in the enum. Without that value, the agent that doesn't find the charge has to pick between four states that don't apply. Giving it a correct value for the "no data" case is the same idea as allowing null, applied to a closed vocabulary.

Why it works: every field that can be missing in the real world has a legitimate way to be missing in the schema. A schema without one is a schema that manufactures hallucinations.

Exercise 2 — Write the validator. For exercise 1's schema, write the Code node comparing the agent's output against lookup_charge's. Include at least one check detecting a type 3 hallucination.

See solution
// Node: Code — Name: validate_billing_output

const claimed = $input.first().json;
const actual  = $('lookup_charge').first().json;
const violations = [];

// The "charge not found" case gets validated differently: if the
// tool returned nothing, ALL fact fields must come back null.
const toolFoundCharge = actual && actual.charge_id;

if (!toolFoundCharge) {
  if (claimed.status !== "not_found") {
    violations.push(
      `the tool found no charge but the agent reports ` +
      `status="${claimed.status}"`
    );
  }
  for (const field of ["amount", "currency", "charged_at",
                       "related_order_id"]) {
    if (claimed[field] !== null && claimed[field] !== undefined) {
      violations.push(
        `${field} has value "${claimed[field]}" for a charge ` +
        `that doesn't exist`
      );
    }
  }
} else {
  // Exact field-by-field comparison.
  if (String(claimed.charge_id) !== String(actual.charge_id)) {
    violations.push(`charge_id doesn't match`);
  }
  // The amount is compared as a number, not as text: "1200" and
  // "1200.00" are the same money, "1200" and "1250" aren't.
  if (claimed.amount !== null &&
      Number(claimed.amount) !== Number(actual.amount)) {
    violations.push(
      `amount doesn't match: agent=${claimed.amount} ` +
      `tool=${actual.amount}`
    );
  }
  if (claimed.currency !== null && claimed.currency !== actual.currency) {
    violations.push(`currency doesn't match`);
  }
  if (claimed.charged_at !== null &&
      claimed.charged_at !== actual.charged_at) {
    violations.push(`charged_at doesn't match`);
  }
  if (claimed.status !== actual.status) {
    violations.push(
      `status doesn't match: agent="${claimed.status}" ` +
      `tool="${actual.status}"`
    );
  }
  if (claimed.related_order_id !== null &&
      claimed.related_order_id !== actual.order_id) {
    violations.push(`related_order_id doesn't match this charge`);
  }
}

// TYPE 3 — claiming facts without having called any tool.
const sources = Array.isArray(claimed.facts_source)
  ? claimed.facts_source : [];
if (!sources.includes("lookup_charge") &&
    (claimed.amount !== null || claimed.charged_at !== null)) {
  violations.push(
    "the agent reports charge data without declaring lookup_charge " +
    "as a source"
  );
}

// TYPE 3 about references: a dispute_id can only exist if
// open_dispute ran this turn.
if (claimed.dispute_id && !sources.includes("open_dispute")) {
  violations.push(
    `dispute_id "${claimed.dispute_id}" made up: open_dispute ` +
    `wasn't called this turn`
  );
}

return [{
  json: { ...claimed,
          validation_passed: violations.length === 0,
          violations }
}];

The dispute_id check is the most valuable one. A made-up reference number is the hallucination with the worst cost-benefit ratio of all: the customer gets an identifier in the right format, feels reassured, and finds out it doesn't exist when they ask about it again — usually days later, once frustration's already built up.

Why it works: the validator treats the "no data" case as a first-class scenario, with its own rules, instead of as an exception. Most hallucinations happen precisely there, in the gap, and a validator that only compares values when they exist doesn't cover the gap.

Exercise 3 — Close the door before it opens. The worked example's validator detects the made-up date after the agent wrote it. Propose a redesign where that specific hallucination is impossible from the start, and say what you lose with your proposal.

See solution

The response to the customer stops being drafted by the model as far as data goes, and gets built with a template fed by the tool:

# The agent no longer produces message_to_customer.
# It only produces the classification and the tone:
#   { order_id, status, customer_sentiment, needs_escalation }
#
# Node: Switch by status
#   └─► Set — Name: compose_response
#
#   status = in_transit:
#     "Your order {{ $('lookup_order').item.json.order_id }} shipped
#      on {{ $('lookup_order').item.json.shipped_at }} and is on the
#      way. You can track it with code
#      {{ $('lookup_order').item.json.carrier_tracking_code }}.
#      We don't have an exact delivery date, but tracking updates
#      daily."
#
#   status = delivered:
#     "Your order {{ ... }} shows as delivered on {{ ... }}.
#      If you didn't receive it, write to us and we'll look into it."
#
#   status = pending:
#     "Your order {{ ... }} is confirmed and hasn't shipped from
#      our distribution center yet. We'll let you know as soon as it does."

The made-up date becomes impossible, not improbable. There's no point in the flow where a model writes a date: the three dates showing up in the templates come from the tool's fields, and where the tool has no data, the template explicitly says there isn't one.

What you lose:

  • Naturalness. Four customers with the same status get exactly the same text. It shows, and on a channel like WhatsApp it shows more.
  • Ability to handle the unexpected. If someone asks about their order and while they're at it mentions the previous package arrived broken, the template doesn't see it. You need the agent to still be there for the rest of the conversation; the template only covers the factual stretch.
  • Maintenance work. Every new status is a new template. With four states it's comfortable; with twenty, it stops being.

When it's still worth it: for high-volume, high-risk responses. "Where's my order?" is probably 40% of TuTienda's traffic and it's where a made-up date generates complaints. It's worth having that path be template-based and letting the model handle the remaining 60%, where the risk per response is lower.

Why it works: it's the lesson's closing principle taken to the extreme — the best way for a data point not to be false is for the model not to write it. And the exercise shows that principle has a real cost, so it gets applied where the risk justifies it, not across the whole system.

Summary and next step

A hallucination is a plausible output with no backing in the data, and in an agent with tools it takes three forms: the data point made up while drafting, the parameter made up while calling a tool, and the claim made with nothing checked. The central defense is a design one: separating the fact from the wording with a structured output where every fact has its own field, where fields that can be missing explicitly accept null — because a required field manufactures inventions — and where the agent declares its sources. On top of that structure, the comparison is deterministic: a Code node that confronts, field by field, what the agent claims against what the tool returned, never another model. When it fails: retry with feedback exactly once, a template-built degraded response, or escalate. And as the last gate, the Guardrails node over the output text, looking not for attacks anymore but for commitments and leaks.

Before moving on to lesson 7 you should be able to: tell apart the three types of hallucination and say what each one is defended against with; write a JSON Schema letting the model say "I don't have that data" without breaking the schema; explain why verification can't be done by another agent; and point out, in your own system, which is the highest-volume response worth building with a template instead of drafting freely.

With this you have the module's five layers set up. And what's left is the question running through all of them: when something slips through anyway — and something's going to slip through — how do you find out, and how do you reconstruct what happened? An improper refund shows up on the statement three days later. A customer complains about something the agent told them last week. A ticket got misclassified and nobody knows why. Lesson 7 is the forensic capability: reading an agent's trace to know what entered its context, what it decided, and with what parameters — and instrumenting your own log, because n8n's executions don't live forever.

Resources