The model's output is untrusted

Overview

There's a word that sums up the most expensive mistake when integrating an LLM: trust. When you call a normal function and use what it returns without checking it, that trust is justified —the function is deterministic and tested—. When you do the same with an LLM, the trust is a gift you give to a component that hasn't earned it. A model's output always sounds right: correct grammar, confident tone, reasonable format. And that fluency is exactly the trap, because a perfectly worded output can be empty of meaning, malformed, out of policy, or an object that isn't even the type you expected. This lesson installs the root principle of the whole module: the LLM's output is untrusted until you validate it, and it's treated as user input, not as the trusted return of a function.

In lesson 1 you saw a battery of guardrails rejecting bad outputs at the boundary. Here we go down to the idea that justifies it: why that boundary is needed. Not because the model is "bad," but because its output is a hypothesis, not a fact. You're going to see, executed, the contrast between two designs of Mercado's "describe your product" generator: the antipattern that trusts the output and publishes it as is —and ends up publishing a dict that isn't even text, and a corrupt response with control characters— against the pattern that treats the output as untrusted and validates it by properties at the boundary before using it.

Connection with the module. Lesson 1 showed the boundary working; this lesson installs the principle that makes it necessary —the output is untrusted— and the base technique to validate it: verify properties, don't trust. It's the same turn you saw in module 1 with the probabilistic contract (you don't assert the exact output), taken to the trust boundary: here you don't assert the output, you guard it. Lessons 3 to 6 are concrete cases of this principle —schema, input, injection, moderation— and lesson 7 composes them. The boundary with the security guide holds: here we validate the AI component's output, we don't design the system's threat model.

An analogy: the brilliant intern's draft

Imagine you hire an extraordinary intern. They write lightning-fast, with impeccable prose, and most of their drafts are excellent. One day you ask them to draft the reply to an important customer and, with the same confidence and the same impeccable prose as always, they hand you a text that mentions an order number that doesn't exist, promises a discount the company doesn't offer, and in one paragraph a meaningless sentence slips in because they got distracted. The text looks perfect. If you skim it —trusting that "the intern always writes well"—, you send it and create a real problem.

Now think about how a sensible boss works with that intern. They don't check whether they like the style —the style is always good, that's not the point—. They check facts and rules: does the order number exist?, is the discount within policy?, is the text complete and coherent?, does it promise nothing forbidden? They verify properties they can check objectively, not the general impression. And they don't do it because they distrust the intern's talent; they do it because talent isn't the same as correctness, and what goes out with the company's name has to be correct, not just well-written.

Here's the point: an LLM is that brilliant intern, and its output is a draft, not a fact. The text's fluency —which is real, the model writes very well— says nothing about whether the content is valid. Trusting the output "because it sounds good" is skimming the draft and sending it. Validating it at the boundary is the sensible boss checking facts and rules before it goes out. In Mercado, every description the generator produces is the intern's draft; the boundary that validates it is the boss checking there's no invented claim, corrupt text, or something that isn't even a description, before publishing it.

Worked example: trust vs validate the output

We're going to model the two designs. The ai_component is a deterministic stub that simulates the generator: it deliberately returns a realistic mix —sometimes a useful description, sometimes garbage of different types: an empty string, a text with control characters (corruption), an object that isn't text (a dict), a forbidden claim—. A probabilistic model produces everything, and the point is to see what each design does with that mix. The is_publishable function is the guardrail: it doesn't assert what the output says, it verifies it meets properties.

# Lesson 2: the model's output is UNTRUSTED until you validate it.
# It's treated as user input: validate at the boundary before using it.
# LLM simulated by a deterministic stub; no network or APIs.
import random

_RNG = random.Random(1)


# The stub simulates "describe your product": sometimes it gives useful text,
# sometimes realistic garbage (empty, control noise, wrong type, forbidden
# claim). A probabilistic model produces everything.
def ai_component(_attributes):
    outputs = [
        "29-inch mountain bike, 21 speeds, disc brakes.",                      # ok
        "",                                                                    # empty
        "\x00\x00 corrupt response \x07",                                      # control noise
        {"unexpected": "json"},                                                # wrong type
        "The best product in the world, guaranteed, cures everything.",        # forbidden claim
        "600W blender with a 1.5-liter glass jar.",                            # ok
    ]
    return _RNG.choice(outputs)


MAX_LEN = 200
BANNED = ("cures", "best in the world", "guaranteed", "100%")


def is_publishable(text):
    # PROPERTY contract over the output (not exact equality).
    if not isinstance(text, str):
        return (False, "not text")
    if text.strip() == "":
        return (False, "empty")
    if any(ord(c) < 32 and c not in "\n\t" for c in text):
        return (False, "contains control characters")
    if len(text) > MAX_LEN:
        return (False, "exceeds the length limit")
    low = text.lower()
    for claim in BANNED:
        if claim in low:
            return (False, f"forbidden claim '{claim}'")
    return (True, "ok")


N = 6
print("=== Antipattern: trust the output and publish it directly ===")
_RNG.seed(1)
for _ in range(N):
    out = ai_component("attrs")
    print(f"  PUBLISHED as is -> {out!r}")

print()
print("=== Pattern: the output is untrusted; validate at the boundary ===")
_RNG.seed(1)
published = rejected = 0
for _ in range(N):
    out = ai_component("attrs")
    ok, reason = is_publishable(out)
    if ok:
        published += 1
        print(f"  PUBLISHED   {out!r}")
    else:
        rejected += 1
        print(f"  REJECTED    ({reason})")

print()
print(f"Summary: {published} published, {rejected} rejected at the boundary.")

What to expect. When you run the file, the output is exactly this:

=== Antipattern: trust the output and publish it directly ===
  PUBLISHED as is -> ''
  PUBLISHED as is -> 'The best product in the world, guaranteed, cures everything.'
  PUBLISHED as is -> '29-inch mountain bike, 21 speeds, disc brakes.'
  PUBLISHED as is -> '\x00\x00 corrupt response \x07'
  PUBLISHED as is -> '29-inch mountain bike, 21 speeds, disc brakes.'
  PUBLISHED as is -> {'unexpected': 'json'}

=== Pattern: the output is untrusted; validate at the boundary ===
  REJECTED    (empty)
  REJECTED    (forbidden claim 'cures')
  PUBLISHED   '29-inch mountain bike, 21 speeds, disc brakes.'
  REJECTED    (contains control characters)
  PUBLISHED   '29-inch mountain bike, 21 speeds, disc brakes.'
  REJECTED    (not text)

Summary: 2 published, 4 rejected at the boundary.

Read the two sections in contrast, because that's where the whole lesson is.

In the antipattern, the system trusts and publishes whatever. Look at what reached the store: an empty description (a product with no text), a forbidden claim ("the best in the world, guaranteed, cures everything"), a corrupt text with control characters ('\x00\x00 corrupt response \x07', which can break the render or even be an attack vector), and —the most telling— a {'unexpected': 'json'}, an object that isn't even text. That last case is the one to burn into memory: the system expected a product description and published a data structure. It's not that the model "wrote badly"; it's that the output wasn't the type the rest of the system assumed, and nobody checked. All this reached production because the design trusted.

In the pattern, the same stub with the same seed produces exactly the same, but now each output hits is_publishable before being used. The empty one is rejected for being empty; the claim is rejected for being forbidden; the corrupt text is rejected for control characters; the dict is rejected for "not text". Only the two legitimate outputs —the two bike descriptions, well-formed and with no claims— pass. Two published, four rejected. The system never exposed the garbage, not because the model improved —it's identical—, but because the output was treated as untrusted and validated at the boundary.

Notice what the validation does: it doesn't assert what the output says, it verifies it meets properties —it's text, non-empty, no control characters, within a limit, no forbidden claims—. That's exactly what the sensible boss does with the intern's draft: they don't check the style, they check facts and rules. And that's why it works even when the model produces something you never anticipated: the "must be text" property caught a dict that no specific content test would have foreseen.

Going deeper: what "treat the output as user input" means

There's a phrase worth internalizing because it reorders the whole design: the LLM's output is treated as user input. Think about it. Do you trust what an anonymous user types into a web form and insert it straight into your database, execute it, show it to other users without sanitizing? No. You validate it, escape it, bound it —decades of web security were built on "never trust the user's input"—. This lesson's thesis is that an LLM's output deserves exactly the same treatment, and for a deep reason: that output may have been influenced by untrusted user input (an injection, which we'll see in lesson 5), so it effectively is a form of user input that took a trip through the model.

It's worth making the anatomy explicit, because it's the shape you'll apply to every AI component:

        seller attributes (input, untrusted)
                        │
                        ▼
              ┌───────────────────┐
              │   ai_component    │   the LLM (stub): PROPOSES an output.
              │   (the LLM)       │   The output is a HYPOTHESIS, not a fact.
              └───────────────────┘
                        │  proposed output (UNTRUSTED)
                        ▼
              ┌───────────────────┐
              │  is_publishable   │   output GUARDRAIL (deterministic):
              │  (guardrail)      │   verifies PROPERTIES, doesn't trust.
              └───────────────────┘
                   │            │
                 PASS        REJECT
                   │            │
                   ▼            ▼
          public store   (nothing published; fallback/retry)

Verify properties, not equality. You can't write assert output == "the correct description" —the model gives different outputs for the same input, as you saw in module 1—. The output's contract is a set of properties and invariants: it's the expected type, non-empty, within a range, with the right format, no forbidden content. is_publishable is that contract made code. And because it's deterministic, it's testable with an exact assert: assert is_publishable("")[0] is False always passes. The uncertainty lives in the model; the boundary that contains it is certain.

The type is the first property, and the most forgotten. The {'unexpected': 'json'} case isn't exotic: when you ask the model for structured output (JSON) and the model returns text that doesn't parse, or parses to something that isn't what you expected, you have an object of the wrong type running through your system. The first line of is_publishable —if not isinstance(text, str)— catches a whole family of bugs that would otherwise explode much deeper, when something tries to do len() or .lower() on a dict. Validating the type at the boundary turns a mysterious crash into a clean, explained rejection. Lesson 3 takes this in depth with schema validation.

"It almost always gets it right" is exactly the trap. The most common argument against validating the output is "the model gets it right 95% of the time, why so much guardrail." Flip it around: if the model gets it right 95% of the time, it fails 1 in 20 outputs, and at a marketplace's scale that's thousands of bad descriptions published a month. And the problem isn't just the volume: it's that you don't know which of the 20 is the bad one until you validate it, because they all sound good. Validation doesn't exist for the normal case —for that it wouldn't be needed—; it exists precisely to catch that 5% that, without a boundary, reaches production with the same confidence as the good 95%.

Rejecting isn't the end: it's the start of the fallback. When the guardrail rejects an output, the system isn't left with no response; it degrades. It can retry the model (sometimes the second output does pass), fall back to a deterministic template, or ask for human intervention. That degraded route is module 5's theme (resilience). For now, the important thing is that rejecting a bad output is always better than publishing it: the cost of a retry or a template is minimal compared to that of an illegal claim in the store.

Common mistakes

Trusting the output because the model "is very good." What happens: the team measures that the model gets it right 95% in its tests and decides validating is paranoia. In production, the 5% that fails —at scale— is a constant stream of bad outputs published, and since they all sound good, nobody detects them until a customer complains or a regulator asks. Why it happens: "good hit rate" is confused with "trustworthy." A component with a 5% silent failure isn't trustworthy for a path that touches the public. How to spot it: your design uses the model's output with no deterministic validation in between, and your justification is the hit rate. How to fix it: always validate, because validation is cheap and silent failure is expensive. The hit rate decides how much you retry or degrade, not whether you validate.

Validating the content but not the type/form. What happens: the guardrail checks forbidden claims and length, but assumes the output is a string and calls .lower() on it; the day the model returns an object or None, the guardrail itself blows up with an AttributeError. Why it happens: you validate "what can be wrong in the text" and forget that "it might not even be text." How to spot it: your validation assumes the output's type without checking it in the first line. How to fix it: validate the type/form first —isinstance, parsing, required fields— and only then the content. As in the example: if not isinstance(text, str) goes before any claims check. Lesson 3 formalizes this with schema.

Putting the validation after using the output. What happens: the system publishes the description and then runs a job that reviews content and unpublishes the bad. Between the publication and the unpublication, the bad output was visible —maybe hours—, and the damage (an illegal claim seen by customers, an offensive text) already happened. Why it happens: validating "after" feels simpler and doesn't block the flow. How to spot it: in your design, the output crosses to production before passing the guardrail. How to fix it: the validation goes at the boundary, before using the output —the guardrail is a gate the output crosses to reach production, not an auditor that reviews it once it already arrived—. Rejecting before publishing is prevention; unpublishing after is cleaning up an incident that already happened.

Exercises

Exercise 1 — The order of the properties. In is_publishable, the first check is isinstance(text, str) and the last is the forbidden-claims one. Explain why that order matters, and what would happen if the claims check (text.lower()) were first and the model's output were the dict {'unexpected': 'json'}.

Exercise 2 — The hit rate doesn't decide whether you validate. A colleague argues: "we measured that the generator gets 98% of the descriptions right; validating 100% of the outputs is a waste of compute." Refute the argument with a concrete calculation at Mercado's scale (assume 50,000 descriptions generated a month) and explain which design decision does depend on the hit rate.

Exercise 3 — Properties for the support agent. The example's guardrail validates product descriptions. Now the support agent proposes an action in the form of an object: {"action": "reply", "text": "..."} or {"action": "refund", "order_id": "...", "amount": ...}. Write (in pseudocode or Python) the properties a guardrail should verify over that output before using it, and explain why verifying the type/form is here even more critical than in the descriptions.

def is_valid_action(proposal):
    # 1) Type/form: it must be a dict with a known action.
    if not isinstance(proposal, dict):
        return (False, "not an object")
    action = proposal.get("action")
    if action not in ("reply", "refund"):
        return (False, "unknown action")
    # 2) Fields per action type.
    if action == "reply":
        text = proposal.get("text")
        if not isinstance(text, str) or text.strip() == "":
            return (False, "reply: empty text or not a string")
        # (also: moderation, no leaking internal data — lessons 5 and 6)
    if action == "refund":
        oid = proposal.get("order_id")
        amount = proposal.get("amount")
        if not isinstance(oid, str) or not isinstance(amount, (int, float)):
            return (False, "refund: fields with invalid type")
        if amount <= 0:
            return (False, "refund: non-positive amount")
        # (also: validate against the business policy — module 6)
    return (True, "ok")

Summary and next step

In this lesson you installed the module's root principle: the LLM's output is untrusted until you validate it, and it's treated as user input, not as the trusted return of a function. You saw it with the brilliant intern whose draft looks perfect and can be wrong, and you measured it: the antipattern that trusts published to the store an empty output, a forbidden claim, a corrupt text with control characters, and —the most telling— a dict that wasn't even text; the pattern that validates by properties rejected four of six and let only the two legitimate ones through, with the same identical model. Safety didn't come from the model improving; it came from treating its output as a hypothesis that must be verified. And you saw why "it almost always gets it right" is exactly the trap: the 5% that fails sounds just as good as the good 95%, and you don't know which it is until you validate it.

Before moving on you should be able to: explain why an output's fluency says nothing about its correctness; verify properties of an output instead of trusting it; argue why the type/form is validated first; and refute "it almost always gets it right" with a scale calculation.

Lesson 3 takes the most structural property —the type and the form— and formalizes it: schema validation at the boundary. When you ask the model for a structured output (a JSON with title, description, category, tags), the schema is the contract that verifies types, required fields, ranges, and a closed set of values. You're going to see, executed, six simulated generator outputs against that schema: two comply and pass, and the rest are rejected —not JSON, invalid category, empty title, too many tags—. The way to turn "not text" into a precise, testable contract.

Resources

• OWASP Top 10 for LLM Applications — owasp.org/www-project-top-10-for-large-language-model-applications. The LLM05 Improper Output Handling risk is exactly this lesson's theme: treating the model's output as trusted and using it without validation is a cataloged vulnerability. Read it for its framing of the output as a risk surface. In English.
• Martin Fowler and Bharani Subramaniam, "Emerging Patterns in Building GenAI Apps" — martinfowler.com/articles/gen-ai-patterns. The guardrails section treats output validation as a first-class pattern around the AI component. In English.
• Anthropic, Claude documentation — docs.anthropic.com. The structured-output and tool-use guides show, at a conceptual level, how to ask for outputs with a verifiable form —the starting point for validating them—, without fixating on a model version. In English.
• Chip Huyen, AI Engineering (O'Reilly, 2024). The chapters on the reliability of foundation-model applications treat the model's output as untrusted by default and validation as part of the design. In English.