Moderation as a gate

Overview

Lesson 3 showed that the schema validates the form of an output but not its truth or decency: a description can meet the schema to the letter and contain an illegal medical claim or an insult. That gap —what the schema lets through— is closed by this lesson with a different gate: content moderation. Moderation reviews the content of a text against a set of forbidden categories —offensive language, false claims, disallowed product categories, self-harm content— and blocks what falls into any of them. And, like every good boundary gate, it's applied on both edges: it moderates what COMES IN (the messages customers write to the support agent) and what GOES OUT (the descriptions the generator sends to the public store).

Moderation is a policy gate: it encodes what content Mercado doesn't allow and acts as a binary filter at the edge. It's not an aesthetic judgment ("is this description good?"); it's a policy check ("does this description violate a content rule?"). You're going to see, executed, the same moderation function applied to two batches: on the input, blocking an insult and an attempt to list a forbidden weapon before they reach the agent; on the output, blocking a description with a false claim before it reaches the store. The same gate, the two edges.

Connection with the module. Moderation is one of the three gates of lesson 1's battery (moderation), and the one that covers the gap the schema (lesson 3) left open: form vs content. Together with input validation (lesson 4) and the trust boundary (lesson 5), it completes the catalog of gates that lesson 7 composes into a stack. The boundary with the security guide holds: here moderation is an architectural content gate for the AI component; the content policy in depth, legal compliance, and human moderation at scale are from other disciplines, which we link to.

An analogy: the quality filter that reviews what comes in and what goes out

Go back to the factory of lesson 1, but look closely at its quality-control department, because it has a rule that applies to both edges. At the raw-material receiving dock, an inspector reviews what arrives: if a supplier sends an ingredient that doesn't meet the standards —contaminated, expired, of a category the factory doesn't process—, it's rejected at the dock, before entering the line. And at the finished-product output, another inspector reviews what goes to the stores: if a batch comes out with a defect, with a label promising something the product doesn't do, it's set aside before shipping it. It's the same quality discipline, applied at two points: you don't trust that the raw material is good just because it arrived, nor that the product is sellable just because it was produced.

Notice something about how the inspector works: they don't evaluate tastes ("do I like this flavor?"), they verify standards ("does it meet the food-safety norm?, does the label make a forbidden claim?"). It's a policy filter, with written criteria, not opinion. And when something falls in a gray zone —a case the rules don't clearly cover— they don't let it through by default nor reject it blindly: they escalate it to a human supervisor. The rules cover most cases quickly and cheaply; human judgment covers the edges.

Here's the point: moderation is that quality department, applied on the two edges of the AI component. On the input, it reviews the customer's message —does it carry abusive language?, does it ask for something in a forbidden category?— before the model processes it. On the output, it reviews the generated description —does it have a false claim?, offensive content?— before it reaches the store. It's a policy gate, with written rules, not a judge of tastes. And for the edge cases the rules don't cover, it escalates to a human instead of guessing. In Mercado, input moderation protects the agent (and the system) from abusive content; output moderation protects the customers and Mercado's brand from what the model might publish.

Worked example: the same gate on both edges

We're going to execute a moderation gate with per-category rules —offensive language, false claims, forbidden categories, self-harm— and apply it to two batches: one of input (customer messages) and one of output (descriptions to the store). The LLM is simulated: we work with already-produced texts, because the focus is the gate.

# Lesson 6: MODERATION as a gate. A content gate reviews what
# COMES IN (customer messages) and what GOES OUT (generated descriptions)
# before they cross to a human or to the public store.
# LLM simulated by a stub; no network or APIs.

# Categories Mercado doesn't allow (simplified and deterministic).
MODERATION_RULES = {
    "hate":            ("idiot", "fool", "stupid", "useless"),
    "false_claim":     ("cures", "best in the world", "guaranteed", "100% effective"),
    "banned_category": ("firearm", "ammunition", "illegal substance"),
    "self_harm":       ("harm myself", "hurt myself"),
}


def moderate(text):
    # Returns (allowed, category|None, hit|None). Deterministic.
    low = text.lower()
    for category, phrases in MODERATION_RULES.items():
        for phrase in phrases:
            if phrase in low:
                return (False, category, phrase)
    return (True, None, None)


# INPUT batch: customer messages to the support agent.
INBOUND = [
    ("in_ok",     "My order didn't arrive, can you help me please?"),
    ("in_hate",   "Your service is terrible, you're all useless."),
    ("in_weapon", "I want to sell a firearm, how do I list it?"),
]

# OUTPUT batch: generated descriptions that would go to the public store.
OUTBOUND = [
    ("out_ok",    "Sports watch with GPS and heart rate monitor."),
    ("out_claim", "Cream that cures acne and is the best in the world, guaranteed."),
    ("out_ok2",   "Nonstick frying pan set, 3 pieces, induction-compatible."),
]


def run_gate(label, batch):
    print(f"=== Moderation of {label} ===")
    allowed = 0
    for case_id, text in batch:
        ok, category, hit = moderate(text)
        if ok:
            allowed += 1
            print(f"  {case_id:<11} PASS")
        else:
            print(f"  {case_id:<11} BLOCK    [{category}] for '{hit}'")
    print(f"  {allowed}/{len(batch)} passed the gate.")
    print()


run_gate("INPUT (customer messages)", INBOUND)
run_gate("OUTPUT (descriptions to the store)", OUTBOUND)

What to expect. When you run the file, the output is exactly this:

=== Moderation of INPUT (customer messages) ===
  in_ok       PASS
  in_hate     BLOCK    [hate] for 'useless'
  in_weapon   BLOCK    [banned_category] for 'firearm'
  1/3 passed the gate.

=== Moderation of OUTPUT (descriptions to the store) ===
  out_ok      PASS
  out_claim   BLOCK    [false_claim] for 'cures'
  out_ok2     PASS
  2/3 passed the gate.

Read the two blocks, because they show the same gate protecting two different edges.

On the input, the gate reviews the customer messages before the agent processes them. The normal message ("My order didn't arrive...") passes. The insult ("you're all useless") is blocked by the hate category. And the attempt to list a weapon ("I want to sell a firearm") is blocked by banned_category. One of three passes. Input moderation does two things: it protects the system from processing abusive content and saves the model call (you don't spend on moderating-with-the-model something a rule already rejected), and it blocks early attempts to use Mercado for something forbidden.

On the output, the same gate reviews the generated descriptions before they reach the store. The two normal descriptions (the watch, the pans) pass. The description with false claims ("cures acne... best in the world, guaranteed") is blocked by false_claim —the same text that, recall from lesson 3, would pass a schema validator because it's a string of the right length in the right category—. Two of three pass. Here's the reason for output moderation: catching exactly the inappropriate content the schema lets through, before it's published with Mercado's brand.

Notice how telling the contrast with lesson 3 is. The schema and moderation are complementary gates: the schema asks "does it have the right form?" and moderation asks "does the content violate a policy?". The out_claim description demonstrates why you need both: it has the right form (it would pass the schema) and the forbidden content (moderation blocks it). Neither alone is enough. And the same moderate function serves on both edges —there isn't a separate "input moderation" and "output moderation"; there's one content policy applied wherever the content crosses the boundary—.

Going deeper: what moderation is and isn't

It's worth pinning down moderation's role at the boundary, its limits, and how it connects with human judgment.

Moderation is a policy gate, not a quality judge. It's easy to confuse "moderate" with "evaluate." Module 3's eval asks "how good is this output?" (quality, a spectrum). Moderation asks "does this output violate a content rule?" (policy, binary). A description can be mediocre (bad quality) without violating any policy (passes moderation); and an excellent description can have an illegal claim (blocked by moderation). They're two different questions with two different gates: the eval measures quality, moderation applies policy.

The rules cover the common; the human covers the edges. The example uses simple phrase-matching rules —deliberately, so you see the mechanism—. In production, moderation combines rules, lists, classifiers, and, for the edge cases, human judgment. What matters architecturally isn't the exact technique, but the arrangement: an automatic, cheap gate resolves most cases quickly, and what falls in the gray zone —a text that may or may not violate a policy— is escalated to a human instead of decided blindly. Just as the factory inspector escalates to the supervisor. Never let the gray zone pass by default (permissive with the doubtful) nor block everything doubtful (you reject legitimate content); route it to review.

The honest limit of phrase matching. Moderation based on exact phrases has the same limits as lesson 5's injection detector: an attacker or a model can express the forbidden in a way the list doesn't cover ("c-u-r-e-s", synonyms, another language), and a too-aggressive list blocks legitimate content (a peeler description that says "the best in the kitchen" isn't a dangerous false claim). Real moderation uses more robust techniques than literal matching. But the architectural pattern —a deterministic policy gate at the edge, blocking the forbidden and escalating the doubtful— is the same regardless of the sophistication of the detection technique. Here we teach the pattern; the detection technique in depth is AI Engineering's.

Block at the edge, don't unpublish after. As with the whole boundary, moderation goes before the content crosses: it blocks the description before publishing it, not unpublishes it hours after customers already saw it. On the input, it blocks the abusive message before processing it. The cost of moderating at the edge is a text that doesn't cross; the cost of moderating after is an incident that already happened.

Moderate the input AND the output, for different reasons. The two edges protect against different things and that's why both are needed. Input moderation protects the system and Mercado: it stops the abuse (insults to the agent), the attempts to use the platform for something forbidden (listing a weapon), and saves compute. Output moderation protects the customers and the brand: it keeps the model from publishing inappropriate content, a false claim, something offensive. Moderating only one edge leaves the other open: if you only moderate the output, the abuse on the input still enters; if you only moderate the input, the model can still generate a forbidden output from a perfectly clean input.

Common mistakes

Confusing moderation with quality evaluation. What happens: the team puts moderation inside the eval —"if the quality score is high, it gets published"— and assumes a high-quality output can't violate a policy. An excellent and persuasive description that makes an illegal claim gets published, because the eval scored it high for how well written it was. Why it happens: two different questions (is it good? vs does it violate a rule?) are collapsed into a single metric. How to spot it: you don't have a policy gate separate from the quality score. How to fix it: keep moderation as a binary gate independent of the eval. An output is published only if it passes moderation (doesn't violate policy) and meets the quality threshold (eval, module 3). They're two conditions, not one.

Moderating only one edge. What happens: the team puts careful moderation on the output (what's published) but leaves the input unfiltered, or vice versa. The unguarded edge is where the problem enters: without moderating the input, the system processes abuse and spends compute on it; without moderating the output, the model publishes whatever it generates. Why it happens: moderation is thought of as a single point instead of a policy applied wherever the content crosses. How to spot it: you can name the moderation of one edge but not the other. How to fix it: apply the same content policy on both edges, understanding that they protect against different things —input: abuse and forbidden use; output: inappropriate content published—.

Letting the doubtful pass (or get blocked) by default. What happens: moderation has a gray zone —texts that might violate a policy but it's not clear— and the system, for simplicity, lets them all through (permissive) or blocks them all (restrictive). Permissive publishes doubtful things that sometimes turn out to be violations; restrictive rejects legitimate content and frustrates users and sellers. Why it happens: moderation is treated as purely binary with no third path. How to spot it: there's no "escalate to human review" route for the edge cases. How to fix it: add the third decision —pass / block / escalate to human— for the gray zone. The rules resolve the clear; human judgment resolves the doubtful. This connects with module 6's human-in-the-loop and module 7's data loop (human decisions on edge cases improve the rules).

Exercises

Exercise 1 — Schema or moderation. For each generator output, say which gate would catch it —schema (form), moderation (content), or both— and why: (a) {"title": "", "description": "x", "category": "home"}; (b) {"title": "Total cure", "description": "Cures cancer, guaranteed", "category": "home"}; (c) the free text "here's the description" (not JSON); (d) {"title": "Kitchen knife", "description": "20cm steel blade", "category": "banned_weapon"}.

See solution

• (a) empty title → SCHEMA. The title violates the range 1..60. It's a form problem; moderation wouldn't touch it (there's no forbidden content). The schema catches it.
• (b) medical claim → MODERATION (passes schema). The form is right: title and description are strings of valid length, category is "home" (valid). The schema would accept it. The problem is the content ("cures cancer, guaranteed"), which moderation blocks by false_claim. It's the case that demonstrates the schema is necessary but not sufficient.
• (c) free text → SCHEMA. It's not JSON, so it fails at the first step of the schema validator, before moderation looks at it. The schema catches it.
• (d) weapon category → depends on the design, probably BOTH. If "banned_weapon" isn't in the closed set of valid categories, the schema rejects it for an invalid category. Also, content moderation might catch "weapon" in the text by banned_category. This case shows the gates sometimes overlap —and that overlap is defense in depth, not waste: if by mistake someone adds "banned_weapon" to the valid categories, moderation still catches it—.

The lesson: schema and moderation catch different classes of problem (form vs content) and sometimes overlap; you need both, and the overlap gives you redundancy.

Exercise 2 — The same function, two edges. The example uses the same moderate function for the input and the output. Discuss an advantage and a possible disadvantage of sharing exactly the same content policy on both edges, and give an example of a rule you might want on only one edge.

See solution

Advantage of sharing the policy: consistency and maintainability. There's a single definition of "what content Mercado doesn't allow," in one place, and it applies on all edges. When you change the policy (add a forbidden category), the change is automatically reflected on input and output. There's no risk of the input and output diverging and leaving a gap.

Possible disadvantage: the two edges have different contexts, and a single policy can be too strict or too loose for one of them. An input message from a frustrated customer that says "your service is a disaster" is legitimate content (a complaint) that shouldn't be blocked, even though it has a negative tone; but that same tone in a published output would be inappropriate. If you apply the same strict rule to the input, you block legitimate customer complaints; if you apply it loosely to the output, you let through content that shouldn't be published.

Example of a rule on only one edge: detecting false claims ("cures", "guaranteed") makes sense mostly on the output —what's published in the store must not make false promises—. On the input, a customer who writes "they promised me this would cure my pain" isn't making an advertising claim; they're describing their experience, and blocking it would be absurd. So the false_claim rule you'd probably want to apply strongly on the output and not (or differently) on the input.

The conclusion: sharing the base policy is valuable for consistency, but it's worth allowing each edge to adjust or disable specific rules according to its context. A common policy with per-edge exceptions, not two independent policies nor one identical blind one.

Exercise 3 — The gray zone. Design the moderation decision policy for an edge case: a generated description says "the best-selling in its category." It's not a forbidden medical claim, but it could be an unverified superiority claim. Explain why "pass by default" and "block by default" are both bad, and how you'd structure the third path.

See solution

"The best-selling in its category" is a gray zone: it's not a dangerous medical claim (it doesn't fall into the hard false_claim rules like "cures"), but it's a superiority statement that, if false, can be misleading advertising. The exact rule doesn't clearly cover the case.

Why "pass by default" is bad: if you let everything doubtful through, you publish unverified superiority claims. Some will be false ("the best-selling" when it isn't), and that's misleading advertising with Mercado's brand. Permissiveness turns the gray zone into a channel through which violations slip.

Why "block by default" is bad: if you reject everything doubtful, you also block legitimate descriptions —a product that really is the best-selling can't say so, and honest sellers get frustrated with rejections they don't understand—. Restrictiveness turns the gray zone into friction for legitimate users.

The third path: route the gray zone to a decision that isn't an automatic binary. Options, depending on the design:

1. Escalate to human review. The text isn't published automatically nor blocked; it enters a queue where a human decides. Cheap if the gray zone is a small percentage of the volume.
2. Ask the seller for verification. If the claim is verifiable ("the best-selling"), the system can require the seller to back it with data, or rewrite the description without the claim.
3. Allow with a label/softening. Publish but flagging the claim for later audit, or softening it ("popular in its category" instead of "the best-selling").

And —connecting with module 7— each human decision on a gray case is data that improves the rules: if the reviewers consistently approve "the best-selling" when there's backing and reject it when there isn't, that signal tunes the moderation over time. A well-managed gray zone isn't a problem to avoid; it's the source of policy improvement.

Summary and next step

In this lesson you added to the boundary the gate that covers the gap between form and content: moderation. It's a policy gate —it blocks what violates a content rule, not what's low quality— and it's applied on the two edges: it moderates what comes in (customer messages) and what goes out (descriptions to the store). You measured it: on the input, the gate blocked an insult and an attempt to list a weapon (1/3 passed); on the output, it blocked a description with a false claim —the same text that would pass the schema— before the store (2/3 passed). You saw that moderation and the schema are complementary (content vs form), that moderation isn't quality evaluation, that the same policy serves on both edges for different reasons, and that the gray zone is escalated to human judgment instead of decided blindly.

Before moving on you should be able to: distinguish moderation (policy, binary) from the eval (quality, spectrum); argue why the schema and moderation are both needed; apply the same content policy on input and output; and structure the third path (escalate to human) for the gray zone.

You now have all the gates: the untrusted output (L2), the schema (L3), the input (L4), the trust boundary (L5), and moderation (L6). Lesson 7 composes them: the guardrail stack. You're going to see, executed, a defense-in-depth pipeline where each layer catches what the others let through —injection, schema, and moderation stopping different items—, and why the order matters. And you're going to see, in a second experiment, the module's hard rule: validation lives in deterministic code, not inside the LLM —a model "self-check" will let itself be injected and approve the forbidden; the deterministic gate won't—.

Resources

• OWASP Top 10 for LLM Applications — owasp.org/www-project-top-10-for-large-language-model-applications. The catalog frames input and output moderation as controls against harmful content and unsafe generation; this lesson applies it on the two edges. In English.
• Anthropic, Claude documentation, responsible-use and content-safety guides — docs.anthropic.com. It treats at a conceptual level how to think about harmful content on input and output and the use of content gates, without fixating on a model version. In English.
• Martin Fowler and Bharani Subramaniam, "Emerging Patterns in Building GenAI Apps" — martinfowler.com/articles/gen-ai-patterns. Moderation appears as one of the guardrail layers around the AI component, complementary to form validation. In English.
• Chip Huyen, AI Engineering (O'Reilly, 2024). The chapters on security and guardrails treat content moderation as a design gate and discuss the balance between automatic rules and human judgment. In English.