7. Assertions with the Evaluation node

Description

By the end of this lesson you will be able to stop eyeballing order-triage's output and start verifying it automatically: writing assertions that check the agent classified each order correctly, running a set of cases with their expected result, and getting a "pass/fail" verdict against a passing threshold. You're going to meet n8n's Evaluation node, its operations for recording metrics about results, how a case dataset with its correct answer gets built, and which metrics work for verifying a classification —including an AI agent's output.

This matters because it's the distinction we set up in lesson 1 and have been postponing until now: the one separating "it ran" from "it got it right." A workflow can finish green and have classified an order wrong; "no technical error" isn't "correct result." Up to now, to know if the agent got it right, you had to read its output and judge it yourself. That doesn't scale —six cases you can eyeball, sixty you can't— and it isn't reproducible —your judgment today might differ from tomorrow's. An assertion turns that judgment into an automatic, objective, repeatable check.

Connection to the module: this is the checklist's sixth point, and the one giving meaning to all the previous ones. Running in sandbox (2), with synthetic data (3), dry (4), reproducibly (5), and at zero cost (6) is useless if in the end you never verify the output is correct: it would be testing with all the world's rigor and never reading the result. This lesson closes the loop. It builds on 3 (the synthetic cases, now with their expected answer), on 5 (fixed inputs so the evaluation is stable), and on 6 (running the agent for free, so evaluating many cases costs nothing). In lesson 8 you're going to bring everything together in one complete pass with its verdict.

The exam with an answer key

Think of how a spelling test gets graded at school. The teacher doesn't read each student's answers and decide "by feel" whether they're right; that would be slow, subjective, and different every time. Instead they use an answer key: a list with the correct word for each question. They compare the student's answer against the key, question by question —does it match? one point; doesn't it match? zero— add up, and apply a threshold: out of twenty words, you pass with eighteen. The process is fast, objective, and produces the same verdict no matter who applies it, because the answer key is the same for everyone.

Verifying your workflow's output with assertions is grading with an answer key. You have your test cases —lesson 3's synthetic orders— and for each one you write down the correct answer: "this 52,000-peso order must be classified as manual_review." That's your answer key. You run the cases, and instead of reading each output by eye, you automatically compare what the agent answered against what you wrote down it should answer. It matches: pass. It doesn't: fail. You add up, apply a threshold, and get an objective, repeatable verdict.

Let's define the term. An assertion is a checkable statement about the result, that can only be true or false. "The agent classified this order as manual_review" is an assertion: it can be checked by looking at the output, and it gives true or false, with no opinion involved. "The agent classified correctly" is not an assertion —"correctly" is a judgment, not a check— you have to make it concrete: "the agent classified it as manual_review, which is what we expected." An assertion is the answer key's correct answer, written so a machine can check it.

"It ran" versus "it got it right," one last time

It's worth seeing this with a concrete case, because it's the lesson's heart. Take the 52,000-peso order that should go to manual review. You run order-triage and there's no red error: the Webhook received it, the agent responded, the gate diverted, everything green. Is it correct?

You still don't know. "All green" tells you the workflow ran —the engine worked, no node blew up. It doesn't tell you it got it right —that the agent classified the order as it should. If the agent responded approved instead of manual_review, the workflow would have finished green just the same: technically there was no failure, only a wrong classification. A 52,000-peso order that a human should have reviewed would have gotten auto-approved, with not a single red line to warn you.

The assertion is what catches that. Comparing the agent's output (approved) against the expected answer (manual_review), the assertion gives false, and there's your warning: the workflow ran but didn't get it right. Without the assertion, "green" would have fooled you. That's why verifying the output isn't a luxury: it's the only way to tell apart a workflow that works from one that just doesn't fail.

n8n's Evaluation node

n8n ships a node dedicated to this: the Evaluation node. Its job is to help you systematically check that your workflow —and the AI part in particular— produces correct results across a set of cases. Let's look at its pieces. First, a warning this guide repeats on principle: this feature's interface evolves fast. The exact names of operations, fields, and metrics I describe below are what the docs report as of this guide's writing (2026); check your n8n version that they exist under that name and behave this way before trusting a fine detail. If something doesn't match, hold onto the concept —comparing the output against an expected answer, across a set of cases, with a threshold— which is stable even if the interface changes.

n8n's evaluation system has three pieces working together:

1. The case dataset. It's your answer key: a table where each row is a case, with the inputs (the order) and the expected answer (how it should get classified). n8n stores that dataset in a Data table or in a Google Sheet. Each row is a different exam with its correct answer written down.

2. The Evaluation Trigger. It's a node that starts the workflow once per row of the dataset, feeding it that row's inputs. Instead of sending an order by hand, the trigger goes through your answer key row by row and runs the workflow with each case. It's what turns "testing one case" into "testing all cases in one pass."

3. The Evaluation node. This is what grades. The docs report three operations for it (check the names on your version):

• Set Metrics — calculates and saves a score for the result: for example, "does the agent's classification match the expected one?" The metrics get recorded in the workflow's Evaluations tab.
• Set Outputs — writes evaluation results back to the dataset (the table or the sheet), so there's a record of what the workflow answered for each case.
• Check If Evaluating — branches the flow depending on whether the current run is an evaluation or a normal execution. It's used so that, during an evaluation, the workflow doesn't trigger its real side effects —it connects directly with lesson 4's dry run: "if I'm evaluating, don't write to the CRM."

Notice how well that last operation fits the module: Check If Evaluating is, in essence, a dry-run gate built specifically for testing. When the workflow runs inside an evaluation, that branch turns off the effects; when it runs for real, it lets them through.

Metrics: how it gets graded

A metric is the rule you grade each case by —the equivalent of "one point if the word matches." n8n's docs report several built-in metrics; these are the ones useful for verifying a classification like order-triage's (ranges per the 2026 docs; confirm them on your version):

Metric	What it measures	What it returns	When you use it in order-triage
Categorization	Whether the answer exactly matches the expected one	1 if it matches, 0 if not	The main one: did the agent classify manual_review when it should have classified manual_review?
String Similarity	How similar the answer is to the expected one, character by character	Between 0 and 1	For text that can vary a bit but should be similar
Correctness (with AI)	Whether the meaning of the answer agrees with a reference answer	From 1 to 5	For free-language outputs, where "same" is about meaning, not letters
Helpfulness (with AI)	Whether the answer answers what was asked	From 1 to 5	For evaluating a reply drafted to the customer
Tools Used	Whether the run used tools or not	Between 0 and 1	For verifying the agent used (or didn't use) the expected tool

And you can define Custom Metrics: you compute whatever score you want inside the workflow and map it to the Evaluation node with the Set Metrics operation. So you're not limited to the built-in ones.

For order-triage, the star metric is Categorization: the classification is a closed set of labels —approved, manual_review, missing_info— so "exactly matches the expected one" is exactly the right question, and it returns a clean 1 or 0. The AI ones (Correctness, Helpfulness) are for when the agent drafts free text, where letter-by-letter comparison doesn't help.

Worked example: evaluating order-triage's classification

Let's see it end to end. You want to verify the agent classifies your six synthetic cases correctly.

Step 1 — Build the answer key. You create the dataset (a Data table or a Google Sheet) with one row per case: the order's inputs and an expected column with the correct classification.

case	order_id	amount	customer_name	expected
happy-path-approve	ORD-TEST-001	1200	Café Aurora	approved
large-order-manual-review	ORD-TEST-002	52000	Tostaduría del Sur	manual_review
missing-customer-name	ORD-TEST-003	900	(empty)	missing_info
dirty-amount-as-string	ORD-TEST-004	"3,500.00"	Rincón del Café	approved
edge-zero-amount	ORD-TEST-006	0	Café Aurora	missing_info

Step 2 — Connect the evaluation trigger. You set the Evaluation Trigger to run order-triage once per row, feeding that row's inputs into the workflow. You point the agent at the local Ollama model (lesson 6), so evaluating the cases costs nothing.

Step 3 — Grade with the Evaluation node. After the agent, you put an Evaluation node with Set Metrics and the Categorization metric, comparing the classification the agent produced against the row's expected column. And you use Check If Evaluating so that, during the evaluation, the workflow doesn't write to the CRM.

Step 4 — Run the evaluation and read the verdict. You trigger the evaluation. n8n goes through the five rows, runs the workflow with each one, grades each result, and aggregates the scores in the Evaluations tab.

What to expect: you see a results table, one case per row, with its Categorization score —1 where the agent got it right, 0 where it failed— and an aggregate (for example, "4 of 5" or "80%"). If the agent misclassified dirty-amount-as-string —say it interpreted "3,500.00" as 3.5 and got confused by the formatting, sending it to missing_info when it should have gone to... well, that one should have been approved; let's say it got confused— that case comes out at 0, and you see it instantly, without having read a single output by eye. The evaluation points you exactly where the agent didn't get it right, across every case, in one pass. That's verifying the output.

The passing threshold

Grading each case isn't enough; you need a rule that says, looking at the aggregate, "this passes" or "this doesn't." That's the pass threshold: the minimum score for calling the test good, like the exam's "pass with eighteen out of twenty."

Choosing the threshold has more nuance than it looks, and depends on how critical each case is.

For critical cases, the threshold is 100%. There are classifications where getting it wrong is unacceptable, and there's no "almost" there. A 52,000-peso order going to manual review isn't negotiable: if that case fails, the test fails, even if every other one passes. Safety cases —the ones preventing real harm— get evaluated with a perfect threshold: zero tolerance.

For cases where some variation is acceptable, the threshold can be lower. An AI agent is probabilistic (lesson 6): even with low temperature, on an ambiguous case it might get it right 9 out of 10 times. For non-critical classifications, a threshold of, say, 90% can be reasonable —you acknowledge the agent isn't perfect and decide how much imperfection to tolerate. The key is that you choose the threshold consciously, not that you leave it to chance.

The practical way to combine them: split your cases into critical (100% threshold, any failure sinks the test) and normal (aggregate threshold, say 90%). The test passes if all the critical ones pass and the normal ones' aggregate beats its threshold. That way you protect what admits no error without demanding impossible perfection from what tolerates some noise.

An honest note about determinism: since the agent can vary between runs, an evaluation giving 100% today could give 95% tomorrow with the same cases. That's why the threshold and low temperature (lesson 6) work together: you lower the variation as much as you can, and the threshold absorbs what's left. If a critical case "sometimes passes and sometimes doesn't," that is a failure —a critical case has to pass every time— and it's the signal the agent's prompt needs more work, not that you should lower the threshold.

Assertions as a regression safety net

There's a use of assertions that goes beyond this single test and is worth seeing now, because it's where they're worth the most in the long run: they're your regression safety net. A regression is when a new change breaks something that used to work —you fix one thing and, without meaning to, break another. It's one of the most treacherous kinds of failure, because nobody's looking for it: you were focused on the new thing, and the old one broke behind your back.

A set of assertions is the defense against that. Once you have your six cases with their expected answer, every time you touch order-triage —change the prompt, add a rule, adjust the gate— you run the full evaluation again. If your change broke the classification of a case that used to pass, that case's assertion flips from 1 to 0, and you see it instantly. Without the net, that regression would travel silently all the way to production; with the net, you catch it on the next run, for free (local model, lesson 6). That's why assertions aren't a one-time expense: they're an asset that accumulates. Every case you add is one more guard watching forever that behavior doesn't break. A workflow with twenty assertions is a workflow twenty guards protect from regressions on every change.

Light evaluations and metric-based evaluations

n8n distinguishes two ways of using this system, and it's worth knowing which one applies in this module. The docs call them light evaluations and metric-based evaluations.

A light evaluation is the one you do during development, against a handful of cases you picked by hand —your six synthetic dummies. It's fast, runs a few carefully chosen cases, and answers "did my change break something that used to work?" It's exactly this module's thing: pre-production tests over representative cases.

A metric-based evaluation is heavier: it runs a large dataset and uses aggregate scores to watch quality continuously, typically already in or near production. It answers "is overall quality holding up over time, across many cases?" That's closer to quality monitoring —which, as we said, lives in the production maintenance guide— than to this module's pre-production testing.

For the pre-production checklist, your tool is the light evaluation: a few cases, hand-picked, each with its expected answer, run before promoting. You don't need a dataset of thousands to know whether your change broke the large orders' classification; you need the large-order case, well written, and its correct answer. Metric-based evaluation is where you scale up later, once you move on to watching quality over time; for now, six sharp cases are worth more than a thousand uncurated ones.

An assertion without the Evaluation node (the backup that always works)

Since this feature evolves and its availability can vary between versions, it's worth knowing how to build an assertion without depending on the Evaluation node, with basic nodes that exist in any n8n. Because the concept —comparing the output against what's expected and counting hits— matters more than the node, and if the node isn't there or changed, the idea is still yours.

The manual pattern has three pieces:

1. Compare case by case with an IF. After the agent, an IF node evaluates whether the agent's classification equals the case's expected column. The true output is a hit; the false one, a miss. It's the Categorization metric done by hand: matches or doesn't.

2. Mark each result. An Edit Fields node adds a passed: true/false field to each case depending on which side of the IF it came out on. That way every case carries its verdict.

3. Count and apply the threshold with a Code node. A Code node goes through every case, counts how many passed, calculates the percentage, and compares against your threshold. Since this is generating and transforming data in memory, n8n 2.0's Code node handles it fine —it doesn't call any API, just counts:

// Counts hits and applies the threshold. Receives all cases already marked with passed.
const cases = $input.all();                       // every evaluated case
const total = cases.length;
const passed = cases.filter(c => c.json.passed).length;
const rate = total === 0 ? 0 : passed / total;    // hit rate, 0 to 1

// Cases marked as critical must ALL pass, no exceptions.
const criticalFailed = cases.filter(
  c => c.json.critical && !c.json.passed           // critical AND failed
);

const verdict = (criticalFailed.length === 0) && (rate >= 0.9);  // 90% threshold + perfect criticals

return [{
  json: {
    total,
    passed,
    rate,                                          // e.g. 0.8 = 80%
    critical_failures: criticalFailed.length,
    verdict: verdict ? "PASS" : "FAIL",
  },
}];

What to expect: the Code node returns a verdict —PASS or FAIL— along with how many cases passed, the percentage, and how many criticals failed. It's the same criticality-based threshold logic from above, written by hand: the criticals must all pass (criticalFailed.length === 0) and the aggregate must beat 90% (rate >= 0.9).

When should you use the manual version and when the Evaluation node? The Evaluation node gives you the integration with the dataset, the trigger that goes through the rows, and the Evaluations tab where the record is kept —it's more complete and cleaner if it's available on your version. The manual pattern works on any n8n, doesn't depend on the feature existing under that name, and lets you see the mechanics with no layers. My recommendation: use the Evaluation node if your version has it and it works for you; keep the manual pattern in your back pocket for when it doesn't, or to understand what the node does underneath. In both cases, what matters —comparing the output against what's expected, counting, applying a threshold— is identical.

Common mistakes

Confusing "ran with no error" with "the output is correct" (conceptual, the central mistake). What happens: someone runs order-triage, sees everything green, and calls the test good without checking what the agent classified. If the agent got it wrong, the green hid it. Why it happens: green is very convincing; it feels like "approved." How to spot it: ask yourself whether you checked the output against an expected answer, or only that there was no technical error. If it's the latter, you didn't verify the result. How to fix it: write assertions. An answer key with each case's correct classification, and an automatic comparison. "Green" tells you the engine ran; the assertion tells you it landed where it should.

Writing vague assertions that can't be checked (practical). What happens: someone writes as "expected answer" things like "the agent classifies correctly" or "the output makes sense." That isn't an assertion: it can't be compared automatically because "correctly" and "makes sense" are judgments. Why it happens: it's easier to write a judgment than a concrete criterion. How to spot it: if your expected answer can't be compared against a 1/0 or a score, it's vague. How to fix it: turn every assertion concrete and checkable —"classifies as manual_review," not "classifies correctly." The Categorization metric needs an exact value to compare against; give it one.

Applying the same threshold to every case (conceptual). What happens: someone demands 100% across the board, and a test fails because a non-critical, ambiguous case got it right 9 out of 10 times; or the other way around, sets 80% for everything and lets a failure through on a security case that should have been perfect. Why it happens: a single threshold is simpler to configure. How to spot it: if you didn't split your cases by criticality, your threshold doesn't distinguish what's negotiable from what isn't. How to fix it: classify the cases —critical (100%, no exception) and normal (aggregate threshold)— and demand every critical one pass in addition to the aggregate. The 52,000 order going to manual review doesn't deserve the same treatment as an ambiguous formatting case.

Exercises

Exercise 1 — Make these assertions checkable. Each one is written vaguely. Rewrite it as a concrete assertion the Categorization metric could check, for an order-triage case. (a) "The agent handles large orders well." (b) "Incomplete orders get detected." (c) "The agent doesn't get confused by dirty data."

See solution

(a) → "For order ORD-TEST-002 at 52,000 pesos, the agent's classification equals manual_review." (b) → "For order ORD-TEST-003 with no customer name, the classification equals missing_info." (c) → "For order ORD-TEST-004 with amount "3,500.00" (text), the classification equals approved" (because it's 3500 pesos, a normal order, even though it's badly formatted).

Why it works: each concrete version names a specific case, an exact expected output, and a comparison (equals) that Categorization returns as 1 or 0. The difference from the vague ones: "handles well," "get detected," "doesn't get confused" can't be compared against anything; "equals manual_review" can. An assertion is a judgment turned into a criterion.

Exercise 2 — Choose the metric. For each thing you want to verify in a workflow, say which built-in metric you'd use and why: (a) that the classification is exactly one of three labels; (b) that the email drafted to the customer says, in essence, the same thing as a reference text; (c) that the agent used the CRM-query tool; (d) that the generated summary really answers the customer's question.

See solution

(a) Categorization — the classification is a closed set of labels; you want exact matching (1/0). (b) Correctness (with AI) — a drafted email varies in exact wording, but should agree in meaning with the reference; letter-by-letter comparison would fail on synonyms, so you need the meaning metric (1–5 scale). (c) Tools Used — measures precisely whether the run used tools. (d) Helpfulness (with AI) — measures whether the answer answers what was asked, which is exactly "does the summary answer the customer's question?"

Why it works: the metric gets chosen by the nature of the output. Closed label: exact match (Categorization). Free text that must mean the same: Correctness. Free text that must be useful: Helpfulness. Tool use: Tools Used. Using Categorization on free text would fail on every synonym; using Correctness on a closed label would be overkill.

Exercise 3 — Design the thresholds. You have six cases: two critical safety ones (a huge order that should go to review, and one with missing data that should be flagged incomplete) and four normal ones. Design the test's passing rule and justify it.

See solution

Rule: the test passes if both critical cases pass at 100% (each must come out 1, no exception) and the four normal ones' aggregate beats, say, 90% (at least the majority getting it right consistently).

Justification: the two criticals prevent real harm —auto-approving a huge order a human should have reviewed, or letting one with missing data through— and there's no "almost" there: one failure sinks the test even if everything else shines. The four normal ones tolerate some of the agent's natural variation (lesson 6), so a high but not perfect aggregate threshold acknowledges that imperfection without demanding the impossible. If a critical "sometimes passes," that counts as a failure: a safety case has to pass every time, and its inconsistency is a sign the prompt needs work.

Why it works: splitting by criticality lets the threshold protect what admits no error (100% on criticals) without punishing the agent's probabilistic nature on what does tolerate noise (aggregate on normals). A single threshold can't do both at once.

Summary and next step

In this lesson you took the leap from eyeballing the output to verifying it. An assertion is a checkable statement about the result —"the agent classified it as manual_review"— giving true or false with no opinion involved, your exam's answer key. With it, you finally distinguish "it ran" (the engine worked, all green) from "it got it right" (the output is correct), which is the only thing that really matters. You met n8n's Evaluation node and its three pieces: the dataset of cases with their expected answer (a Data table or Google Sheet), the Evaluation Trigger that runs the workflow once per row, and the Evaluation node with its operations —Set Metrics for scoring, Set Outputs for recording, Check If Evaluating for turning off effects during evaluation (a built-in dry run). You chose metrics based on the output —Categorization for closed-label classification, Correctness/Helpfulness for free text, Tools Used for tools— and defined passing thresholds by criticality: 100% for safety cases, a high aggregate for the ones tolerating the agent's natural variation. And you noted, as always, that the exact names and fields need checking on your version, holding onto the stable concept if the interface changed.

With this you check off the checklist's sixth point: the output passes assertions.

Before moving on you should be able to: explain the difference between "it ran" and "it got it right" with an example; name the three pieces of n8n's evaluation system; choose the right metric based on the output type; and justify why critical cases carry a 100% threshold.

You have the checklist's seven techniques. Lesson 8 brings them together into one complete test pass over order-triage: synthetic data, sandbox keys, dry run, replay, and assertions with the Evaluation node, all at zero cost with the local model. The deliverable is the checklist executed with its evidence —the portfolio artifact demonstrating you don't just build workflows, you test them like a system owner.

Resources

• Evaluation node — n8n Docs — the node and its operations (Set Metrics, Set Outputs, Check If Evaluating); check the exact names for your version there.
• Use metrics to measure quality — n8n Docs — the built-in metrics (Categorization, Correctness, Helpfulness, String Similarity, Tools Used) with their ranges, and custom metrics.
• Test and improve AI workflows — n8n Docs — the AI workflow evaluation section: datasets, the evaluation trigger, and the Evaluations tab.
• Data tables — n8n Docs — where your case dataset with its expected answer can live, the evaluation's answer key.
• IF node — n8n Docs — for the manual assertion pattern: comparing the agent's output against the expected answer without depending on the Evaluation node.