8. Project: a full sandbox test pass

Description

By the end of this project you will have run, end to end and at zero cost, a full test pass over order-triage: the seven points of the pre-production checklist, chained into a single repeatable procedure, with its evidence saved in the repository. You're not going to learn new techniques —you already have them all— you're going to bring them together into the artifact that gives the whole module its meaning: an executed checklist demonstrating, with proof, that order-triage is ready to promote to production without risking data, money, or reputation.

This matters because this deliverable is, literally, what the market asks for in writing and what you defend in an interview. When a posting says "test in a sandbox with synthetic data before deploying," it isn't asking that you know the theory; it's asking that you can show an executed test pass. By the end you have that: a file in cumbre-automations anyone —your team, an interviewer, you in three months— can read to see exactly how this workflow was tested and with what result. It's the tangible proof you crossed from "workflow builder" to "system owner."

Connection to the module: this is the checklist's seventh and last point —the complete pass with evidence— and the synthesis of the previous six. Every phase of this project is a lesson: sandbox keys (2), synthetic data (3), the dry run (4), pinned data (5), the zero-cost agent (6), and assertions (7). Here they don't get explained again; they get executed in order. And it sets up Module 6: once you have this pass green, the next step is promoting the tested change from staging to prod, with its rollback —which is exactly where the following module starts.

The deliverable: the executed checklist with evidence

Before doing anything, be clear on what you're going to produce. The deliverable is not a new workflow or a configuration; it's a document: a docs/pre-production-checklist.md file inside cumbre-automations, filled with the result of having run the seven checks on order-triage, plus the evidence backing each one.

Think of it as a pilot's signed pre-flight sheet. Before the plane leaves the gate, the pilot goes through a list —brakes, fuel, instruments, surfaces— and signs off every point: not "I think it's fine," but "I checked this, with this result, at this time." That signed sheet is the artifact saying "this aircraft is ready to fly," and it gets recorded. Your executed checklist is order-triage's pre-flight sheet: every point checked, with its evidence, signed with a Git commit.

Here's the template you're going to fill:

# Pre-production test pass — order-triage

- Workflow: order-triage
- Version tested (commit): __________
- Pass environment: dev + staging
- Date: __________
- Run by: __________

## Result

- [ ] 1. Sandbox keys — ran against the test CRM, never production
- [ ] 2. Synthetic data — 6 cases + volume batch, with edge and dirty cases
- [ ] 3. Dry run — side effects guarded (environment gate)
- [ ] 4. Pinned data — frozen inputs, reproducible pass
- [ ] 5. Zero-cost agent — iterated on Ollama, validated against the current model
- [ ] 6. Assertions — output verified against expected answers
- [ ] 7. Complete zero-cost pass — with attached evidence

## Verdict: PASS / FAIL

## Evidence
(screenshots, evaluation results table, cases that failed and their fix)

The project's goal is to fill each box with a real check and attach its evidence. Once all seven are checked and the verdict is PASS, order-triage is ready for Module 6. Let's go phase by phase.

So you see the goal before starting, here's what the checklist looks like already filled after a successful pass —this is what you produce at the end:

# Pre-production test pass — order-triage

- Workflow: order-triage
- Version tested (commit): a1b2c3d
- Pass environment: dev + staging
- Date: 2026-07-23
- Run by: (your name)

## Result

- [x] 1. Sandbox keys — dev's CRM API credential points at sandbox-api...; read returned mock data
- [x] 2. Synthetic data — 6 cases in test/fixtures/orders.json (happy, large, incomplete, 2 dirty, edge)
- [x] 3. Dry run — environment gate active; CRM node not executed in dev (screenshot attached)
- [x] 4. Pinned data — inputs pinned at the Webhook; two runs started from the same input
- [x] 5. Zero-cost agent — iterated on local llama3.2; validated 1 pass against the current model in staging
- [x] 6. Assertions — Categorization over the 6 cases: 6/6 at 1; criticals at 100%
- [x] 7. Complete zero-cost pass — ran entirely with no real writes or LLM charges

## Verdict: PASS

## Evidence
- test/evidence/2026-07-run-eval.png (evaluation results table)
- test/fixtures/orders.json (the 6 cases with their expected)

Notice the level of detail in every line: it doesn't say "done," it says what got checked and with what result. That concreteness is what separates a signed pre-flight sheet from a "yeah, I tested it." Now let's build each of those lines, phase by phase.

The full pass, phase by phase

A note on how to work through this project: don't read it straight through and call it done. Open it with your dev instance next to you and run each phase against order-triage as you go. If you don't have order-triage set up as is, do it against any workflow of yours with at least one side effect (a write somewhere) and, if you can, an AI node. This project's value isn't in understanding the phases —you already did that in the lessons— it's in actually running them once, because the first real pass is where you discover the details reading hides: that your $env wasn't accessible, that a dirty case broke the parsing, that the gate had the condition backward. Those stumbles are the learning; hunt for them here, in sandbox, where they're free.

Phase 0 — Preparation

Before the first check, have three things from Module 4 and previous lessons handy: your dev instance running (with the Starter Kit and Ollama), your staging instance, and the cumbre-automations repository where order-triage lives versioned. Write down in the template the exact commit you're going to test —the workflow version's identifier— because testing without recording which version you tested is like signing a flight sheet without saying which plane. The pass tests one specific version, and that version has to be written down.

Phase 1 — Sandbox keys (lesson 2)

What you do: you confirm your dev instance's CRM API credential points at the CRM's sandbox key, not the production one. You run lesson 2's smoke test: a read-only query that returns the test CRM's mock data —"Café Prueba" and company— not Cumbre's real customers.

What to expect: the read returns test data. You check the key's marker (test/sandbox) and the base URL (sandbox-api...).

Evidence: a screenshot of the credential (with the key blacked out, of course) showing the sandbox URL, or of the read's result showing mock data. You check box 1.

Phase 2 — Synthetic data (lesson 3)

What you do: you prepare your set of synthetic orders —the six labeled cases covering the family (happy, large, incomplete, two dirty, edge)— in the fixtures file test/fixtures/orders.json, and optionally the volume generator for a batch of a hundred. You confirm no data corresponds to a real customer.

What to expect: a versioned fixtures file, with cases that deliberately include the hard stuff: the 52,000 order, the one with the empty name, the one with the amount as text, the one with zero amount.

Evidence: the orders.json file itself, committed (it lives in Git, it's its own evidence). You check box 2.

Phase 3 — Dry run: guarded effects (lesson 4)

What you do: you confirm order-triage has its environment gate —the IF reading {{ $env.CUMBRE_ENV }} before the node writing to the CRM— and that in dev it diverts the flow to the node that doesn't write. You run the large-order-manual-review case and verify the CRM node stays gray, unexecuted.

What to expect: the flow goes down the dry-run branch, the CRM's HTTP node never gets colored in, and neither the sandbox nor the production CRM receives a new record. The run was complete; the write, zero.

Evidence: a screenshot of the canvas with the CRM node unexecuted and the dry-run branch active, or the Edit Fields node's output with { "dry_run": true, ... }. You check box 3.

Phase 4 — Pinned data: reproducibility (lesson 5)

What you do: you pin your cases' inputs —at the Webhook or the input Code node— so the pass is reproducible. You run the same case twice and confirm the input was identical both times.

What to expect: the input node shows its pin, responds instantly with no wait, and both runs start from exactly the same order. Remember: when exporting the workflow for versioning, you normalize and remove pinData (Module 3); the cases live in the fixture, not in pinData.

Evidence: a screenshot of the input node with the pin active, or a note that both runs started from the same input. You check box 4.

Phase 5 — Zero-cost agent (lesson 6)

What you do: two moves. First, in dev, you point the AI Agent at the local Ollama model (llama3.2 or mistral) with low temperature, and iterate the prompt until it classifies your cases correctly —free, as many times as needed. Second, in staging, you do one validation pass against production's current model, to confirm the logic holds up with the truffle and not just the mushroom.

What to expect: dozens of runs in dev with no charge; one validation pass in staging with the real model confirming (or fixing) what you tuned. You confirm the production model is current, not retired.

Evidence: a note of which local model you used to iterate and which current model you validated against, plus the validation's result. You check box 5.

Phase 6 — Assertions: verifying the output (lesson 7)

What you do: you set up the evaluation —with the Evaluation node and the Categorization metric, or the manual IF + Code pattern if your version doesn't have it— comparing the agent's classification against each case's expected column. You define the thresholds: 100% on critical cases (the large order, the incomplete one), a high aggregate (e.g. 90%) on the normal ones.

What to expect: a results table, one case per row, with 1 where the agent got it right and 0 where it failed, plus an aggregate and a verdict. If any case fails —the agent classified differently from expected— you see it here, without having read a single output by eye.

Evidence: the evaluation's results table (or the Code node's output with the verdict), including the cases that failed and how you fixed them. You check box 6.

Phase 7 — The complete pass and its evidence (synthesis)

What you do: you run the whole pass, chained: the trigger goes through the cases, the agent (local, free) classifies, the gate protects the effects, the evaluation grades against the expected answers, and you get an overall verdict. You confirm the whole pass ran at zero cost —local model, no real writes— and reproducibly.

What to expect: a PASS verdict if all the criticals passed and the aggregate beat its threshold; FAIL if not, with the culprit cases flagged. A FAIL isn't a project failure: it's the system doing its job —it caught a problem before production. You fix it, run again (free, because it's zero cost), and repeat until PASS.

Evidence: you fill in pre-production-checklist.md with the verdict, attach each phase's evidence, and commit it in cumbre-automations. That commit is your signature on the pre-flight sheet. You check box 7.

A pass that fails, and how to read it

It's worth walking through a pass that doesn't come out green on the first try, because that's what's going to happen to you in real life, and knowing how to read it is half the module.

You run the complete pass over order-triage. The trigger goes through the six cases, the agent (local, free) classifies each one, the evaluation grades. And the verdict arrives: FAIL. The results table says:

case	expected	agent answered	Categorization
happy-path-approve	approved	approved	1
large-order-manual-review	manual_review	manual_review	1
missing-customer-name	missing_info	missing_info	1
dirty-amount-as-string	approved	missing_info	0
dirty-whitespace-and-case	approved	approved	1
edge-zero-amount	missing_info	missing_info	1

Five of six at 1, one at 0. The culprit is dirty-amount-as-string: the 3,500-peso order with the amount written as text "3,500.00". You expected approved —it's a normal 3,500 order— but the agent sent it to missing_info. What to expect reading this: the agent got confused by the amount's format; it probably interpreted "3,500.00" as broken or incomplete data instead of as three thousand five hundred.

Now comes what matters: what you do with that 0. You don't lower the threshold. You investigate why the agent got confused and decide the right fix, which here can be of two kinds:

• Fix the prompt so the agent understands amounts with thousands separators as text. You adjust, and since iterating on Ollama is free (Phase 5), you run again at no cost until the case passes.
• Fix the workflow so it cleans the amount —converting "3,500.00" to 3500— before it reaches the agent, with a normalization Code node. Maybe the agent shouldn't have to deal with dirty data; maybe that's a previous step's job.

Which of the two is "the correct one" depends on your design, and that decision is exactly the value of having found the failure in sandbox: you're solving it calmly, for free, on synthetic data, and not at three in the morning with a real 3,500-peso order stuck at missing_info in production. You fix it, run the whole pass again (at zero cost), and this time dirty-amount-as-string comes out at 1. Verdict: PASS. Now it's ready.

Notice what just happened: the checklist did exactly its job. It caught a realistic dirty case lesson 1's "ran once and looked fine" would never have seen —because that happy case never included a badly formatted amount. The FAIL wasn't a project problem; it was the project working.

When you run the pass: full versus smoke

Not every change deserves the full seven-phase pass, and pretending it does leads to nobody running it. It's worth calibrating the effort to the change, with two levels.

The complete pass —the seven phases— runs when the change is substantive: you touched the agent's logic, changed the classification prompt, added a new side effect, or you're going to promote to production. Anything that might alter what the workflow decides calls for the complete pass, because that's exactly what the assertions verify.

A smoke test —a quick subset— runs for minor changes almost certainly not touching the behavior: you renamed a node, tweaked a comment, moved something on the canvas. A smoke test runs only the critical cases —the large order, the incomplete one— to confirm in thirty seconds "nothing obvious caught fire," without the full ceremony. The name comes from electronics: you power the device on and see if smoke comes out; if not, at least the basics work.

The calibration rule: when in doubt, complete pass. The smoke test is for when you're sure the change doesn't touch behavior; if you have to ask yourself "could this change how it classifies?", the answer is already to run the complete one. And before promoting to production, always the complete one, no exceptions —there are no "minor" changes there, because crossing into production is itself the major event. This calibration is what makes the checklist sustainable: if every comma demanded seven phases, you'd abandon it in a week; if nothing demands them, you never test. The middle ground is what turns it into a real habit.

A note opening up Module 6: this pass, which you run by hand here, is a candidate for automation. A CI check —continuous integration— can run at least the most mechanical checks (that the JSON is valid, that the structure is right) on every commit, without you triggering them. That's the next module's topic; for now, running it by hand teaches you what you're later going to automate.

Where the evidence lives (and why in the repo)

All the evidence —the filled checklist, the screenshots, the results table, the case fixture— lives inside the repository cumbre-automations, not in your Downloads folder or in your head. A reasonable structure:

cumbre-automations/
├── workflows/
│   └── order-triage.json          # normalized, no pinData (Module 3)
├── test/
│   ├── fixtures/
│   │   └── orders.json            # the synthetic cases with their expected
│   └── evidence/
│       └── 2026-07-run.png        # screenshots of the pass
└── docs/
    └── pre-production-checklist.md # the pre-flight sheet, filled and signed

Having the evidence live in Git has three consequences that make it truly valuable, not bureaucracy. It gets versioned: today's pass and next month's coexist in the history; you can see how the test evolved. It's reproducible by someone else: anyone on the team clones the repo, reads the checklist, runs the same fixture cases, and gets the same verdict —the test doesn't depend on you. And it's a portfolio artifact: in an interview, instead of saying "I test well," you open the repo and show a real workflow's signed pre-flight sheet. The second one weighs infinitely more than the first.

This is the deep reason the checklist is a file and not a habit: a habit can't be shown, versioned, or handed over. A file can. The day you leave Cumbre, the next person opens pre-production-checklist.md and knows exactly how order-triage gets tested before anyone touches it. That's being a system owner: leaving the system operable by someone else.

An honesty note about evidence: not all evidence ages equally well. A screenshot is useful the day you take it, but it grows stale —the interface changes, the data changes— and nobody can "re-verify" it. The most valuable evidence isn't the kind that shows the result, but the kind that lets you reproduce it: the fixtures file with the cases and their expected, and a note of how to run the evaluation. With that, someone else doesn't have to trust your screenshot; they run the test and get their own verdict. Always prefer reproducible evidence (the fixture, the procedure) over static evidence (the screenshot); the screenshot accompanies, the fixture proves. It's the same "reproducible" principle from lesson 5, now applied to how you save proof that you tested.

Common mistakes

Checking a box with no evidence (conceptual, hollows out the exercise). What happens: someone goes through the checklist checking boxes from memory —"yeah, I ran against sandbox, check"— without attaching proof. The checklist ends up filled but hollow: nobody, not even the author in a month, can confirm the check actually happened. Why it happens: checking is fast; capturing evidence takes one more minute. How to spot it: if a checked box has no screenshot, table, or file backing it up next to it, it's a box of faith, not of proof. How to fix it: the pilot's rule —you don't sign off on a point you didn't verify with the instrument in front of you. Every checked box carries its attached evidence; without evidence, the box stays unchecked.

Testing one version and promoting another (practical and dangerous). What happens: someone runs the complete pass, gets PASS, and then makes "a tiny last-minute tweak" to the workflow before promoting, without testing again. They promoted a version that never passed the checklist. Why it happens: the tweak seems harmless, and running the pass again feels like a hassle. How to spot it: compare the commit you wrote in the template against the commit you promote; if they aren't the same, you tested one thing and are promoting another. How to fix it: the pass tests one exact version, identified by its commit. Any change after the pass —however small— invalidates the verdict and requires running it again. Since it's zero cost and reproducible, running it again is cheap; do it.

Treating a FAIL as a failure instead of the system succeeding (conceptual). What happens: the pass gives FAIL, and someone gets frustrated or —worse— lowers the threshold so it gives PASS. Why it happens: a FAIL feels like something went wrong. How to spot it: if your reaction to a FAIL is adjusting the threshold instead of fixing the workflow, you're shooting the messenger. How to fix it: a FAIL is exactly what the checklist should produce when there's a problem —it caught a bug before production, which is the module's entire goal. Celebrate the FAIL: it saved you an incident. Fix the workflow, not the threshold, and run again. Lowering the threshold so it passes is like taping over the fire alarm: it silences the warning, it doesn't put out the fire.

Exercises

Exercise 1 — Order the pass. Without looking, write the complete pass's seven phases in the correct order, and for each one, the lesson it comes from. Then justify why sandbox keys come before assertions.

See solution

The order: (1) sandbox keys (lesson 2), (2) synthetic data (3), (3) dry run / guarded effects (4), (4) pinned data / reproducibility (5), (5) zero-cost agent (6), (6) assertions (7), (7) complete pass with evidence (synthesis).

Why sandbox keys come before assertions: the order goes from preparing the safe environment to verifying the result. It makes no sense to verify the output (assertions) if you might still be touching production: first you make sure nothing you run has real consequences (sandbox, synthetic data, dry run), then you make it reproducible (pinned data), then you make it cheap (local agent), and only at the end do you verify the output is correct. Verifying first and protecting afterward would be checking your aim after you already fired.

Why it works: the order isn't an arbitrary list; it's the logical sequence of "make it safe → make it repeatable → make it cheap → verify it's right." If you know it in order, you know the module.

Exercise 2 — Actually run one phase. Take Phase 6 (assertions) and run it against order-triage (or against one of your workflows with an AI Agent node). Build at least three cases with their expected, run the evaluation —with the Evaluation node or the manual pattern— and write the verdict with its evidence. If you don't have order-triage set up, describe it in detail: which three cases, which metric, which threshold, what evidence you'd save.

See solution

There's no single answer, but a good run has these pieces: three cases covering different ground —one happy (approved), one critical (manual_review for the large order), one incomplete (missing_info); the Categorization metric (exact match against expected); thresholds of 100% on the critical one, a high aggregate on the rest; and as evidence, the results table with each case's 1/0 and the verdict.

If a case fails, the correct move isn't lowering the threshold: it's reading why the agent classified differently —is the prompt ambiguous? does the local model differ from production's?— fixing it, and running again (free).

Why it works: actually running one phase, even just one, turns the module from theory into muscle memory. Phase 6 is the richest one because it combines data (the cases), AI (the agent), and verification (the metric); if you can run it alone, you can run the whole pass.

Exercise 3 — Defend it in an interview. An interviewer asks you: "how do you test a workflow that writes to the real CRM and uses an AI agent, without risking production or overspending?" Answer in one paragraph, as you would in the interview, using the module's vocabulary.

See solution

A solid answer sounds like this: "I test it in an isolated sandbox —my dev environment— before touching production. I connect the CRM node to a sandbox key, so no write hits the real CRM; and for cases where not even the sandbox should get touched, I have an environment gate that cuts the effect in a dry run. I test with synthetic data covering edge cases and dirty data, not real customer data, for privacy. I pin those inputs so the test is reproducible. I test the agent against a local Ollama model to iterate at zero cost, and validate once against production's current model before promoting. And I don't stop at 'it ran': I write assertions with the Evaluation node comparing the agent's classification against each case's expected answer, with a passing threshold. Everything ends up as an executed checklist with evidence in the repository, versioned, that anyone on the team can run and reproduce."

Why it works: that answer uses, in order, the module's seven techniques by their market name, and closes with the deliverable. An interviewer asking for "test in a sandbox" recognizes every piece. The difference from a candidate who "knows how to build workflows" is exactly that paragraph: it doesn't describe how they build a flow, it describes how they test and deliver it like a system owner.

Summary and next step

In this project you ran the complete test pass over order-triage: the checklist's seven chained checks —sandbox keys, synthetic data, dry run, pinned data, zero-cost agent, assertions, and the complete pass— at zero cost and reproducibly. You produced the deliverable giving the module its meaning: pre-production-checklist.md filled and signed with a commit, its evidence versioned in cumbre-automations —order-triage's pre-flight sheet. And you saw why that artifact matters: it stays versioned, someone else can reproduce it, and it's what you show in an interview instead of just claiming you test well. You locked in the three mistakes that hollow it out —checking without evidence, testing one version and promoting another, treating a FAIL as a failure instead of the system succeeding.

With this you check off the checklist's seventh and last point, and complete the module's exit capability: fully testing a workflow in a sandbox, without touching production or burning budget, leaving a repeatable checklist with versioned evidence.

Before moving to Module 6 you should be able to: recite the seven phases in order with their lesson; run at least one phase for real against a workflow; defend your test process in one paragraph using market vocabulary; and explain why reproducible evidence (the fixture) is worth more than static evidence (the screenshot).

What follows closes the delivery cycle. You have order-triage versioned (Modules 2 and 3), in isolated environments (Module 4), and now tested with evidence (this module). Module 6 teaches you to promote that tested change from staging to prod in a controlled way, to review changes as diffs —including the AI part— to build a rollback runbook to go back in minutes if something fails, and to close with a CI check validating the JSON on every commit. The green pass you produced here is precisely the green light Module 6 needs to cross the last line: from "tested" to "in production, with a safety net."

Resources

• Test and improve AI workflows — n8n Docs — the section bringing together evaluation, datasets, and metrics; Phase 6's framework.
• Data mocking and pinning — n8n Docs — pinning data for Phase 4's reproducibility.
• Self-hosted AI Starter Kit — n8n Docs — the Ollama environment making Phase 5 zero cost.
• Manual, partial, and production executions — n8n Docs — the manual executions the whole pass runs on (remember: the pin doesn't apply in production).
• Source control and environments — n8n Docs — the environments and versioning framework where the checklist lives; the bridge to Module 6's promotion.