8. Final project: the operations kit

Description

By the end of this lesson you'll have assembled and tested the complete Terra Market operations kit: the runbooks for the three most frequent symptoms, the on-call rotation with its escalation criteria, the retrospective template, and the inventory updated with owners. And you'll have validated it with the only test that counts: a drill where another person executes a runbook without your help. It's this module's project and the closing of the whole guide.

This matters because an untested operations kit is exactly as reliable as an error workflow that never fired, and you already know that argument well. Everything you built in the previous seven lessons —the runbooks, the on-call policy, running the incident, the retrospectives, the handover— exists for a single purpose: for the operation to survive without you. And that purpose isn't declared, it's demonstrated, with another person resolving an incident you provoked, while you bite your tongue. This lesson is that demonstration, assembled piece by piece into concrete and verifiable deliverables.

Connection to the module: this lesson integrates the four pieces of the kit that lessons 2 to 7 built separately. It doesn't introduce new ideas: it gathers, assembles, and tests what you already know. And since it's the last lesson of the last guide —or of this guide, at least— it closes two things: the module, with the kit delivered and validated; and the whole guide, with a tour through the eight modules and a map of where to go next. The module's boundary stands until the end: here there's no Git, no environments, no backups, no migration —that lives in the sister guides you'll see at the close—; here is the human operation, assembled.

What you're going to deliver

The kit has four deliverables, one for each piece. Think of it as the folder you leave to whoever takes on the operation when you're not around:

OPERATIONS KIT · Terra Market
│
├── 1. RUNBOOKS (piece: the script)
│      RUNBOOK-01 · the ERP isn't responding
│      RUNBOOK-02 · duplicate orders arrived
│      RUNBOOK-03 · the dead_letter queue grew
│
├── 2. ON-CALL (piece: who responds)
│      Weekly rotation + level policy + escalation criteria
│      + the weekly announcer (workflow)
│
├── 3. RETROSPECTIVE (piece: how you learn)
│      The five-section template, ready to use
│
└── 4. LIVING INVENTORY (piece: the memory)
       The 18 workflows with criticality, owner, runbook and living notes

And on top of the four, the validation: the drill, which isn't a document but a test that gets executed and passed. A kit with the four deliverables written but no drill isn't finished, just as error handling that's configured but not tested isn't finished.

Let's go step by step. For each deliverable you'll see what it must contain and how to verify it turned out well. You already wrote the runbooks in lesson 3, so here the work is to gather them and verify they meet the bar; the other three pieces get assembled here.

Phase 1 — The three runbooks

The first deliverable is the three complete runbooks from lesson 3: the ERP isn't responding (RUNBOOK-01), duplicate orders arrived (RUNBOOK-02), and the dead_letter queue grew (RUNBOOK-03). We don't repeat them here —you already have them written— but we do run them through a checklist, because a runbook that exists isn't the same as one that works.

The checklist for a runbook. Run each of the three through these boxes:

□ The title is an observable SYMPTOM, not a cause.
  (✔ "the ERP isn't responding" · ✘ "authentication problem")
□ It starts with a QUICK CHECK that answers "is anything being lost?"
  or the question that orients the urgency of that symptom.
□ The diagnosis is in ONE sentence, with the reassurance of whether
  it's expected.
□ Each action step is an ACTION, not a decision, and has its
  EXPECTED RESULT.
□ It says WHEN TO ESCALATE and TO WHOM, by name, and there's a direct
  route to escalate from the check when the case is serious.
□ It has a "WHAT YOU MUST NEVER DO" section with the shortcuts
  that destroy data.
□ It has a last-reviewed DATE and an OWNER in the header.
□ Someone who's NOT you read it and understood it without asking you.

The last box is the most important one and the one almost nobody checks: a runbook isn't finished until another person read it and understood it without you. The previous seven boxes you can check on your own, fooling yourself; the eighth requires another person, and it's the one that really proves the runbook is written for whoever's going to use it and not for whoever wrote it.

How to verify it turned out well. Take RUNBOOK-01 and give it to the least experienced person on the team —Daniela— without any context. Ask her to tell you, reading it, what she'd do step by step if she got the alert. Every time she asks you something the runbook should answer, you have a gap: you fix it. When Daniela can narrate the complete resolution with only the runbook in hand, that runbook passes.

Phase 2 — On-call with its escalation criteria

The second deliverable assembles what's in lesson 4 into an operable document. It has three parts:

Part A — The rotation. Who's on the wheel, in what order, and with what stretch. For Terra Market:

ON-CALL ROTATION · Terra Market
Shift: weekly (Monday to Monday). Wheel of 5:
  You → Marco → Sofía → Iván → Renata → (back to You)

OFF THE WHEEL (for now):
  · Daniela: in accompaniment. Joins when she completes her drill
    of the 3 runbooks without help.

Each week's shift is announced automatically on Mondays at 09:00
in #ops-daily (workflow "on-call-announcer").

Part B — The level policy. The decision, made beforehand, of what wakes someone up. It's the table from lesson 4, with the golden rule visible:

ALERT POLICY · what wakes you and what doesn't

WAKE NOW (Slack #ops-alerts + mention + phone)
  · Something is being LOST (the quick check returned "doesn't grow").
  · The damage GROWS uncontained (irreversible emails going out).
  · A critical trigger dead (no orders coming in).
  Target: 0 or 1 per night.

WAITS UNTIL TOMORROW (summary in #ops-daily)
  · dead_letter growing BUT with everything safe.
  · Visible but contained failures (duplicates that weren't sent).
  Target: fewer than 10 a day.

JUST LOGGED (error_log, no human channel)
  · Successful retries, deliberate business rejections.

GOLDEN RULE: a notice is only postponed if nothing is being lost AND
the damage is contained. If in doubt, wake.

Part C — The escalation criteria. Who you turn to when the shift isn't enough. This is the piece that makes it possible for someone inexperienced to take on-call without fear:

ESCALATION · when the shift isn't enough

LEVEL 1 · whoever has the shift this week.
  Resolves with the runbooks. If the runbook is enough, doesn't escalate.

LEVEL 2 · the BACKUP (the next one on the wheel).
  Escalate if: the runbook doesn't cover the symptom, or the runbook's
  action didn't work, or a second person is needed to communicate while
  someone resolves.

LEVEL 3 · automation (Mike) for the technical part / Marco for
  business decisions (e.g.: unpublish order-sync).
  Escalate if: something is being lost and can't be contained, or
  the incident exceeds what the backup can resolve.

Escalating is ALWAYS valid. Escalating isn't failing: it's what makes
on-call takeable by someone who doesn't know everything.

How to verify it turned out well. Two checks. First: measure a real week. Count how many alerts reached #ops-alerts and what fraction required action. If the volume exceeds "0 or 1 per night" or the actionability drops below ~80%, the policy isn't calibrated: review what's slipping through and reclassify it (don't raise the threshold). Second: verify that the weekly announcer posts to #ops-daily every Monday. It's the only workflow in the kit and it's easy to forget.

Phase 3 — The retrospective template

The third deliverable is the template from lesson 6, ready to copy when the next incident occurs. It's not a document filled out once: it's a format that gets reused. This is it, so it stays in the kit:

RETROSPECTIVE TEMPLATE · copy for each incident

RETROSPECTIVE · [title by symptom]
Incident: [date] · Retro: [date, a few days later]
Participants: [who was at the front + who analyzes]

① WHAT HAPPENED (3 lines, plain language)

② TIMELINE (from the log written DURING the incident,
   with real times)

③ IMPACT (in business language, MEASURED: how many orders,
   how much time, how many customers)

④ CONTRIBUTING CAUSES (what made it possible — the chain of
   "why?" until reaching SYSTEM causes, never a person. If a
   proper name appears, keep asking.)

⑤ ACTIONS (each with an OWNER with a name and a verifiable DATE.
   No owner and nobody does it; no date and it's done never.)

REMINDER WHEN OPENING THE MEETING (read out loud):
"We're not looking for culprits, we're looking for causes. Tell the
uncomfortable details: they're the ones that fix the system. The
question is what made it possible, not who did it."

Notice the reminder at the end: it's not decorative. Reading out loud that you're not looking for culprits, at the opening of each retrospective, is what creates the conditions for the information to come out. Without that explicit framing, the meeting drifts on its own toward the hunt, because looking for culprits is the default state of any group faced with a mistake.

How to verify it turned out well. The template is tested by using it. Take a recent real Terra Market incident —Daniela's reprocessing from lesson 6, for example— and fill it out entirely. If by the end you have causes that are all from the system (no proper name in section ④) and actions that all have an owner and a date, the template works. If you struggle to fill in the timeline, it's a sign that in the next incident you have to write the log better during —the template depends on that raw material—.

Phase 4 — The living inventory

The fourth deliverable is the inventory from lesson 7: the 18 workflows with criticality, owner, runbook, and living notes, with a last-reviewed date and someone responsible for maintaining it. You already saw its form; here the work is to make sure it's complete and alive, not just started.

A complete inventory covers all 18, not just the 3 protagonists. The three critical ones carry their runbooks; the support ones (error-handler, dead-letter-watch) carry their special notes (like "don't test by hand"); and the other thirteen carry at least criticality and owner, even if it's one line each. An inventory that only covers the three you know by heart leaves out precisely the fifteen nobody remembers when they fail.

How to verify it turned out well. Three checks. First: are all 18 there? Count the rows. Second: does each row have a named owner? A workflow with no owner is a workflow nobody's going to tend to. Third —the proof that it's alive—: take three concrete statements from the inventory (a credential name, a rotation date, a contact) and verify them against the real system now. If any lies, the inventory has already started rotting and it has to be fixed and the quarterly review scheduled.

The validation: the drill

Here is the part that turns four documents into a real operations kit. The four deliverables can be impeccable on paper and still not work, for the same reason an error workflow can be well configured and not fire on the day it's needed. The only way to know is to run a drill: another person resolves an incident you provoke, with the kit, without your help.

This is the complete drill of the Terra Market kit, step by step:

Step 1 — Choose the incident and the person. The incident: the ERP that isn't responding (RUNBOOK-01), being the most frequent and critical. The person: Daniela, who's in accompaniment and whose entry onto the on-call wheel depends precisely on passing this drill.

Step 2 — Provoke the symptom without touching production. Set up a disposable order-sync-test that points at a test endpoint returning 503, so that Daniela sees the real alert, the node failing, and the code, without any real order being at risk. The symptom has to be indistinguishable from the real one; a description on paper proves nothing.

Step 3 — Daniela resolves on her own, and you stay quiet. You give her the kit —the runbooks, the inventory, the escalation policy— and let her work. You observe and take notes of where she gets stuck. You don't intervene. Every time she doubts and doesn't find the answer in the kit, you note a gap. You remind her before starting: this tests the kit, not her; getting stuck is fine and is useful information.

Step 4 — Each stumble is a fix to the kit. Didn't find the quick check? It was buried in the runbook: you move it up. Didn't know who to escalate to in the predawn? The escalation didn't say clearly: you clarify it. Didn't know that reprocessing without verifying duplicates? It was missing from the "never": you add it. The drill produces a list of concrete fixes to the kit.

Step 5 — Repeat with an incident she hasn't practiced. When Daniela resolves the ERP one without help, test with another she hasn't seen —the duplicate orders, for example—. The handover is done when she resolves a new incident with only the kit, without asking you anything. That moment is the operational definition that the kit works, and it's the criterion for putting Daniela on the on-call wheel.

What to expect. The first drill almost always uncovers several gaps —it's normal and it's the point—. The second, fewer. At some point, Daniela resolves an incident she hadn't seen using only the kit, and there you know two things at once: that the kit is complete, and that you have one more person who can take on-call. Both are the goal of the whole module, demonstrated in one stroke.

And a consequence worth naming: the day Daniela passes the drill, you finally answered the question that opened the guide back in lesson 1 of Module 2 —what happens if you're not here tomorrow?—. The answer stopped being "they call me" and became "there's a kit, and there's at least one more person who knows how to use it". That's what it means for an operation to be in production and not in your head.

The kit delivery checklist

The kit is finished when you can check these boxes. It's the publication checklist of the whole module:

□ RUNBOOK-01, 02, 03 written, each passing the 8 boxes of the
  checklist (including "someone else understood it").
□ On-call rotation defined, with Daniela OFF until she passes the
  drill.
□ Level policy written, with the golden rule visible.
□ Escalation criteria with names and the three levels.
□ Weekly announcer posting to #ops-daily every Monday.
□ Retrospective template ready, with the "blameless" reminder
  at the opening.
□ Living inventory with the 18 workflows, each with an owner, a
  review date, and someone responsible for maintaining it.
□ DRILL PASSED: another person resolved a new incident with the
  kit, without help.

Notice that the last box is the one that counts for all the others. You can have the first seven checked and, if the eighth isn't, you don't know if the kit works —you only know it looks good—. And you can have a modest kit that checks the eighth, and that one is a real operations kit. The drill isn't the last step of the project: it's its only judge.

Common mistakes

Delivering the kit without the drill (conceptual). What happens: the four deliverables are written with care, saved in a folder, and the kit is considered finished. It was never tested with a real person resolving an incident, so the gaps —the runbook with an ambiguous step, the escalation that doesn't cover the predawn— are still there, invisible, waiting for the first real incident to appear at the worst moment. Why it happens: writing the documents feels like the work, and the drill feels like an optional extra "if there's time". How to spot it: if nobody but you has resolved an incident with the kit, the kit isn't tested. How to fix it: treat the drill as part of the definition of done, not as a luxury. A kit without a drill is error handling without provoking failures: infrastructure that's supposed to work and that nobody verified.

Confusing "four documents" with "sustainable operation" (conceptual). What happens: the kit is produced, filed, and it's assumed that the operation now survives without the key person. But a kit is a snapshot of a moment; without maintaining it, in six months the runbooks lie, the inventory is outdated, and the rotation has unraveled. The kit exists but no longer describes reality. Why it happens: the kit is a tangible artifact, and it's easy to confuse having the artifact with having the capability it represents. How to spot it: look at the last-reviewed date of your inventory and your runbooks. If it's more than a quarter ago, the kit is rotting. How to fix it: tie the maintenance to events that already happen —every retrospective updates the runbooks and the inventory it touched— and schedule a complete quarterly review. The kit isn't a deliverable that gets finished; it's an organism that's kept alive or dies.

Putting someone on on-call before they pass the drill (practical). What happens: out of haste to share the load, the new person is put on the on-call wheel as soon as the kit is written, without having proven they can use it. An incident falls to them, they get stuck in a gap the drill would have found, and they end up calling you —or worse, making an ill-informed decision—. Why it happens: there's pressure to unload the expert, and putting someone on the wheel looks like immediate progress. How to spot it: if someone in your rotation hasn't resolved a simulated incident with the kit without help, they're not ready for the shift. How to fix it: the drill is the entry door to on-call, not the calendar. Daniela joins the wheel the day she passes the drill, not before and not for any other reason. It's the order from lesson 4 —capability first, shift after— made a requirement.

Exercises

Exercise 1 — Audit an incomplete kit. A team says its operations kit is ready. On reviewing it you find this: three well-written runbooks; a rotation of six people including the newest one; a policy that notifies of every failure to a single channel; no retrospective template; an inventory of the three critical workflows; and no drill done. Identify the four problems and order them by severity.

Exercise 2 — Design the escalation criteria of a new workflow. Terra Market launches refund-process (processes refunds, moves real money, ~40 executions/day). Write its escalation entry in three levels, deciding what justifies moving up a level, given that there's money involved here.

ESCALATION · refund-process (moves money)

LEVEL 1 · whoever has the shift.
  · Transient payment-processor failure that the retry resolves
    → doesn't escalate, it resolved itself.
  · Failed validation (order doesn't exist, out of window) → doesn't
    escalate, it's a deliberate rejection, to error_log.

LEVEL 2 · the backup + finance informed.
  Escalate if: the payment processor fails after exhausting retries and
  there's a refund that didn't go out. Each case matters because it's a
  person waiting for their money.

LEVEL 3 · automation + Marco, NOW, at any hour.
  Escalate if: the state is AMBIGUOUS — you don't know whether the payment
  went out and only the confirmation was lost, or whether it didn't go out.
  That ambiguity with money involved doesn't wait until morning, even at 3 a.m.

Exercise 3 — Run a mental drill. You don't have Daniela in front of you, but you can do the drill in your head, which is a good gap detector. Take RUNBOOK-01 (the ERP isn't responding) and put yourself in the shoes of someone who's never used it, at 3 a.m. Walk through it line by line and note every point where someone without your context would get stuck. Propose the fix for each gap.

Summary and next step

In this lesson you assembled the complete Terra Market operations kit: the three runbooks run through their checklist, on-call with its rotation, its level policy, and its three-level escalation criteria, the retrospective template with the "blameless" reminder, and the living inventory with the 18 workflows. And you validated it with the only test that counts: the drill, where another person resolves an incident you provoke, without your help, and where each stumble fixes the kit instead of reproaching the person. You saw that the last box of the delivery checklist —the drill passed— counts for all the others, because a kit that looks good but wasn't tested is exactly as reliable as an error workflow that never fired. And the day Daniela passes the drill, you answered the question that opened the guide: what happens if you're not here tomorrow. The answer is no longer "they call me".

Summary and closing of the guide

You reached the end. It's worth looking back and seeing, all at once, everything you now know how to do, because it's eight modules and it's easy to lose sight of the whole.

You started without knowing what makes a workflow "production-grade" and now you operate automations a business depends on. This was the path, module by module, and the capability each one left you:

Module 1 — What changes when a workflow goes to production. It gave you the criterion: the criticality × reversibility matrix, the readiness audit, and the inventory with owners. You learned to look at a workflow and say how critical it is and what happens if it fails. Without this, everything else would be effort spread out blindly.

Module 2 — Robust error handling. It gave you the five layers of defense: the retry that heals on its own, the branch that sets aside, the failure you provoke, the global error workflow that collects, and the notification that alerts. You learned to design for failure, because in production failure isn't a possibility but an agenda.

Module 3 — Debugging with the replay engine. It gave you how to stop guessing: read an execution, replay a past failure, pin data, re-run a single node. You learned to understand what really happened, instead of assuming it.

Module 4 — Observability. It gave you the eyes: structured logging, audit trail, metrics, health checks, and SLA. You learned to answer "what happened with order 4471?" nine days later, which is the difference between operating and hoping.

Module 5 — Credential and secrets security. It gave you the keys well kept: least privilege, rotation, and the spending cap on the AI token. You learned to keep a compromised system from dragging the others down.

Module 6 — Workflow performance. It gave you how to make the slow fast: measure where the time goes, process in batches, mind the memory, reduce calls, handle concurrency. You learned to attack the real bottleneck, not the imagined one.

Module 7 — The cost of AI in production and MCP. It gave you control of the spending: the cap, the per-execution measurement, and the risk of exposing and consuming MCP servers. You learned to keep AI in production from ruining the budget or opening a security hole.

Module 8 — Runbooks, on-call, and continuity. It gave you what none of the previous seven could give: for the operation to survive without you. The runbook another person executes at 3 a.m., the on-call that shares the load without burning anyone out, running the incident that stops the bleeding before understanding it, the retrospective that learns without blaming, and the handover that makes everything learned survive the people.

Put the eight together and you have the sentence the guide opened with, now defensible in an interview: "I operate automations a business depends on, and I can go on vacation." The first seven modules made you capable of operating; the eighth made you dispensable in the good sense, which is the only state in which a system is truly in production.

Where to go next

This guide has deliberate boundaries —things it decided not to cover in order to do its own well—, and the sister guides in the ecosystem cover precisely those boundaries. Here they are, grouped not by catalog but by the problem that's going to lead you to them. When you run into one of these problems, you already know where to go:

If you need to version workflows and move them between environments without breaking production —"I want a separate dev and prod", "I want to review the changes before they go in", "who can edit what"— that's n8n-git-and-environments-guide. It's the boundary this module pointed to over and over: Git, dev→staging→prod environments, promotion, and permissions.

If you have to set up, harden, or scale the instance itself —"how do I install n8n on a server", "how do I back up and restore", "how do I update the version without breaking anything", "how do I move from Cloud to self-hosted", "queue mode with Redis and workers"— that's n8n-self-hosting-and-operations-guide. This guide assumed an instance already running; that one builds it and maintains it.

If the problem is that the system does incorrect things, not that it fails —"a retry duplicated a charge", "how do I design an idempotency key", "how do I undo an effect that already happened", "how do I define a contract between two workflows"— that's n8n-workflow-contracts-and-idempotency-guide. It was the most-cited boundary of this whole module: that guide designs the system's correctness, this one operates it.

If you want to build AI agents, chatbots, or document pipelines —"how do I set up an agent that uses tools", "how do I make a chatbot over my documents", "how does a RAG pipeline work"— that lives in the ecosystem's AI guides (n8n-ai-chatbots-agents-guide, n8n-document-ai-guide, n8n-ai-automation-basics-guide). This module only controlled what AI costs and risks in production; building it is over there.

If you're still learning to build workflows —nodes, triggers, expressions, the basic logic— that's the fundamentals guide (n8n-fundamentals-guide), and the ones on design patterns (n8n-workflow-design-patterns-guide) and working with APIs without code (n8n-apis-without-code-guide). This guide assumed you already knew how to build and taught you to operate; if you feel you lacked the base, it's there.

And one last thought to close. Operating seriously isn't a destination you arrive at, it's a craft you practice —that of the second bridge builder, the one who walks over it looking for cracks—. The kit you assembled isn't finished work: it's an organism you're going to keep alive, incident by incident, retrospective by retrospective. Every time something breaks and you learn something, that learning goes to a runbook or a living note, and the system is left a little better prepared for the next person who arrives —who's sometimes going to be you yourself, eight months later, at three in the morning, thanking your past self for having taken the trouble to write it down—.

Thank you for making it this far. Not bad for someone who started by wondering what makes a workflow "production-grade".

Resources

• Handle errors gracefully — n8n Docs — the technical foundation the kit's runbooks rest on; their failures are what each runbook turns into action.
• Schedule Trigger — n8n Docs — the trigger of the weekly on-call announcer, the only workflow in the kit.
• Postgres node — n8n Docs — the node of the runbooks' quick-check queries and of the impact measurement in retrospectives.
• Executions — n8n Docs — the executions list from which the drill is provoked and verified, and from which the real times of the timelines come.
• n8n-git-and-environments-guide · n8n-self-hosting-and-operations-guide · n8n-workflow-contracts-and-idempotency-guide — the three sister guides that cover the boundaries this one left out on purpose: versioning and environments, instance setup and scaling, and system correctness design.