Module 6: Retries, Alerts, and Recovery

8. Capstone: a reliable multi-workflow system end to end

Description

By the end of this lesson you will have built, running and defensible, a reliable end-to-end multi-workflow automation system: Cumbre's system that receives orders through a webhook that sometimes fires twice, validates every input against a contract, deduplicates with a Postgres ledger, executes idempotent effects, coordinates four sub-workflows with the outbox pattern, retries without duplicating, compensates what's left halfway done, alerts only where it matters, and routes real failures to a dead-letter queue. And — the crowning proof — you'll be able to show a replay that demonstrates a duplicate trigger doesn't create a second effect. This is the whole guide's portfolio deliverable: "flow assembler vs. automation system owner" turned into something you can open in an interview and defend decision by decision.

This matters because a capstone isn't one more exercise: it's the evidence. Anyone can say "I know idempotency and error handling." Very few can open a real system, show the dependency graph, point at the chosen idempotency key and explain why, and run a live replay that proves the system doesn't duplicate. That difference — between asserting and demonstrating — is exactly what separates someone hired as a system owner from someone hired as a flow assembler. The capstone is where you build that evidence, with your name on it, on a case you own.

Connection to the module and the guide: this lesson introduces no new concept. It integrates the six modules. Module 2's idempotency, Module 3's contracts, Module 4's ledger, Module 5's outbox coordination, and this Module 6's five pieces — retries, compensations, alerts, an error workflow with a dead-letter queue, and replay — all come together here in one system. If anything that follows isn't familiar, that's the module worth revisiting for a moment. And it honors lesson 7's boundary: the capstone demonstrates correctness, not operations; using n8n's credentials and a single instance is the right choice for this scope.

What you're going to deliver

A system with this shape, working end to end, plus three artifacts that are worth as much as the system:

                 webhook (sometimes fires twice)
                          │
                          ▼
                   ┌─────────────┐
                   │ order-triage│  1. dedup against run_ledger
                   │             │  2. validates against the contract
                   └──────┬──────┘  3. writes intents to the outbox
                          │
              ┌───────────┼───────────┐
              ▼           ▼           ▼
       ┌────────────┐ ┌──────────┐ (outbox consumers)
       │check-credit│ │inventory-│
       │  (effect)  │ │sync(eff) │
       └─────┬──────┘ └────┬─────┘
             │ fails midway │ fails to reserve
             ▼             ▼
          ┌──────────────────┐        ┌────────────────────┐
          │   issue-refund   │        │ cumbre-error-handler│
          │  (compensation)  │        │ (Error Trigger)     │
          └──────────────────┘        │  ├─ alert (critical)│
                                       │  └─ dead_letter     │
                                       └────────────────────┘

The four business workflows (order-triage, check-credit, inventory-sync, issue-refund), plus the central error workflow (cumbre-error-handler), plus the Postgres tables (run_ledger, outbox, dead_letter). And the three defense artifacts:

The dependency graph, with the idempotency keys noted on every effect.
The design justification: why each decision, and what would break with the alternative.
The proof replay: the recorded evidence that a duplicate trigger doesn't create a second effect.

On timing: if you've been building each module's project along the way, most of the pieces already exist and this capstone is mostly about integrating them and defending them — about two or three hours. If you're starting from scratch, count on a full day. The defense part — the three artifacts — isn't optional or "documentation at the end": it's half the deliverable's value.

Phase 1 — The skeleton: intake, dedup, and contract

Where you stand: with a Community instance, the Starter Kit's Postgres, and Module 4's tables created.

Reliability's heart starts at the door: order-triage is what keeps the unreliable webhook — which sometimes fires twice — from dirtying everything that comes after.

Step 1.1 — The webhook and the order's idempotency key. The Webhook receives the order. The first thing is computing the order's idempotency key, with a pure-logic Code node — remember: the key gets derived from stable data, never from the time:

// ============================================================
// Node: Code — "Build order key" (in order-triage)
// Mode: Run Once for Each Item
//
// INPUT:   the raw order from the webhook
// OUTPUT:  the order with its stable idempotency key
// WHY:     this key governs the dedup and survives a double trigger
// ============================================================

const order = $json;

// Stable: same order -> same key, even if the webhook fires twice.
// NEVER put Date.now() or a random here (see lesson 6).
const orderKey = `order:${order.order_id}`;

return { json: { ...order, idempotency_key: orderKey } };

Step 1.2 — Atomic dedup against the ledger. Here's the system's most important lesson, the one lesson 6 taught you to debug. Dedup is not "check the ledger, and if it isn't there, carry on" — that's the check-then-act trap, which produces the duplicate when two triggers run at once. Dedup is an atomic conditional write: you attempt to insert the key into the ledger, and the database itself, with a uniqueness constraint, guarantees only one of two simultaneous executions succeeds. The other collides and stops. With the Postgres node:

-- Node: Postgres — "Claim order in ledger"
-- Only ONE execution manages to insert; the second affects no rows.
INSERT INTO run_ledger (order_id, idempotency_key, status, created_at)
VALUES ($1, $2, 'claimed', now())
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING order_id;

What to expect. If the webhook fired once, the INSERT returns the order_id and the flow continues. If it fired twice, nearly simultaneously, only one of the two executions gets a row back; the other gets zero rows — because of ON CONFLICT DO NOTHING — and with an If node checking whether a row came back, that second execution stops without doing anything. The duplicate trigger dies right here, at the door, atomically. This is the piece phase 6's replay is going to prove.

Step 1.3 — Validate the contract. Before dispatching the work, order-triage validates that the order meets Module 3's input contract: that order_id, amount, and line_items are present and correctly typed. If it doesn't meet it, the order doesn't proceed: it gets routed to the failure path (which phase 5 catches). A malformed order shouldn't reach check-credit.

Step 1.4 — Write the intents to the outbox. Instead of calling check-credit and inventory-sync directly, order-triage writes intents to the outbox table (Module 5): "for this order, credit needs checking and inventory needs reserving." Deciding what to do gets separated from executing it, so the fan-out is reliable even if something fails at the moment.

Phase 2 — The idempotent effects

Where you stand: with the order deduplicated, validated, and its intents in the outbox.

The outbox consumers execute the effects. Each is idempotent, because each can get retried (phase 4) or reprocessed from the queue (phase 5).

Step 2.1 — check-credit. Reads its intent from the outbox, and places the credit hold with an HTTP Request that carries the idempotency key as a header (Module 2 + lesson 2). The credit API recognizes repeated keys: two attempts with the same key place a single hold. On success, it records in the ledger that this order has an active hold — data phase 3's compensation needs.

Step 2.2 — inventory-sync. Reads its intent, and reserves the stock with an idempotent upsert by order_id (Module 2): reserving the same order twice leaves inventory in the same state as reserving it once. Never a blind decrement.

What to expect. With both effects idempotent, the whole fan-out is safe to repeat. It doesn't matter how many times an intent gets processed — from a retry, from a reprocess — the result is the same as processing it once. This is the property that lets the rest of the system (retries, queue) exist without fear.

Phase 3 — The compensation

Where you stand: with check-credit having placed the hold, and inventory-sync that just failed because there's no stock.

It's lesson 3's scenario: an effect (the hold) got left orphaned because a later step failed, and it can't be "un-done." It gets compensated.

Step 3.1 — Detect and decide to compensate. When inventory-sync can't reserve, it writes a compensation intent to the outbox: release_credit_hold for this order_id, with the credit_hold_id stored in the ledger. Deciding to compensate and executing the compensation are separate steps (outbox pattern), so the compensation doesn't depend on everything working during the chaotic instant of the failure.

Step 3.2 — Execute the compensation. issue-refund reads the release_credit_hold intents and releases the hold with an HTTP Request that carries its own idempotency key (derived from the credit_hold_id), with Retry On Fail turned on — safe, because the release is idempotent. On success, it marks hold_status = 'released' in the ledger and the intent as done.

What to expect. The order that couldn't be fulfilled ends up with its hold automatically released, with no credit left blocked. And since the compensation is idempotent, releasing twice counts as once. The system cleans up after itself.

Phase 4 — Safe retries

Where you stand: with the effects and the compensation working on the happy path.

Now you harden it against transient failures. On every HTTP Request node that has an effect — check-credit, issue-refund, and inventory-sync's reservations — you turn on Retry On Fail (lesson 2), with Max Tries at 2 or 3 and a Wait Between Tries of at least 1000 ms.

The rule governing this phase: every retry you turn on is safe because the effect it retries is idempotent (phase 2). You don't turn on Retry On Fail on anything not protected by an idempotency key or an upsert. Nodes that receive business responses — "credit rejected," "no stock" — carry no retry: those aren't transient failures, they're results an If routes.

What to expect. An isolated credit API timeout now heals itself: n8n waits a second, retries, the API responds, and the order carries on without anyone finding out. Transient failures stop being incidents.

Phase 5 — Alerts and the dead-letter queue

Where you stand: with retries absorbing the transient stuff; what's missing is catching the terminal stuff.

Step 5.1 — The central error workflow. You create cumbre-error-handler with an Error Trigger as its first node (lesson 5), and you assign it in Options > Settings > Error workflow of the four business workflows. One single funnel for every failure.

Step 5.2 — Classify and route. A Code node reads the Error Trigger's payload (workflow.name, execution.lastNodeExecuted, execution.error.message, execution.id) and decides the severity per lesson 4's policy: issue-refund is always critical; the rest, warning. A Switch routes: critical ones fire an alert HTTP Request to finance and get saved; all of them get saved to dead_letter.

Step 5.3 — Lose nothing. Every terminal failure gets inserted into dead_letter with the order's full payload, status = 'pending', to be reprocessed later with its original idempotency key.

What to expect. On a normal day, with retries and compensations working, this handler produces zero or one alert. When issue-refund exhausts its retries, finance gets a useful alert — with the order_id and a link to the execution — and the order stays stored, not lost. Signal, not noise.

Phase 6 — The replay that proves it

Where you stand: with the complete system. Now you produce the crowning evidence.

This phase is what turns your capstone from "trust me, it doesn't duplicate" into "see for yourself." You're going to demonstrate that a duplicate trigger doesn't create a second effect.

Step 6.1 — Produce a duplicate trigger. With the real effect disconnected or simulated (lesson 6: never against the real API), fire the webhook twice with the same order ORD-3180, as simultaneously as you can.

Step 6.2 — Observe the two executions. In the executions list you'll see two order-triage executions. Load them with Debug in editor and trace, in each one, step 1.2's INSERT ... ON CONFLICT.

What to expect, and this is the proof. One of the two executions got a row back from the INSERT and carried on until placing the hold. The other got zero rows — it collided with the uniqueness constraint — and stopped at the following If, never reaching any effect. Result: one hold, one order processed, despite two triggers. The graph, the idempotency key, and these two executions side by side are your recorded proof. Save it: it's what you open in the interview.

Step 6.3 — The counterexample (optional but powerful). For the proof to carry weight, also show what would happen without the protection. Temporarily replace the atomic dedup with a "check then act" (Module 2's trap), reproduce the double trigger, and observe that now both executions see an empty ledger and place a hold: two holds. Restore the atomic dedup. That contrast — duplicated with the trap, not-duplicated with the atomic write — is the most compelling demonstration you can give that you understand why it works, not just that it works.

Evaluation criteria

Review these before considering the capstone done. They aren't optional: they're what makes the deliverable defensible.

The design decisions, argued

The last artifact, and the one that carries the most weight in an interview, isn't code: it's your ability to justify every decision. Here are the five you'll get asked about most, with the expected level of answer. Write them out for your own system.

1. Why is dedup an atomic write and not a prior check? Because the webhook fires twice nearly simultaneously, and a prior check ("does it exist? no, so I carry on") leaves a window where both executions see an empty ledger before either writes, and both carry on. The conditional write with a uniqueness constraint has no such window: the database guarantees only one insert wins, no matter how close together they run. It's the difference between trusting timing and making timing not matter.

2. Why does the idempotency key get derived from order_id and not from the moment? Because a key has to be stable for a retry, a double trigger, or a late reprocess to recognize it as the same one. A key with Date.now() changes on every execution, and the protection collapses — it's the bug you traced in lesson 6. The order_id identifies the order regardless of when it gets processed.

3. Why compensate instead of avoiding the effect? Because there's no atomic transaction spanning the credit API and the inventory system — they're different systems. You can't guarantee "both or neither." The best you can do is: if the second one fails, undo the first. That's a saga, and compensation is its mechanism.

4. Why does issue-refund always alert and a credit timeout doesn't? Because a stuck refund is real money in limbo that no automatic mechanism is going to release — terminal — while a timeout that heals on retry is transient. Alerting equally on both would produce alert fatigue and bury the one that matters. The policy is per-workflow and per-severity.

5. Is it ready for production? It's correct — idempotent, with contracts, with recovery, demonstrable with the replay — and it runs free on Community. What would be missing to operate it with a team — external secrets, staging, Git, backups — is operations, a separate project with its own decisions, treated in n8n-production-maintenance-guide. The design is ready; putting it into production is the next step. (This is lesson 7's answer, and it's the one that closes the interview.)

Common mistakes

Presenting the system without being able to demonstrate the no-duplicate property (conceptual). What happens: everything gets built correctly, but in the interview, faced with "how do I know it doesn't duplicate?", you can only assert that it doesn't, without showing it. The answer loses all its force. Why it happens: building feels like the work, and the demonstration seems like an extra. It's the reverse: the demonstration is the deliverable. How to detect it: if you don't have two recorded executions of a double trigger showing a single hold, you're missing the proof. How to fix it: do phase 6 and save the evidence. A system you can't demonstrate correct is, in an interview, indistinguishable from one that isn't.

Confusing "more pieces" with "better" (conceptual). What happens: retries get added to every node, alerts on everything, compensations for effects that were already idempotent — "for robustness" — and the system turns into a tangle that produces alert fatigue and useless retries. Why it happens: every piece of the module feels good, and more good things seems better. How to detect it: if you have retries on reads, alerts on transient failures, or compensations for things that create nothing, you have excess machinery. How to fix it: every piece answers a concrete failure mode. Retry what fails transiently and is idempotent; compensate what creates something irreversible; alert on the terminal. A reliable system is precise, not loaded up. Being able to explain why you didn't add a piece is as valuable as explaining why you added another.

Treating the capstone as the end of the road (conceptual). What happens: the capstone gets finished and archived, like someone closing a book. But a capstone is a starting point: it's the system you take to production, the one you show in interviews, the one you adapt to your next job. Why it happens: the "final project" format suggests closure. How to detect it: if your capstone lives in a folder you're never going to reopen, you're wasting it. How to fix it: treat it as a living portfolio. Update it as you learn something new, adapt it as your case changes, and — when you're ready — take it to real production with the operations guide. The capstone doesn't close your learning; it inaugurates it as practice.

Exercises

Exercise 1 — Defend a decision. Pick one of this lesson's five argued design decisions and rewrite it in your own words, in your own context, as if answering an interviewer who just asked "and why did you do it that way?" Include what would break with the alternative.

See solution

There's no single answer, because the point is that it sounds like yours. A strong answer has three parts: (1) the decision, (2) the why in terms of the failure mode it prevents, and (3) what would break with the alternative — this last one is what most people forget and what impresses the most.

Example, for the atomic dedup decision: "I deduplicate with an INSERT ... ON CONFLICT, not by checking the ledger first. I did it that way because the webhook sometimes fires twice almost at the same time, and if I checked first, both executions would see an empty ledger before either wrote, and both would process the order: two holds. With the atomic write, the database guarantees only one insert wins, no matter how close together they run. I proved it by reproducing a double trigger: one execution carries on, the other collides and stops. With the alternative — checking first — I was able to reproduce the duplicate at will."

Notice the answer names the failure mode (simultaneous double trigger), explains the mechanism (the uniqueness constraint has no window), and closes with evidence (I proved it, and the counterexample confirms it). That structure turns a technical decision into a memorable interview answer.

Why this works: in a system-owner interview, you don't get evaluated on whether you know the syntax — that's assumed — you get evaluated on whether you understand why a decision is correct and what failure it prevents. Practicing the defense of your own decisions is practicing exactly what gets you hired.

Exercise 2 — Find the weak link. A colleague shows you their Cumbre system. It deduplicates with an INSERT ... ON CONFLICT, validates contracts, coordinates with the outbox, has retries and alerts. But their idempotency key for the refund is `refund:${order.order_id}:${Date.now()}`. Is their system truly protected against double refunds? Explain.

See solution

No, it has a serious crack, and it's subtle because everything else looks fine. order-triage's atomic dedup protects against the webhook's double trigger — two nearly simultaneous events. But the refund key carries Date.now(), which changes on every execution. That means if issue-refund gets retried (phase 4) or reprocessed from the queue (phase 5) at a different moment, it generates a new key, and the payments API sees it as a different refund. Lesson 2's safe retry and lesson 5's reprocess — the two things their system has — become, with that key, duplicate-generating machines.

It's exactly lesson 6's bug. The system is protected against one failure mode (double trigger) but wide open to two others (retry and reprocess), and the weak link is a single line that "looks" correct because it includes the order_id. The fix: remove Date.now(), leaving `refund:${order.order_id}`.

Why this works: this exercise trains the eye for the most common and most hidden failure of these systems. Correctness isn't "having all the pieces"; it's the pieces being coherent with each other. An unstable key silently breaks every protection that depends on repeating safely, even though each piece looks fine on its own.

Exercise 3 — Extend the system. Cumbre adds a fifth effect: when an order is fully fulfilled, the invoice needs emailing (send-invoice). Without building it, answer: where in the effect order does it go and why? Does it need an idempotency key, and what's it derived from? What alert severity does it get if it fails? And does it have a compensation?

See solution

Order: at the end, after credit, inventory, and — if applicable — the charge have succeeded. The email is an irreversible effect (lesson 3): once sent, it can't be pulled back, so it should only go out once the order is safe and there's no risk of having to walk it back.

Idempotency key: yes, it needs one, derived from stable data like `invoice:${order.order_id}` — never with the time. Without it, a retry of the send would email two identical invoices for the same order, a duplicate that does matter on a fiscal document.

Severity if it fails: medium, generally. An invoice that didn't go out today can go out tomorrow without drama — there's no money moving like with a stuck refund. It goes to the dead-letter queue as a warning, so finance can retry the send or fix the email data. The policy is per-workflow (lesson 4), and this isn't issue-refund.

Compensation: it doesn't have a clean one, because the email is irreversible. The possible "compensation" is another communication that corrects it — "there was an error with your invoice" — which doesn't undo the first one. That's why order matters so much: by going last, there's almost never anything to correct, because it only gets sent once the order is already firm.

Why this works: extending the system with a new effect is applying the whole module at once in a single decision. Order (lesson 3), idempotency (Module 2 + lesson 2), severity (lesson 4), compensation (lesson 3): every new effect you add to any system passes through these same four questions. Being able to answer them without hesitation is the sign the module got internalized.

Summary and guide close

You made it to the end, and it's worth looking back at the whole path before closing.

You started this guide knowing how to build workflows that work once. You finish owning an automation system that survives what the real world throws at it. In Module 1 you made the mindset shift — from flow assembler to system owner — and learned to see the difference between a read and an effect, between "at least once" and "exactly once." In Module 2 you made your effects idempotent, so repeating doesn't duplicate. In Module 3 you put contracts between workflows, so they talk to each other without breaking. In Module 4 you designed the ledger, so the system's truth lives in one place and deduplication survives across executions. In Module 5 you coordinated several workflows with the outbox pattern, separating deciding from executing. And in this Module 6 you made it resilient: retries that don't duplicate, compensations that undo what got left halfway, alerts that only sound where it matters, an error workflow with a dead-letter queue that loses nothing, a replay that turns an intermittent bug into a reproducible one, and a clear boundary with what comes next.

And in this capstone you put it all together into a Cumbre system you can open, run, and defend: with its graph, its idempotency keys, and a replay that proves — not asserts, proves — that a duplicate trigger doesn't create a second effect. That's the deliverable. That's what separates, in an interview and in a real job, someone who assembles flows from someone who owns a system.

There's a final honesty this guide owes itself. What you built is correct, and running a correct system for free on Community is a real achievement. What comes next — operating it in production with a team: external secrets, environments, Git, backups, scaling, monitoring — is another discipline, treated seriously in n8n-production-maintenance-guide. It isn't that your system is incomplete; it's that the correctness design, which is what this guide promised to teach you, is finished. Operations is the next chapter, and now you know exactly where it starts.

Congratulations on making it this far. Six modules of discipline — idempotency, contracts, ledgers, coordination, recovery — aren't a short path, and the system you have at the end shows it. When someone asks you "what happens if the API goes down mid-execution?", you no longer have to improvise: you have a system that answers that question, and you know why it answers that way. That's the result. Not bad at all for someone who started out assembling flows that worked once.

Resources

Error handling — n8n Docs — the reference that gathers the retries, the Error Trigger, and the error workflows you integrated in the capstone.
Error Trigger — n8n Docs — the central cumbre-error-handler's node, with the payload you classify in phase 5.
Debug and re-run past executions — n8n Docs — phase 6's replay tool, which you use to produce the no-duplicate proof.
Postgres node — n8n Docs — the node behind the ledger, the outbox, and the dead-letter queue, with the atomic dedup's INSERT ... ON CONFLICT.
Self-hosted AI Starter Kit — n8n — the free kit with Postgres the capstone's whole system runs on at zero cost.
n8n production maintenance — n8n Docs — the starting point toward production operations, the next step once your system is correct.

← Previous7. The line with production and teams