4. Where failures should alert

Description

By the end of this lesson you will be able to decide, for every failure in your system, whether it deserves an alert that interrupts a person or just a log that gets reviewed later. You will tell signal apart from noise: the transient failure that heals itself versus the terminal failure that needs a human. You will understand alert fatigue — why alerting on everything is the fastest way for no one to look at any alert — and you will learn to define what a "real" failure is for each of Cumbre's workflows, who the owner that responds is, and at what urgency level. This lesson is about the design decision, not the tool: we're not going to wire up a Slack channel or build a dashboard — that's operations — we're going to decide what deserves notifying and who it goes to, which no tool decides for you.

This matters because a poorly calibrated alert is worse than none. If your system notifies every time an API takes an extra second, in two weeks the team learns to ignore those notifications, and the day the alert that actually mattered arrives — a stuck refund, money in limbo — it's buried under forty irrelevant notices no one read. Reliability isn't only that the system recovers on its own; it's that, when it can't recover on its own, the right person finds out in time and with the right urgency. That doesn't come from a tool. It comes from a decision, and this lesson is that decision.

Connection to the module: lessons 2 and 3 were the automatic reactions to failure — retrying and compensating. This is the human reaction: when a failure goes beyond what the machine can resolve and someone needs to be woken up. It's the hinge between the automatic and the manual. It leans on everything before it: a failure a retry heals (lesson 2) doesn't alert; a compensation that works (lesson 3) doesn't either; but a compensation that fails — the case lesson 3 closed with — does, because no one else is going to resolve it. Lesson 5 builds the machinery that executes this decision: the error workflow that routes the failure and decides, based on what you define here, whether to alert or just log.

Signal vs. noise: not every failure is an emergency

Let's start with the distinction that governs the whole lesson, with an image.

Think of the sensors in a house. There's a smoke detector: when it goes off, you drop what you're doing and go check, because it could be a fire. And there's a blinking light when an appliance finished its cycle: you notice it as you pass by, and if you don't notice it now, nothing happens, you'll see it later. Both are signals that "something happened." But one interrupts your life and the other waits. No one calls the fire department because toast burned, and no one lets the house burn down because "I got used to that sensor going off."

The most common mistake in alert handling is treating every failure like a smoke detector. Every timeout, every slow response, every transient stumble fires an urgent notification. And since the vast majority of those failures heal themselves — a retry, a compensation — most of the "emergencies" turn out to be burnt toast. The team learns, quite rightly, that the detector lies. And a detector that lies is worse than no detector, because it gives a false sense of coverage while training everyone to ignore it.

The concrete distinction you're going to use is this:

• Transient failure: a passing stumble that resolves itself or with a retry. A network timeout, a temporary 503, a service saturated for an instant. It needs no one. It's the blinking light: it gets logged, and if a pattern of many transient failures showed up, that would be worth looking at — but a single isolated transient failure, no.
• Terminal failure: a failure no automatic mechanism is going to resolve, and that leaves the system in a state requiring a human decision. A compensation that exhausted its retries, a refund that couldn't be issued, an order charged but never fulfilled. It's the smoke detector: someone needs to find out now.

Almost all the art of alerting is correctly classifying every failure into one of these two categories. And most failures in a well-designed system — with lesson 2's retries and lesson 3's compensations — fall on the transient side. Real alerts should be few, precisely because the module's other pieces already resolved the rest.

Alert fatigue, and why it kills systems

It's worth pausing on the phenomenon, because it has a name and it's the number one reason alert systems fail.

Alert fatigue is what happens to a team when it receives so many alerts it stops paying attention to them. It isn't laziness or carelessness: it's a rational response. If ninety-eight out of a hundred alerts don't require you to do anything, your brain correctly learns that the probability the next one matters is low, and your guard drops. The problem is that the two that did matter get lost with the rest.

In a Cumbre system this looks like: someone wires "notify me of any failed execution" to a channel. At first every one gets reviewed. But it turns out the credit API times out three or four times a day — and recovers on its own every time, thanks to the retries — so three or four daily alerts arrive about something that already resolved before anyone read them. Add the transient inventory failures, the refund retries that worked on the second try, and before long the channel has dozens of messages a day, all saying "something failed" about things that aren't failed anymore. The person stops opening it. And the day a real refund gets stuck, the alert lands in the same dead channel.

The practical lesson is counterintuitive but firm: fewer alerts is safer than more alerts. An alert should be rare and reliable enough that, when it arrives, the person knows without a doubt they have to act. That's the standard. If your alert channel receives more than a handful of messages a day, and most lead to no action, you don't have an alert system: you have noise that trained your team not to look.

That's why this lesson comes before connecting any tool. The temptation of "wire everything up and we'll see" is exactly what produces fatigue. First you decide what deserves an alert. Then, and only then, you connect it.

The three design questions for each workflow

For every workflow in your system, three questions give you its alert policy. Let's answer them for Cumbre.

Question 1: What counts as a "real" failure for this workflow? Not every technical error is a real failure. A check-credit that responds "credit rejected" didn't fail: it did its job, the answer is a no. A check-credit that times out and recovers on retry isn't a real failure either: it's a stumble that healed. A real failure is one that leaves something unresolved that matters. Defining it forces you to separate the technical error from the business problem.

Question 2: Who's the owner that responds? An alert with no clear recipient is an alert no one attends to, because "everyone's" means "no one's." Every workflow needs an owner: the person or role who, when that workflow really fails, is responsible for acting. At Cumbre — twelve people — it's probably the operations lead for orders, and whoever handles finance for refunds. The point isn't the org chart; it's that someone with a name gets the alert and knows it's theirs.

Question 3: How urgent? Not every real failure has the same rush. A stuck refund — money in limbo — is urgent: handle it today. An order left in the dead-letter queue because of a malformed field can wait until someone reviews the queue during normal hours. Urgency decides the channel and the moment: what's urgent interrupts; what's important-but-not-urgent piles up for review.

Worked example: Cumbre's alert policy

Let's apply the three questions to the four workflows, and produce a table that's, literally, the system's alert design.

order-triage. What's a real failure? That an order can't even be evaluated: an event so malformed it doesn't pass contract validation (Module 3), or the ledger connection being down so it can't deduplicate. A webhook that fires twice is not a failure — dedup absorbs it silently. Who responds? Operations. Urgency? Medium: an order that couldn't be evaluated needs reviewing, but it wakes no one up in the middle of the night; it goes to the dead-letter queue and gets reviewed during the day.

check-credit. What's a real failure? Not the timeout that recovers on retry — that's transient, it doesn't even register as an alert. Not "credit rejected" — that's a legitimate business response that follows its own path. The real failure is the credit API being down sustainedly, exhausting all the retries for several orders in a row: that's no longer a stumble, it's a downed service. Who responds? Operations, maybe with technical support. Urgency? High if it's many orders: the system can't process anything.

inventory-sync. What's a real failure? That the reservation fails from a persistent technical error — not from lack of stock, which is a normal business response that triggers lesson 3's compensation. Who responds? Operations. Urgency? Medium.

issue-refund. What's a real failure? That a refund or a hold release can't be executed after exhausting the retries. This is the system's smoke detector: it's money trapped or credit blocked that no automatic mechanism is going to release. Who responds? Finance, by name. Urgency? High, always. A stuck refund is the kind of failure that interrupts.

The resulting table:

Workflow	Real failure	Transient (log only)	Owner	Urgency if real
order-triage	Order not evaluable (contract/ledger)	Double webhook; isolated item validation	Operations	Medium (review same day)
check-credit	Credit API sustainedly down	Timeout that recovers; credit rejected	Operations + technical	High if several orders
inventory-sync	Persistent technical error in the reservation	Lack of stock (triggers compensation)	Operations	Medium
issue-refund	Refund/release that doesn't execute	Retry that works on the second try	Finance	High, always

What to expect from this policy. On a normal day, with retries and compensations doing their job, this policy produces zero or one alert. Almost everything falls on the transient side and gets logged without bothering anyone. The alerts that arrive are rare and reliable: when issue-refund's goes off, finance knows without a doubt there's real money to attend to. Compare that to "notify me of everything," which on the same day would have produced ten or fifteen messages, most of them burnt toast. The difference isn't the tool — both use the same channel — it's the decision you made before connecting it.

Severity levels: not everything that alerts interrupts

Once you've separated transient from real, one more nuance is worth adding within real failures: not all of them deserve the same channel. A three-level scheme covers almost any system:

• Critical (interrupts): money or data at risk, or the entire system stopped. Goes to a channel people check even on a weekend. At Cumbre: a stuck refund, the credit API down for every order. These should be extremely rare.
• Warning (reviewed soon, doesn't interrupt): something was left unresolved but isn't bleeding out. Goes to a channel reviewed during business hours. At Cumbre: an order in the dead-letter queue because of a malformed field.
• Informational (log only): transient failures and normal events. Goes to no human channel; it gets logged in case you ever want to analyze patterns. At Cumbre: every timeout that recovered, every duplicate webhook dedup absorbed.

The reason for having levels is that the channel communicates urgency without anyone having to read the detail. When something lands in the critical channel, the person already knows — from the channel — to drop what they're doing. When it lands in the warning one, they know they can finish what they're doing and review it later. The level is, itself, information.

A detail that matters: the level isn't a fixed property of the error, it's your decision about that error in your business. A stuck refund is critical for Cumbre because it involves customer money. In another business, the same kind of technical failure might be a warning. There's no universal table; there's your judgment about what hurts your system. That's exactly what makes this a design decision and not a configuration.

From isolated failure to pattern: when many transients become signal

There's a nuance that prevents a misunderstanding. We said an isolated transient failure doesn't alert. But many transient failures in a row are a signal, and it's worth understanding why the two don't contradict each other.

A credit API timeout is burnt toast: it happens, it recovers, it doesn't matter. But forty credit API timeouts in five minutes is no longer burnt toast: it's the smell of something actually catching fire. Each one, individually, recovered with its retry — so none of them alone is a terminal failure. But the rate at which they happen tells you the API is degrading, and that deserves someone's attention before it starts exhausting retries and falling onto the terminal side.

This is the difference between alerting on an event and alerting on a rate. The policy you've designed so far alerts on specific terminal events: this refund got stuck. A finer layer alerts on rates: "more than N transient failures of the same type in M minutes." The first is easy to build inside n8n with what you'll see in lesson 5. The second — counting failures over a time window and firing when they cross a threshold — is more natural in an observability tool that aggregates metrics, and that's why it lives on the operations side, in the production guide.

What matters to take away: the same failure can be noise when isolated and signal when it's a pattern. Your informational log — where you send the transients — isn't a trash can; it's the material that later lets you detect patterns. Logging the transients instead of discarding them is what makes it possible, later on, to notice "the credit API fails a lot more on Mondays" and do something about it. Don't alert on each one; but don't throw them away either.

What a useful alert looks like

A design decision that gets forgotten: what the alert says when it arrives. An alert that only says "something failed at Cumbre" forces whoever receives it to go investigate from scratch, and that's friction that delays the response right when it matters.

A useful alert gives the owner, at a glance, what they need to decide what to do:

• What failed and in which workflow. Not "system error," but "issue-refund couldn't issue the refund for order ORD-3180."
• The identifier for tracing it. The order_id, and if possible a link to the specific execution that failed, so the owner goes straight to the detail without searching.
• What the system already tried. "All three retries were exhausted" tells the owner this isn't going to fix itself, and that their intervention is needed.
• Where it got stored. "The item is in the dead-letter queue" tells them nothing got lost and they have time to decide calmly.

Notice that all of that information comes from pieces you already have: the order_id comes from the order, the Error Trigger (lesson 5) knows which workflow failed, the retry state and the dead-letter queue are lessons 2 and 5. A useful alert requires no new data; it requires deciding to include it in the message instead of sending a bare "something failed." It's the difference between an alert that speeds up the response and one that only announces there's work ahead.

Where this lesson ends: the decision, not the dashboard

It's worth marking the explicit limit, because it's easy to cross without noticing — and lesson 7 is going to formalize it for the whole module.

What this lesson designs is the policy: which failure alerts, who receives it, at what urgency. That's correctness: it defines how the system should behave in the face of a failure.

What this lesson does not do is set up observability infrastructure: a dashboard with charts of error volume over time, integration with an external monitoring tool, metrics aggregation, panels the team looks at all day. That's operations, and it lives in n8n-production-maintenance-guide. The distinction is the usual one: here you decide a stuck refund should alert finance with high urgency; there the board gets built where finance sees the history of how many refunds got stuck this month.

Why I keep insisting: it's tempting, when talking about alerts, to jump straight to "and I connect it to this tool and build this chart." But if you connect the tool before deciding the policy, you end up with the alert fatigue from the start of the lesson. The correct order is: first the decision (this lesson), then the machinery that executes it inside n8n (lesson 5), and only much later — in the other guide — the production observability infrastructure. Don't invert the order.

A note for small teams, which is Cumbre's case with its twelve people. When "the owner who responds" and "whoever built the workflow" are the same person — or when the whole team fits in one channel — it's easy to think the alert policy is overkill: "we'll find out anyway." It's the opposite. The smaller the team, the more alert fatigue hurts, because there's no on-call rotation to spread the load: the same person receives everything, and if they receive noise all day, they burn out or disconnect. A single owner needs, more than anyone, their alert channel to be rare and reliable, so they can safely ignore it 99% of the time and trust that when it goes off, it's real. This lesson's policy isn't a luxury for big companies; it's especially valuable when you're few.

Common mistakes

Alerting on every failed execution (conceptual). What happens: "notify me of any execution that fails" gets wired to a channel, with the intention of not missing anything. Within two weeks, dozens of alerts arrive a day, almost all from transient failures already resolved, and the team stops looking at the channel. Why it happens: it's the easiest option to set up and the one that feels safest — "this way I cover everything." How to detect it: count how many alerts you receive a day and what fraction led to an action; if most required no one to do anything, you have fatigue in progress. How to fix it: apply the three design questions to each workflow before connecting anything, and alert only on real failures, sending the transient ones to a log no one looks at except to analyze patterns. Fewer alerts, more reliable.

Confusing a business result with a failure (conceptual). What happens: "credit rejected" or "out of stock" gets treated as system errors and gets alerted on. But a customer with no credit line isn't a failure: it's a legitimate response the business must handle differently — maybe notifying the customer, not waking up an engineer. Why it happens: technically, those cases sometimes get implemented as an "error" branch, and from there they leak into the alerts. How to detect it: review your alerts and ask, for each one, "is this something that broke, or is it the system working and giving a response we don't like?" How to fix it: route the business results with an If or Switch node toward their own flow — "order not fulfilled due to credit," "order on hold for stock" — and reserve the technical alerts for what really broke.

Alerting with no named owner (practical). What happens: the alert lands in a general "everyone's" channel, and when something important fails, everyone assumes someone else is watching it, and no one acts. Why it happens: it's more comfortable to send everything to a shared channel than to assign owners. How to detect it: for every critical alert in your system, ask yourself "if this goes off at 2 p.m., who exactly drops what they're doing?" If the answer is "someone on the team," there's no owner. How to fix it: assign each workflow a concrete owner for its real failures, and make the alert mention them or go directly to them. An alert with no owner is one that gets handled sometimes, which is almost as bad as not having one.

Exercises

Exercise 1 — Transient or terminal. Classify each of these six Cumbre failures as "transient (log only)" or "terminal (alert)," and why in one sentence:

(a) The credit API times out and responds fine on the second try. (b) An order's webhook fired three times in one second. (c) issue-refund exhausted its three retries and couldn't issue a refund. (d) check-credit responded "credit rejected" for a delinquent customer. (e) The ledger's Postgres connection has been down for ten minutes and no order can be deduplicated. (f) An order arrived with a blank amount field and failed contract validation.

See solution

(a) Transient. It healed itself with the retry. It gets logged informationally; no one gets alerted. If many timeouts like this piled up, the pattern would be something else — but an isolated one, no.

(b) Transient, and not even a failure. The ledger's dedup absorbs it: the second and third triggers do nothing. It doesn't even deserve an error log, at most an informational trace.

(c) Terminal. No automatic mechanism is going to resolve it, and there's money in limbo. Critical alert to finance, high urgency. It's the system's smoke detector.

(d) Neither transient nor terminal: it isn't a failure. It's a legitimate business response. It doesn't go to technical alerts at all; it follows its own flow — notifying the customer, not fulfilling the order.

(e) Terminal. The entire system is stopped: with no ledger there's no dedup and nothing moves forward. Critical alert, high urgency, probably to operations and whoever handles the infrastructure.

(f) Terminal but low urgency. The order couldn't be evaluated and we don't want to lose it, so it goes to the dead-letter queue and generates a warning reviewed during the day. It interrupts no one, but it doesn't get ignored.

Why this works: notice that out of six failures, only three deserve an alert, and only two are urgent. The other three are noise (a, b) or not even failures (d). That proportion — few real alerts among many events — is the mark of a well-calibrated system. If you classified all six as "alert," you'd have guaranteed fatigue.

Exercise 2 — Assign severity and owner. For the three failures you classified as terminal in the previous exercise (c, e, f), assign a severity level (critical / warning / informational) and an owner, and say what kind of channel each would go to.

See solution

(c) Refund that couldn't be issued: critical, owner finance, a channel checked even on weekends. There's a customer's money trapped; handle it today.

(e) Ledger down for ten minutes: critical, owner operations + technical, an interrupting channel. The whole system is stopped; every minute is orders not being processed.

(f) Order with a blank amount in the dead-letter queue: warning, owner operations, a channel reviewed during business hours. The order is stored and isn't lost; someone will review it and decide — maybe asking the customer for the data — without rushing.

The key difference between (c)/(e) and (f): the first two involve money or the entire system, and they don't wait; the third is safe in the queue and can wait for normal hours. The channel communicates that difference without anyone reading the detail.

Why this works: separating severity from "is it a failure" gives you a second filter. It's not enough to know something requires human action; you need to know how fast and from whom. A system that sends everything to the same channel, with the same urgency, blurs the critical into what could have waited.

Exercise 3 — Design the policy for a new workflow. Cumbre adds a fifth workflow, send-invoice, which generates and emails the invoice when an order gets fulfilled. Apply the three design questions: what's a real failure for this workflow? Who responds? At what urgency? Pay special attention to what happens with the email, remembering lesson 3.

See solution

What's a real failure? Three cases need separating. (1) Generating the invoice fails from missing data: real failure, the order got fulfilled but has no invoice. (2) Sending the email times out and recovers on retry: transient, log only. (3) Sending fails persistently — the email address is invalid, the mail server is down: real failure, the invoice got generated but never reached the customer.

Who responds? Probably finance or administration, because an invoice is a fiscal document and its absence has accounting consequences.

At what urgency? Medium, generally. An invoice that didn't go out today can go out tomorrow without drama — unlike a stuck refund, there's no money moving. It goes to the dead-letter queue as a warning, so finance can retry the send or fix the data.

The email detail, connecting to lesson 3: the email is an irreversible effect — once sent, it can't be pulled back. That's why send-invoice should go at the end of the effect order, once the order is already fulfilled and confirmed. If it went out earlier and the order later fell through, you'd have to walk back an invoice, which is messy and confusing. And if the same email got sent twice because of a non-idempotent retry, the customer would receive two identical invoices — a duplicate that does matter on a fiscal document — so the send needs its idempotency key like any other effect.

Why this works: this exercise ties the three lessons together. The alert policy (this lesson) decides the send failure is a medium-urgency warning for finance; the effect order (lesson 3) puts the email last because it's irreversible; and idempotency (Module 2, lesson 2) prevents the double invoice. Designing a new workflow means, every time, applying the whole module at once.

Summary and next step

In this lesson you saw that the human reaction to failure starts with a decision: is this a smoke detector or a blinking light? You told apart the transient failure — which a retry or a compensation heals on its own, and which goes to a log — from the terminal failure — which no automatic mechanism resolves and which needs a person. You understood alert fatigue and why fewer alerts is safer than more: a channel saturated with burnt toast trains the team to ignore the one alert that mattered. You applied the three design questions — what's a real failure, who's the owner, at what urgency — to Cumbre's four workflows, and produced a policy that on a normal day results in zero or one alert. You added severity levels so the channel communicates urgency on its own. And you marked the boundary: this lesson designs the alert policy — correctness — not the observability dashboard — operations, from the other guide.

Before moving on you should be able to: classify any failure in your system as transient or terminal; explain what alert fatigue is and why "notify me of everything" causes it; and write, for a workflow, what a real failure is, who responds, and at what urgency.

What you haven't seen yet is the machinery that executes this decision. You defined that a stuck refund should alert finance and a malformed order should get saved without being lost — but where does the code live that catches the failure, decides its level, and acts on it? Lesson 5 builds it: the Error Trigger node as a central error workflow that catches any system failure in one single place, and the dead-letter queue — a dead_letter table in the same ledger Postgres — where the item that failed after all its retries gets set aside with its full context, so nothing gets lost and everything can be reprocessed.

Resources

• Error handling — n8n Docs — the basis of how n8n detects and routes a failure, which lesson 5 turns into the machinery that executes this alert policy.
• Error Trigger — n8n Docs — the node that catches the failure and from which you'll decide, based on what's designed here, whether to alert or just log.
• Handle errors gracefully — n8n Docs — official error-handling design guide, with the framework for deciding what to do about each type of failure.