5. The incident timeline

Description

By the end of this lesson you'll be able to run an incident instead of suffering it. You'll know the five phases that order the response —detect, communicate, contain, resolve, verify—, you'll understand why containing comes before understanding the cause (first you stop the bleeding, then you figure out why it was coming out), you'll know why the person who fixes and the one who talks to the business shouldn't be the same one, and you'll learn to write the incident log while it's happening, not afterward.

This matters because an incident without running is an incident that drags on, that's communicated badly, and that gets resolved in a way that sometimes makes things worse. When something big breaks, the natural reaction of a technical person is to dive into understanding the cause —"why is this happening?"— and start investigating. It's exactly the wrong instinct, and it is for a concrete reason: while you investigate the cause, the bleeding keeps going. The order keeps getting lost, the duplicate email keeps being sent, the queue keeps growing. A well-run incident inverts that instinct: first stop the damage, and only then allow yourself the luxury of understanding it.

Connection to the module: this is the third piece of the kit —running the incident— and it comes after the runbook and on-call for a reason. A runbook covers a known symptom: you open it, follow the steps, resolve. But there are situations no runbook covers, because they're new, because they're big, or because several things fail at once. That's an incident in the full sense, and it isn't resolved by reading a card: it's run. Lesson 4 gave you who responds; this one gives you what that person does when what's in front of them exceeds the runbook. And it prepares lesson 6: the log you write during the incident is, literally, the raw material of the blameless retrospective that comes afterward. A boundary note: here we don't redesign the system's correctness so the incident doesn't happen again —that's design and it lives in n8n-workflow-contracts-and-idempotency-guide—; here we operate the incident that's already happening.

The emergency room

Think about how someone who arrives in critical condition at an emergency room is attended to. There's an order, and the order is sacred, because breaking it costs lives.

A person arrives with a wound that's bleeding heavily. What's the first thing the medical team does? It's not to ask how they got the wound. It's not to order tests to know the cause. It's not to discuss the best long-term treatment. The first thing is to stop the hemorrhage: pressure, tourniquet, whatever, now. Because a person who's bleeding out doesn't have time for you to figure out why.

Only when the bleeding is contained —when the patient is stable— does the team move to the next thing: understanding what happened, running the tests, deciding the underlying treatment. The order is non-negotiable: stabilize first, understand after. A doctor who started investigating the cause while the patient bled out would be a bad doctor, however brilliant their diagnosis, because the patient dies on them during the analysis.

And notice something else about that room. There isn't a single person doing everything. While someone stops the hemorrhage, another person talks to the family waiting outside: explains what's happening, keeps them informed, absorbs their distress. Those two tasks —saving the patient and talking to the family— aren't done by the same person, and not for lack of hands: it's that they're incompatible. Whoever has their hands inside the wound can't stop every two minutes to explain to the family, and if they did, the patient would pay for it. They're two different jobs happening in parallel.

A technical incident is an emergency room. The order that's lost is the blood. Containing is the tourniquet. And the business —Gerardo waiting for his orders, Andrea asking what's happening— is the family in the waiting room. Everything that follows in this lesson is how to transfer the order of the emergency room to your operation.

Worked example: the ERP incident, run and not run

Let's go back to Thursday's ERP incident —the one Daniela handled— but raise its severity so it's a real incident: this time the ERP doesn't just return 503, it starts accepting orders and losing them. It creates them halfway: it responds 200, but the order doesn't get registered on the warehouse side. It's an ugly failure because the system believes everything's fine. Let's see it run two ways.

Version A — no running (the natural instinct).

It's 02:51. The alert arrives. The on-call person —let's say this time it's you— dives into what feels natural: understanding why. You open the ERP, review logs, compare successful requests with failed ones, form a hypothesis, discard it, form another. You're absorbed, focused, doing good diagnostic work. Forty minutes pass.

Meanwhile, three bad things happen in parallel and you're not looking at them:

• order-sync kept running all that time. Every order that arrived in those forty minutes got its 200 from the ERP and was lost, because the failure makes them get lost. Forty minutes of evaporated orders that you believed, while diagnosing, were coming in fine.
• At 03:20, Andrea —who's an early riser— saw the alert in the channel and wrote "what's happening?". You didn't answer: you were focused. At 03:35 she wrote again, now anxious. The silence scared her more than the problem.
• When you finally understood the cause at 03:31, you had to start the containment you should have done first, and by then orders had already been lost that aren't going to be recovered, because they never ended up in dead_letter —the failure lost them before—.

You did an excellent diagnosis. And the incident was a disaster, because the diagnosis was the third thing to do, not the first.

Version B — run (the order of the emergency room).

It's 02:51. The alert arrives. Before asking yourself why, you execute the order:

• 02:52 — Detect and assess. You read the alert. You run the runbook's quick check. You discover the worst: dead_letter is not growing, and yet orders are arriving. That is, they're being lost. The golden rule broke: this isn't contained.

• 02:54 — Communicate that you're starting. You write ONE line in the thread: "02:54 — order-sync is losing orders (the ERP accepts them but doesn't save them). I'm containing. Update in 15 min. —you". You don't explain the cause (you don't know it). You just say: there's an incident, I'm handling it, when I'll speak again.

• 02:56 — Contain, without understanding yet. How do you stop the bleeding without knowing the cause? You unpublish order-sync. Yes, that means orders stop coming in. But storefront retries the webhooks for six hours (you knew this, it was in the inventory), so the orders aren't lost: they accumulate in storefront waiting. You traded "orders lost forever" for "orders on pause, recoverable". That's containing: you didn't fix anything, you stopped the damage.

• 03:10 — Now, resolve. With the bleeding contained and no clock on you, you investigate the cause calmly. You discover the ERP had a failed deployment from its vendor. There's nothing to fix on your side: you have to wait for the vendor to revert it. You write to their emergency contact (it was in the inventory).

• 03:15 — Communicate the status. "03:15 — Contained. order-sync is on pause, the orders accumulate in storefront and none are being lost. Cause: failed ERP deployment (its vendor). Waiting for them to revert it. Nothing to do on the warehouse side until 07:00." Andrea reads that and calms down: she knows what's happening, knows it's contained, knows when it matters.

• 04:40 — Verify and close. The vendor reverted at 04:20. You republish order-sync. storefront starts resending the accumulated webhooks. You verify with the query that the orders come in and really get registered this time. By 04:40 everything is normal, with zero orders lost.

What to expect. Look at the difference, which isn't in the technical knowledge —in both versions you diagnosed equally well— but in the order:

	Version A (not run)	Version B (run)
First action	Understand the cause	Contain the damage
Orders lost forever	40 min of orders	0
Andrea (the business)	Scared by the silence	Informed and calm
Time to contain	~40 min	~5 min
Diagnosis quality	Excellent	Excellent (but done unhurried)

The second row is the one that hurts: forty minutes of lost orders, which in version B were zero, and the only difference was what you did first. Containing before understanding isn't a style preference: it's what decides how many orders exist tomorrow.

The five phases, in order

Now the complete order, phase by phase. Think of it as the protocol you execute, not as a list you consult.

Phase 1 — Detect. Knowing there's an incident and how big. It almost always starts with the alert, but detection doesn't end there: it includes the runbook's quick check that answers the question that governs everything —is anything being lost? is the damage growing or contained?—. This phase is short but decides everything else, because it distinguishes between "a runbook is enough" (known symptom, contained) and "this is an incident" (new, big, or not contained). Don't confuse detecting with understanding: in this phase you know what is happening (orders are being lost), not why.

Phase 2 — Communicate (that you're starting). Before putting your hands in, one line: there's an incident, I'm handling it, when I'll report again. This initial communication costs fifteen seconds and avoids the worst pattern of incidents: the silence that scares the business more than the problem itself. It's not the complete communication —that comes later, with the status— it's just the "I'm on this". Notice it goes before containing: fifteen seconds of notifying don't delay the containment significantly, and in exchange they keep Andrea from writing three times while your hands are busy.

Phase 3 — Contain. Stop the damage, without needing to understand the cause yet. This is the phase intuition puts in the wrong place, and it's the heart of the lesson. Containing is trading a damage that grows for one that's stopped, even if the solution is crude: unpublishing a workflow, pausing a trigger, diverting the traffic to a queue. It's not elegant and it's not the final solution. It's the tourniquet. The question of this phase isn't "how do I fix it?" but "how do I make it stop getting worse while I fix it?".

Phase 4 — Resolve. Now, understand the cause and fix it at the root, with the bleeding already contained and no clock on you. This is the phase where your technical instinct —the diagnosis— finally has its place, and it does it better because it does it without the pressure of losing something every second. Sometimes resolving is waiting (the vendor reverts their deployment); sometimes it's a fix of yours (renewing the token). What matters is that resolving happens after containing, not in its place.

Phase 5 — Verify. Confirm, with evidence, that the system really returned to normal. Not "it looks like it's fine now": the query that shows the orders come in and get registered, the reprocessing of what accumulated, the count that drops to zero. This phase exists because the biggest mistake at the end of an incident is declaring it resolved when it was only paused, and discovering the next morning that half of it was still broken. Verifying is the difference between "I think it's fine" and "I confirmed it is".

And something that wraps around the five phases and isn't a separate phase: the log, which is written while all this is happening. We develop it further down, but keep it in mind from now: each line of version B carried a time and an actor. That wasn't decoration; it was the log building itself in real time.

DETECT ──► COMMUNICATE ──► CONTAIN ──► RESOLVE ──► VERIFY
   │        (I'm starting)  (tourniquet)  (calmly)   (with evidence)
   │                                                       │
   └──────────── the LOG is written throughout ────────────┘
        (each line: time + what was done or learned)

The two roles that shouldn't be the same person

This is the second big idea of the lesson, and it comes straight from the emergency room: whoever fixes the incident and whoever talks to the business shouldn't be the same person.

The two roles have names worth using:

Whoever runs it (or whoever resolves). They have their hands inside the problem. Their attention is on containing and resolving, and that attention is a scarce and fragile resource: each interruption to explain something costs them minutes of recovering their train of thought. They need protected concentration.

Whoever communicates. They talk to the business —Andrea, Gerardo, Lucía—, absorb their questions, translate the technical status into business language, and give whoever runs it the shield they need to work. They don't touch the system; their tool is the incident thread and the conversation.

Why they can't be the same person, even if there are spare hands to try it:

Because the two tasks compete for the same attention. Explaining an incident well to someone anxious requires concentration; containing and resolving too. One person doing both does both badly: contains distractedly and communicates halfway. Version A of the example was exactly that: absorbed in the diagnosis, didn't communicate, and Andrea got scared.

Because the business shouldn't interrupt whoever's fixing. If Andrea writes directly to whoever has their hands in the system, each of her messages is an interruption at the worst moment. With a communication role, Andrea has someone to talk to who is not whoever's resolving, and so her legitimate questions don't cost the solution any time. The communication role is, in part, a bumper.

Because communication is real work, not an afterthought. Trying to communicate as something done "in the spare moments" of the fix guarantees it gets done late and badly, because during an incident there are no spare moments. Giving one person the explicit job of communicating is what ensures it happens.

An honest nuance for small teams: in a team of six, at three in the morning, there may be only one person awake. What do you do then? Two things. First: if the incident is big, you escalate to get the second person —waking someone up to communicate is a legitimate use of escalation—. Second: if you're really alone, communication becomes asynchronous and spaced out —one line in the thread every fifteen minutes— instead of a live conversation, and that's acceptable as the lesser evil. What is not acceptable is for communication to disappear because you were busy. Even if you're alone, the "update in 15 min" from the example protects the business without stealing you barely any time.

The log written while it's happening

The incident log is the piece almost everyone postpones —"I'll write it later, when this calms down"— and that's exactly the way to get it wrong. A log written afterward is a log reconstructed from memory, and the memory of an incident is terrible: you compress the time, forget the order, and rationalize the decisions ("obviously I contained first") even if it wasn't so. The log has to be written while it's happening, for three reasons.

Because memory lies about the times. After an incident, everyone remembers it was faster than it was, and in the order that makes sense, not the one it happened in. Only a log with real times, written in the moment, preserves the truth of how long each phase took. And that truth is what the retrospective in lesson 6 needs: without it, the retrospective argues about memories, not about facts.

Because it's the communication and the log at once. Notice that in version B, the lines you wrote in the thread to communicate are the log. They're not two jobs: communicating in a thread with a time leaves, as a byproduct, the incident timeline. That's the cheapest way to log: make your communication end up written and timestamped.

Because it takes mental load off at the worst moment. Writing "02:56 — unpublished order-sync" frees you from remembering it. During an incident your head is saturated; the log is external memory that lets you focus on the problem without fear of forgetting what you already did. It's the same as a surgeon calling out each step aloud: it's not ceremony, it's not losing count.

What do you note down? Not everything —a log that tries to capture every click becomes unwritable in the middle of an incident—. You note the milestones: when it was detected, when it was communicated, each containment action, when the cause was understood, when it was resolved, when it was verified. Each line with two things: the time and what was done or learned. Nothing more. The practical rule: if someone who arrived late to the incident read only the log, would they understand what happened and in what order? If yes, the log is complete.

An example of the minimal format, which is the same one from the worked example:

INCIDENT · order-sync losing orders · 2026-07-16
Runs: you · Communicates: (alone, asynchronous)

02:51  Alert: order-sync failing in Create order in ERP
02:52  Check: dead_letter is NOT growing and orders are arriving → they're LOST
02:54  Communicate start in the thread. Don't know the cause yet.
02:56  CONTAIN: unpublish order-sync. storefront retries 6h → safe
03:10  Diagnosis: failed ERP deployment (vendor). Nothing on my side
03:15  Communicate status: contained, cause, nothing being lost
04:20  The vendor reverted their deployment
04:30  Republish order-sync. storefront resends what accumulated
04:40  VERIFY: orders come in and get registered. Zero lost. Closing

Ten lines. Anyone who reads them understands the complete incident, and they're the exact raw material of the retrospective. That log cost no extra time: nine of its ten lines were communication you had to do anyway.

Common mistakes

Diving into understanding the cause before containing (conceptual). What happens: the incident arrives and the technical instinct takes control —"why is this happening?"— and goes straight to investigating. While diagnosing, the damage continues: orders being lost, emails being sent, queue growing. When the cause is finally understood, damage has already accumulated that containment would have avoided. Why it happens: understanding is what a technical person knows and enjoys doing, and containing with a crude solution (unpublishing something) feels like giving up or like "doing something ugly". How to spot it: in your last incident, how long did it take you to stop the damage versus to start diagnosing? If you diagnosed first, you inverted the order. How to fix it: train the reflex of asking "how do I make it stop getting worse?" before "why is it happening?". Containing with something crude and reversible is almost always possible without knowing the cause, and buying time isn't giving up: it's what makes the later diagnosis calm instead of desperate.

Staying silent while fixing (practical). What happens: whoever resolves gets so focused on the problem that they stop communicating. The business, with no information, gets distressed, writes again and again, and sometimes starts making its own ill-informed decisions ("I'm going to tell customers there was a hack"). The silence does more damage than the incident. Why it happens: during an incident communication feels like a distraction from the real work, and it gets postponed until "when I have something to say". How to spot it: if in your last incident the business found out about the status by asking rather than by a notice from you, you communicated late. How to fix it: communicate that you're starting before putting your hands in, even if you don't have the cause, and set a rhythm ("update in 15 min") even if the update is "still on it, contained, no news". And if you can, separate the role: have another person communicate so you don't have to choose between fixing and talking.

Declaring it resolved without verifying (practical). What happens: the fix is applied, the visible symptom disappears, and the incident is closed with a "done". The next morning it turns out only part had been paused, or that what accumulated was never reprocessed, and the incident resurrects, now with less team energy to handle it. Why it happens: at the end of an incident everyone is tired and wants it to be over, and "it looks like it's working now" is a very tempting conclusion. How to spot it: if your closing criterion was "I no longer see the error" instead of "I ran the query and confirmed the data is complete", you closed without verifying. How to fix it: define for each incident a verification with concrete evidence —a count at zero, a query that shows the data, a confirmed reprocess— and don't close it until you have it. The verify phase exists precisely to resist the temptation of believing it's fine, when it's only almost.

Exercises

Exercise 1 — Order the actions. During an incident at Terra Market, someone did these six things, but out of order. Reorder them according to the five phases and point out which order mistake was made:

(1) Wrote in the thread: "resolved, the ERP is responding now". (2) Investigated the ERP logs to understand why it was failing. (3) Ran the dead_letter query and saw that orders were being lost. (4) Unpublished order-sync to stop the loss. (5) Ran the final query and confirmed that orders come in and get registered. (6) Notified in the thread that there was an incident and they were handling it.

See solution

The correct order by phases:

• Detect: (3) ran the query and saw orders were being lost.
• Communicate (start): (6) notified there was an incident and they were handling it.
• Contain: (4) unpublished order-sync to stop the loss.
• Resolve: (2) investigated the logs to understand the cause.
• Verify: (5) ran the final query and confirmed complete data.

And one action is out of place: (1) "resolved, the ERP is responding now" is a closing declared before verifying. "The ERP is responding now" isn't the same as "orders come in and get registered"; it's exactly the mistake of declaring it resolved by the visible symptom instead of by evidence. The correct closing is (5), not (1). If the incident was closed with (1), most likely the next morning the orders accumulated in storefront would never have been reprocessed and the problem would resurrect.

Why it works: the exercise shows that the actions of an incident almost never happen in the correct order by themselves —instinct wants (2) before (4), and (1) instead of (5)—. Having the five phases as a template is what imposes the order instinct doesn't impose. Notice that (2), the diagnosis, which feels like "the real work", is the fourth thing done, not the first.

Exercise 2 — Design the containment. For each of these three Terra Market incidents, propose a containment that stops the damage without needing to know or fix the cause. Remember: the containment can be crude and reversible; its only job is that the damage stops growing.

(a) shipment-notify is sending duplicate emails in bulk due to a carrier resend. (b) order-sync creates duplicate orders in the ERP because an idempotency guard is failing under load. (c) inventory-update is pushing zero prices to storefront due to bad data from the ERP, and the store is already showing free products.

See solution

(a) Containment: unpublish (or pause) shipment-notify. It stops the sending of emails immediately. You accept that while it's paused no shipment notices go out, but a delayed notice is repairable and a sent duplicate email isn't. The carrier events can be reprocessed later. You didn't need to know why the carrier resends: you just closed the valve.

(b) Containment: unpublish order-sync. Since storefront retries the webhooks for six hours, the orders accumulate safely instead of being duplicated. You traded "duplicates someone will have to cancel one by one" for "orders on pause, recoverable". You didn't fix the idempotency guard —that's design, goes to the other guide and to the retrospective—; you just stopped the production of duplicates.

(c) Containment: pause inventory-update AND —this is key— assess whether what was already pushed needs to be reverted. Pausing stops new bad prices, but here the damage already went out to the store: there are zero-price products visible to customers who could buy them. The complete containment includes either taking those products down from the store, or forcing a push of the last known good inventory. This case teaches that sometimes containing isn't just "closing the valve" but also "cleaning up the puddle already made", when the damage is visible and still having an effect.

Why it works: in all three cases the containment is the same mental operation —what's the fastest, most reversible lever to make the damage stop growing?— and in all three it's unpublishing or pausing something, accepting a minor cost (delayed notices, orders on pause) in exchange for stopping a greater, irreversible damage. Notice that no containment required understanding the cause, and that case (c) adds the nuance that sometimes there's a puddle to clean up in addition to a valve to close.

Exercise 3 — Distribute the roles. It's a big incident at Terra Market at 15:00 (business hours, people are awake). Available: you (n8n expert), Daniela (junior, already accompanied), Marco (lead), and Andrea (head of operations) is asking what's happening. Distribute the roles of "whoever runs it" and "whoever communicates", justify it, and say what would happen if you mixed them.

See solution

A reasonable distribution:

• Whoever runs it: you. You're the one who contains and resolves the fastest in n8n. Your hands should be in the system, with your concentration protected.
• Whoever communicates: Marco. He's the lead, speaks the business language, and is the natural person to deal with Andrea. His job during the incident is to absorb Andrea's questions, translate your technical status into business language, and be the bumper that lets you work. He doesn't touch the system.
• Daniela: observes and supports whoever runs it. This is a golden moment for her accompaniment —seeing a real incident run well is worth more than ten runbooks read—. She can run verification queries that you dictate to her, without making decisions, and so she learns the order without the pressure of responsibility.
• Andrea: is "the family". She has no operational role; she's whom you communicate to. Her anxiety is legitimate and Marco's job is to manage it, not ignore it.

What would happen if you mixed the roles —if you ran it and communicated with Andrea at the same time—: each of Andrea's messages would pull you out of the problem, you'd contain distractedly, and you'd probably communicate late and badly, which is exactly version A of the worked example. Andrea, without a dedicated interlocutor, would write more and more anxiously, and her unmanaged anxiety could lead her to make ill-informed business decisions. Separating the roles isn't bureaucracy: it's what allows you to resolve well and Andrea to be informed, two things a single person can't give at once.

A nuance: in an incident at 03:00, this ideal four-person distribution doesn't exist —maybe it's just you—. That's where the escalation from lesson 4 comes in: if the incident is big, waking Marco up to communicate is a correct use of escalation, not a nuisance. And if you're really alone, you communicate asynchronously and spaced out, which is the acceptable lesser evil.

Why it works: the exercise shows that the roles of an incident don't depend on hierarchy but on the task —whoever contains best runs it, whoever talks to the business best communicates— and that the separation protects two scarce resources at once: your concentration and the business's peace of mind. Mixing them sacrifices both.

Summary and next step

In this lesson you learned to run an incident instead of suffering it. With the image of the emergency room you saw the sacred order —stabilize first, understand after— and its two consequences: containing comes before diagnosing, and whoever saves the patient isn't whoever talks to the family. You compared the ERP incident run and not run, and saw that the difference between forty minutes of lost orders and zero wasn't in the technical knowledge —the diagnosis was equally good in both— but in what was done first. You walked through the five phases in order —detect, communicate, contain, resolve, verify— with containing in the place intuition denies it. You understood why the two roles shouldn't be the same person, because they compete for the same attention and because the business shouldn't interrupt whoever's fixing. And you learned to write the log while it's happening, with a time and a fact per line, which comes almost free because it's the same communication you already had to do, and which is the raw material of what comes next.

Before moving on you should be able to: name the five phases in order and justify why containing comes before resolving; explain why whoever runs it and whoever communicates shouldn't be the same person; and write an incident log with times and facts that another person can understand.

You have the incident run and closed, and you have its log. What's missing is what turns an incident into learning instead of a scare that gets forgotten: the retrospective. Lesson 6 is half of the fourth piece of the kit —the memory— and it brings the most counterintuitive argument of the module. "Blameless" isn't a gesture of kindness: it's a method, and it has a hard logic behind it. If looking for someone to blame has a cost for whoever participated, people hide information —the real times, the questionable decisions, the "I thought that…"—, and without that information the system can't really learn. You're going to see the structure of the retrospective document, and above all the question that orders it: what made this possible?, not who did it?.

Resources

• Executions — n8n Docs — the executions list from which the scope of an incident is detected and what accumulated is reprocessed in the verify phase.
• Configure workflow settings — n8n Docs — the reference for how a workflow is activated and deactivated, which is the most-used containment lever (unpublish to stop the bleeding).
• Postgres node — n8n Docs — the node used to run the queries that detect whether anything is being lost and the ones that verify, with evidence, that the system returned to normal.
• n8n-workflow-contracts-and-idempotency-guide — the sister guide where the correctness redesign lives (idempotency guards, compensation) that an incident identifies as a cause but that gets fixed afterward, not live.