5. Rollback and operations runbook

Description

By the end of this lesson you will be able to revert a change that broke production: going back to the repository's last good JSON and re-importing it into the affected environment, with the Git and CLI you already master. You're going to know the difference between the two ways of "going back" in Git —git revert and going back to a previous commit— and when to use each. And you're going to write a runbook: a numbered procedure, with an owner and verification, that someone else —or you yourself under pressure— can follow without thinking, at three in the morning, when something breaks.

This matters because no system is safe without an emergency exit, and the moment to build it is before you need it. Everything you did up to here —versioning, isolating environments, testing, promoting carefully— reduces the probability of something breaking, but doesn't bring it to zero. A well-tested change sometimes fails in prod because of something staging never reproduced. When that happens, the question isn't "how do I avoid the mistake?" —it's already too late— it's "how fast do I get back to the version that worked?" The difference between two minutes and two hours of broken orders is exactly what this lesson gives you: a written, rehearsed plan, instead of improvising in a panic.

Connection to the module: lessons 2, 3, and 4 built the road forward —promoting, reviewing, building with AI— and its quality control. This lesson builds the net from the other side: what you do when, despite all the care, a change reaches prod and fails. Here the phrase you've repeated since Module 1 finally pays off: "the repository is the source of truth." Rollback is the moment that source of truth saves you: you go back to it. And the runbook you write here is a direct piece of lesson 8's final deliverable. It's the lesson turning your repository from "a place where I save versions" into "my insurance against disasters."

What a rollback is

Let's start with the word, because it's the central concept.

A rollback is returning a system to a previous state that did work, after a change broke it. It isn't "fixing the bad change"; it's abandoning it and going back to the last known good version. The distinction matters: when prod is broken and failing orders are coming in, it isn't the moment to calmly debug the bug —that comes later, in dev, with no pressure. It's the moment to stop the bleeding by going back to what worked. Rollback is the tourniquet, not the surgery.

Think of it as a car's spare tire. When you get a flat on the highway, you don't fix the damaged tire on the shoulder, with cars passing by: you put on the spare, which you know works, and continue on your way. You fix the damaged tire later, at the shop, calmly. Rollback is putting on the spare: the previous version, already known good, goes back in its place so the system keeps running. Debugging the change that failed is the shop, and it's a dev problem, not a prod one.

This is where your repository pays back all the work you put into it. Because the "previous working state" isn't a memory or a hope: it's a concrete commit in cumbre-automations, with its date, its author, and its reason. The spare is in the trunk, inflated and ready. A well-done rollback is, literally, pulling that version out of the repository and putting it back in the affected environment. Without a repository, there's no spare; just a flat tire and the road.

The two ways of going back in Git

Git offers two ways of going back to a previous state, and confusing them causes problems, so it's worth telling them apart carefully. Both assume the good commit exists in your history —that's why you versioned.

Way 1: git revert — undoing a change by creating a new one. git revert takes a specific commit and creates a new commit undoing exactly what that one did. If the bad commit added three lines and removed one, the revert removes those three and adds the one back. The key thing: it doesn't delete the bad commit from history; it neutralizes it with a commit on top that cancels it. Your history stays complete —you see the bad change and its reversal— which is exactly what you want in a shared repository: nobody loses the record of what happened.

git revert <bad-commit-hash>

This opens a new commit with a message like Revert "order-triage: lower threshold to 100", canceling that change. The history gains an entry, it doesn't lose any.

Way 2: bringing the file back to how it was in a previous commit. Sometimes you don't want to revert "the last commit" entirely, but bring a specific file back to the state it had in a known good commit. For that, you recover that file's version from that commit:

git checkout <good-commit-hash> -- workflows/order-triage.json

This brings order-triage.json back to how it was in the good commit, and leaves it in your working tree ready to commit. It's surgical: it only touches that file, not the rest of the repository. Useful when you know exactly what the last good version of one workflow was and want that one, regardless of what other commits happened in between.

Which one to use? The practical rule:

• If the problem is "the last change was bad, cancel it" and you want the history to show it clean and shareable, use git revert.
• If the problem is "I want this workflow exactly as it was in this good version", use git checkout <commit> -- file.

In both cases, the result that matters is the same: in your repository, workflows/order-triage.json becomes the good version again. And that's just the rollback's first step. Because the repository isn't the instance.

The rule that keeps you from destroying others' work. There's a third way, git reset --hard, which deletes commits from history instead of neutralizing them. Don't use it for a rollback in a shared repository. Deleting history in a repo where more people work destroys others' in-flight work and rewrites history others already have. To revert, git revert is the safe way: it leaves history intact and adds the fix on top. Rewriting history is a delicate operation this guide deliberately doesn't use for operating production.

From the repository to the instance: the complete rollback

Here's the point most often forgotten, and the one making the difference between a rollback that works and one that doesn't. Bringing the file back to the good version in the repository doesn't fix prod. The broken workflow runs on the prod instance, not in your repository. Fixing the repo is necessary but not sufficient: you still have to bring that good version back to the instance.

Remember Module 1's model: the repository is the source, the instance is where it runs (runtime). A rollback touches both, in order:

1. In the repository (source): you bring order-triage.json back to the last good version, with git revert or git checkout <commit> -- file, and commit it. Now the source of truth says, again, "this is the correct version."
2. In the instance (runtime): you re-import that good version into prod with lesson 2's import:workflow, and activate it. Now the instance runs, again, the good version.

Both steps, always. If you only do step 1, your repository is healthy but prod is still broken —the spare is in the trunk but you didn't put it on. If you only do step 2 —re-importing an old version without fixing the repo— prod works but your repository lies, and the next person promoting from it is going to re-break everything. Rollback means bringing the good version back to the source and the runtime.

This is, exactly, lesson 2's promotion put to work for reversion. There's no new rollback command: there's the import:workflow you already know, importing a previous version instead of a new one. A rollback is a promotion backward in time: you promote to prod the version that worked yesterday.

Rollback versus "roll forward": when to use each

There's a decision worth having thought through beforehand, because during the incident there's no time to deliberate it: revert to the previous version (rollback) or fix the problem with a new forward change (roll forward)?

• Rollback is going back, to the known good version. It's the right call in the vast majority of incidents, for a simple reason: the previous version you already knew worked, so going back to it is predictable and fast. Under pressure, predictable wins.
• Roll forward is fixing the bug with a new change moving forward, instead of going back. It makes sense in a specific case: when going back is impossible or worse than moving on. For example, if the "previous" version was also broken in a different way, or if a data change already happened and rolling back the workflow doesn't undo it. In those cases, the forward fix might be the only way out.

The practical rule under pressure: rollback by default, roll forward only when rollback isn't a viable option. The instinct of "I'd rather just fix it right, once and for all" pushes toward roll forward, but a new fix made in a hurry, untested, is exactly what broke prod in the first place. Go back to the known thing first, fix it properly afterward, in dev, calmly. Emergency roll forward is the exception proving the rule, not the usual path.

What a runbook is

Now the lesson's second half, and the one making it professional. Knowing how to revert isn't enough; it has to be written down.

A runbook is an operating procedure written as a list of numbered steps, meant to be run under pressure, by someone who might not have designed the system. The name comes from operations: the "book" you "run" when something happens. It isn't explanatory documentation —it doesn't explain why things are the way they are— it's an emergency recipe: do this, then this, then verify this. Its value is that, at the worst moment —production down, people waiting, your pulse racing— you don't have to think or remember: you follow the steps.

The analogy is a pilot's emergency checklist. When an engine fails, the pilot doesn't improvise or try to remember what they learned years ago: they pull out the matching emergency card and run the steps in order, one by one, skipping none. That card was written calmly, on the ground, by people who carefully thought through what to do when an engine fails. The pilot doesn't design it in the air; they execute it. A runbook is that card for your system: written calmly, when everything works, to be executed without thinking when nothing does.

A good runbook has a recognizable anatomy. These are its elements:

• Title and trigger. What situation this runbook covers. "When to use this": for example, "when order-triage in prod is processing orders incorrectly after a promotion."
• Preconditions. What you need on hand before starting: access to the repository, access to prod's container, knowing what the last good commit was.
• Owner. Who runs this runbook. On a team, which person (or role) has the authority and access to do it.
• The numbered steps. The exact sequence, each step a concrete, verifiable action. No ambiguity, no "and then you fix the problem": steps that get executed, not interpreted.
• Verification. How you know it worked. The concrete signal prod returned to normal: orders being processed correctly again, the CRM node no longer failing.
• What to do afterward. The follow-up once the emergency's passed: debugging the bad change in dev, calmly, to understand what happened.

Notice the runbook includes an owner and verification, not just steps. Without an owner, in an emergency two people do the rollback at once and step on each other, or nobody does it because each thinks it's the other's job. Without verification, you run the steps and don't know if it worked —and in production, "I think it's fine now" isn't enough. A runbook without those two isn't a runbook; it's a list of commands.

Worked example: order-triage's rollback runbook

Let's write Cumbre's real runbook, the one going into the repository (at docs/runbook-rollback.md) and that lesson 8 includes in the deliverable. This is the document you follow when order-triage breaks production.

# Runbook: order-triage rollback in prod

## When to use this
When order-triage, in the prod environment, is processing orders
incorrectly after a promotion, and you need to go back to the last good version.

## Owner
The on-call operator with access to the cumbre-automations repository and
the n8n-prod container. One person executes; others observe.

## Preconditions (have this ready before starting)
- Write access to the cumbre-automations repository.
- Access to the n8n-prod container (docker exec).
- The hash of the last known good commit (find it with: git log --oneline
  workflows/order-triage.json).

## Steps
1. Stand in the repository:
       cd cumbre-automations

2. Identify order-triage's last good commit:
       git log --oneline workflows/order-triage.json
   Note the hash of the commit BEFORE the one that broke prod. Example: a1b2c3d.

3. Bring the file back to that good version in the repository:
       git checkout a1b2c3d -- workflows/order-triage.json
       git commit -m "rollback: order-triage to the last good version (a1b2c3d)"

4. Copy the good JSON to prod's container:
       docker cp workflows/order-triage.json n8n-prod:/tmp/order-triage.json

5. Re-import the good version into prod:
       docker exec -u node -it n8n-prod n8n import:workflow \
         --input=/tmp/order-triage.json

6. Open order-triage in prod's editor. Verify the logic is the good
   version and the credentials (AI Agent, HTTP Request) resolve.

7. Activate order-triage in prod (activation switch in the editor).

## Verification (how you know it worked)
- A test order comes in and gets processed like before the incident.
- The CRM node no longer fails on new executions.
- Real orders go back to classifying correctly.

## What to do afterward (once there's no pressure)
- In dev, reproduce and debug the change that broke prod. Understand the cause.
- Once you have the fix, promote it through the normal flow:
  dev -> staging (with sandbox test) -> diff review -> prod.
- Update this runbook if any part of the procedure didn't work as expected.

Read it paying attention to what makes it executable under pressure and not just "correct":

• Every step is a concrete action with its exact command. It doesn't say "revert the workflow"; it says git checkout a1b2c3d -- workflows/order-triage.json, the literal command. Whoever runs it doesn't have to remember the syntax at three in the morning: they copy it from the runbook.
• The steps combine the rollback's two halves. Steps 1-3 fix the repository (source); steps 4-7 fix the instance (runtime). The runbook doesn't forget the second half, the one people skip.
• Verification is concrete and observable. It doesn't say "confirm it works"; it says "a test order comes in and gets processed like before." It's a signal you can see, not a feeling.
• There's an "afterward." The runbook stops the bleeding (rollback) and explicitly separates the surgery (debugging in dev) for later. It doesn't mix putting out the fire with investigating its cause.

What to expect running this runbook during a real incident: in five or six minutes, prod goes back to running the version that worked, while the change that failed waits in dev to be debugged calmly. Compare that to the alternative without a runbook: half an hour remembering commands, hunting for which commit was the good one, doubting whether you already fixed the repo or just the instance, all while broken orders keep coming in. The runbook turns panic into procedure.

Rehearsing the rollback before you need it

Here's the advice separating someone who has a runbook from someone who can use it: rehearse it before the incident. A runbook you never ran is a hypothesis, not a plan.

The reason is simple and hard: a written, untested runbook almost always has a mistake. A command with the wrong container name, a step assuming access you don't have, a verification you can't observe. And the worst moment to discover that mistake is during the real incident, when every minute costs orders. Rehearsing it calmly —deliberately causing a "problem" in staging and running the runbook to revert it— is where you find those mistakes at no cost.

The rehearsal is simple and you can do it today: in staging (not prod), promote any change, and then run your rollback runbook to go back, following the steps to the letter, with no improvising. If any step doesn't work as written —the command fails, the verification can't be observed— that's a valuable finding: fix it in the runbook. When the rehearsal runs clean from start to finish, you have a real plan, not an intention.

Think of it as a building's fire drill. You don't do the drill during the fire; you do it beforehand, cold, so that when the fire's real, everyone knows the way out without thinking. Nobody considers the drill a waste of time on the day there's a real fire. Rehearsing the rollback is your fire drill, and the "fire" in production is going to arrive someday. Better to have rehearsed.

There's an extra benefit of rehearsing worth naming: it gives you confidence to promote. When you know your rollback works —because you rehearsed it— you promote to prod more calmly, because you know that if something goes wrong, you have the tested exit. The safety net doesn't just catch you when you fall; it gives you the courage to walk the tightrope. A team that rehearsed its rollback deploys more often and with less fear, because the cost of a mistake stopped being catastrophic.

Common mistakes

Fixing the repository and forgetting to re-import into the instance (practical, the forgotten half). What happens: someone reverts order-triage.json to the good version in the repo, commits it, and calls the rollback done. But prod keeps running the broken version, because nobody re-imported the good one into the instance. Orders keep failing while the person believes they already fixed it. Why it happens: fixing the source gets confused with fixing the runtime; the repo looks fine and gives a false sense of being done. How to spot it: if you reverted in the repo but didn't run import:workflow against prod, you're missing the half that actually fixes production. How to fix it: rollback is two steps, source and runtime. The runbook includes both precisely so the second one doesn't get skipped. Fixing the repo prepares the spare; re-importing puts it on the car.

Starting to debug the bug in prod during the incident (conceptual). What happens: prod is broken, and someone, instead of reverting, starts investigating why the new change fails, editing in prod, testing hypotheses, while orders keep failing. The investigation takes half an hour; the rollback would have taken five minutes. Why it happens: the instinct to "fix the cause" is strong, and reverting feels like "giving up." How to spot it: if you're debugging in production with orders falling, you inverted the order. How to fix it: tourniquet first, surgery later. Revert to the good version to stop the bleeding; debug the cause afterward, in dev, calmly. Production isn't the place to investigate; it's the place to get back to working as soon as possible.

Not knowing what the last good commit was (practical). What happens: the incident hits, the person opens the runbook, reaches the "go back to the good commit" step, and realizes they don't know which one it was, because the commit messages are vague ("changes," "fix," "update") and don't distinguish good from bad. Why it happens: commit messages got neglected while everything worked, and under pressure there's no time to reconstruct the history. How to spot it: if looking at your workflow's git log you can't say in ten seconds which was the last good version, you have this latent problem. How to fix it: write clear commit messages always —"order-triage: raise threshold to 5000," not "changes"— so git log reads clearly in an emergency. Rollback depends on being able to identify the good commit fast, and that's earned with commit discipline before the incident.

Having a written runbook but never having run it (conceptual). What happens: someone writes a polished runbook, saves it, and feels at ease. The real incident arrives, they run it for the first time, and at step 5 the command fails because the container's name was misspelled. Now they're debugging the runbook during the emergency. Why it happens: writing the runbook feels like finishing the work; running it cold seems unnecessary "because it's already written." How to spot it: if your runbook never ran start to finish, you don't know if it works; it's a hypothesis. How to fix it: rehearse it in staging, deliberately causing a problem and reverting it with the runbook to the letter. The mistakes you find in the rehearsal are the ones you won't suffer during the incident. A tested runbook is a plan; an untested one is a wish.

Exercises

Exercise 1 — Choose the way to revert. For each situation, say whether you'd use git revert <commit> or git checkout <good-commit> -- workflows/order-triage.json, and why: (a) the last commit lowered the threshold to 100 and it needs canceling, keeping the history clear in a shared repo; (b) you want order-triage.json exactly as it was five commits ago, regardless of what happened in between.

See solution

(a) git revert <commit>. You want to cancel a specific commit while keeping the history clean and shareable: git revert creates a new commit undoing the bad one without deleting it from the history, so the shared repo keeps the complete record —the bad change and its reversal— and nobody loses history. It's the safe way for production and teams.

(b) git checkout <good-commit> -- workflows/order-triage.json. You want a specific file at the state of a specific commit, regardless of what other commits happened in between. The file checkout is surgical: it only brings that order-triage.json back to how it was in that commit, ready to commit, without touching the rest of the repository.

In both cases, remember reverting in the repository is only the first half: afterward you have to re-import the good version into prod's instance for the rollback to reach production.

Why it works: the distinction is "canceling an entire commit, with the history visible" (revert) versus "bringing a file to a known state" (file checkout). Confusing them leads to reverting too much or too little. And neither one is git reset --hard, which deletes history and doesn't get used to operate production in a shared repo.

Exercise 2 — Find the gap in a runbook. A teammate wrote this mini-runbook: "1. Revert the workflow in the repo with git revert. 2. Verify prod works now." What's this runbook missing to be truly executable? Name at least three things.

See solution

It's missing several critical things:

1. The runtime half. It only reverts in the repo (step 1) and jumps straight to verifying prod (step 2), with no intermediate step of re-importing the good version into prod's instance with import:workflow. As written, the repo ends up fine but prod stays broken: step 2's verification would always fail.
2. Exact commands. "Revert with git revert" doesn't say which commit to revert or how to identify it. Under pressure, you need the literal command and the step of checking git log to find the right commit.
3. Owner. It doesn't say who executes. In an emergency, with no owner, either two people step on each other or nobody acts.
4. Concrete verification. "Verify prod works now" is a feeling, not a signal. It should say something observable: "a test order comes in and gets processed correctly; the CRM node no longer fails."
5. Preconditions and what to do afterward. It doesn't say what you need on hand beforehand (access, the good commit's hash) nor what to do with the bug once reverted (debug it in dev).

Why it works: the exercise trains you to read a runbook with a critical eye, which is how they get improved. The most serious gap is almost always the same: forgetting the runtime half (re-importing into the instance). A runbook that fixes the repo but not the instance gives the illusion of having resolved the incident when production is still down.

Exercise 3 — Rehearse a rollback in staging. Describe, step by step, how you'd rehearse your order-triage rollback runbook in staging, without touching prod, to find mistakes before a real incident. What "problem" would you deliberately cause, and how would you know the rehearsal succeeded?

See solution

A possible rehearsal:

1. In staging, deliberately promote a "bad" version of order-triage —for example, one with the threshold changed to an absurd value like 100— following the normal promotion flow. This is the "caused problem": a change you want to revert.
2. Verify staging now behaves "badly" (sends everything to manual review with the 100 threshold). This is your simulated incident.
3. Now run your rollback runbook to the letter, but pointing at staging instead of prod (n8n-staging container): identify the last good commit, revert the file in the repo, copy and re-import the good version into staging, verify and activate.
4. Confirm the verification: staging goes back to behaving with the good threshold.

You know the rehearsal succeeded if you ran the runbook start to finish without improvising or correcting on the fly. If any step failed —a command with the wrong container, a verification you couldn't observe— that's exactly the finding you were looking for: fix it in the runbook now, cold, so it doesn't surprise you during prod's real incident.

Why it works: rehearsing in staging gives you every benefit of testing the rollback —finding the runbook's mistakes, gaining confidence, measuring how long it takes— with none of the risks of touching prod. It's the fire drill: done cold, precisely so the day of the real fire isn't the first time. And as a bonus, it lets you deploy to prod more calmly, knowing your emergency exit is tested.

Summary and next step

In this lesson you built the system's emergency exit. You understood a rollback is going back to the last known good version when a change broke production —the tourniquet, not the surgery; the spare tire, not the roadside repair— and that your repository is what makes that return possible, because the good version lives there as a concrete commit. You told apart the two ways of going back in Git —git revert, which cancels a commit while keeping the history visible, and git checkout <commit> -- file, which brings a file to a known state— and why git reset --hard doesn't get used to operate production in a shared repo. You locked in that rollback touches source and runtime: reverting in the repository and re-importing the good version into prod's instance, because fixing only the source leaves production broken. You wrote a runbook with its complete anatomy —trigger, preconditions, owner, numbered steps with exact commands, observable verification, and what to do afterward— and saw why rehearsing it before the incident is what turns a written runbook into a plan that actually works, besides giving you confidence to promote.

Before moving on you should be able to: explain what a rollback is and why it isn't the same as debugging the bug; choose between git revert and git checkout <commit> -- file depending on the situation; describe the rollback's two halves (repo and instance); and name a runbook's elements and why it gets rehearsed cold.

Lesson 6 shifts registers: from operation to decision. You already know how to do the whole cycle —promoting, reviewing, building with AI, reverting— by hand, on Community, at zero cost. Lesson 6 puts that against the paid alternative: n8n's native Git (Enterprise), with its push and pull buttons built into the interface. You're going to honestly compare both flows, see a decision matrix by team size, budget, and compliance, and confirm the one thing that doesn't change either way, whether you pay or not: the repository remains the source of truth, and the rollback you just learned works the same in both worlds.

Resources

• git revert — Git Documentation — the reference for the command undoing a commit by creating a new one, with no history deleted; the safe way to revert in a shared repo.
• git checkout — Git Documentation — how to bring a file to a previous commit's state, the surgical reversion of a single workflow.
• Use the command line — n8n Docs — the import:workflow you use to re-import the good version into the instance, the rollback's runtime half.
• Google SRE Book — On-Call and runbooks — the operations thinking behind runbooks: why a written, rehearsed procedure beats improvising under pressure.
• git log — Git Documentation — how to read the history to identify the last good commit, the step the whole rollback depends on.