Module 6: Rollback And Safety Nets

1. Introduction: what to do when the `apply` goes wrong

Description

The five previous modules built a pipeline that works when everything goes right: a change gets proposed, ci.yml reviews it, someone approves, apply.yml applies it, drift.yml watches that nobody touches it outside the pipeline. This module asks the question no previous module had to answer yet: what happens when something goes wrong? Not "wrong" in the sense of a networking error or a plan that fails on syntax —you already saw that fail and get fixed since Module 3—, but "wrong" in a more uncomfortable sense: a change that merged, that apply.yml applied with no technical errors, and that turns out, in hindsight, to be a business mistake or a risk nobody should have approved. You're going to learn infrastructure's specific rollback pattern, a repository control layer no workflow can execute but that completely changes who can touch main, an honest revisit of this guide family's most-cited incident, and you're going to build —for real, executed— the entire pipeline's first automated guardrail.

Connection to the module

This is the third and last module of building blocks before Module 7's big-picture view and Module 8's capstone. Where Module 3 answered "what would change, and who reviews it?" and Module 5 answered "who applies it, and when?", this module answers a different question no pipeline can dodge forever: "what do we do once something that shouldn't have been applied already got applied?". Lessons 2 and 3 answer with the correct mechanism —reverting the HCL commit, not "undoing" anything by hand—, run end-to-end on the same ci.yml/apply.yml you already know. Lesson 4 adds a control layer that lives outside any YAML file: branch protection, the repository setting that turns "anyone can touch main" into "nothing lands without going through ci.yml." Lesson 5 returns, in more depth, to the terraform-and-iac-guide incident you already named in Module 1. Lessons 6, 7, and 8 close the module with the piece the entire pipeline was missing: a real, automated guardrail that reviews the plan before it reaches apply and stops, on its own, an attempt to destroy the Shipments table.

Why "undoing infrastructure" isn't the same as "reverting code"

Think about how you'd revert a bug in a normal application, with no infrastructure involved: someone deploys version v42 of a service, someone else notices a bug, and the fix is deploying v41 again —an earlier version, already built, already tested, ready to run again—. Application rollback works because the earlier artifact still exists: v41's container image is still in the registry, ready to start up again with a single command.

Infrastructure doesn't have that luxury. When apply.yml runs terraform apply on a change, it doesn't deploy a new "version" of an immutable artifact — it modifies real objects that already exist: it adds a tag to a bucket, changes an IAM policy, deletes a table. There's no "earlier version of the bucket" stored in some registry, ready to be reactivated with a docker run. The only thing that exists is the HCL that described the earlier state, and the only way to "undo" the change is to write that HCL again —or, more precisely, let Git write it for you— and let the same pipeline apply it again.

   APPLICATION ROLLBACK                       INFRASTRUCTURE ROLLBACK

   v42 deployed, with a bug                   change applied, with a problem
        │                                          │
        ▼                                          ▼
   v41 ALREADY EXISTS                          NO "earlier version" of the
   (image in the registry)                     bucket/table/role is stored
        │                                          │
        ▼                                          ▼
   deploy v41 again                            git revert of the HCL commit
   (activate an artifact                       (recreate the DESCRIPTION of
    already built)                              the earlier state)
        │                                          │
        ▼                                          ▼
   the runtime switches                        the SAME pipeline (ci.yml → apply.yml)
   active versions                              runs again, with the plan reversed

This module teaches the right-hand column. The left-hand column —rolling back an application, a container, a running service— is a whole topic in its own right, and lives in kubernetes-and-eks-in-production-guide and aws-serverless-and-containers-guide, not here.

This module's map: the 8 lessons

#	Lesson	What you practice
1	Introduction (this one)	The complete map; why "undoing infrastructure" isn't "reverting code"
2	What rollback means for infrastructure	The exact pattern: `git revert` of the HCL commit, not a new Terraform operation; the boundary with application rollback
3	Hands-on: the `git revert` pipeline	Executed: a real `git revert` of a real commit, `ci.yml` running `plan` on the revert, `apply.yml` attempting to apply it
4	Branch protection as a control	`Settings → Branches`, mandatory PR, mandatory checks — representative, repository configuration `act` can't execute
5	Would a pipeline have stopped the Claude Code incident?	An honest review, in more depth than Module 1: what a pipeline WOULD and WOULD NOT have changed
6	Guardrails of `apply`, named	`conftest`/policy-as-code as the next step — named, pointer to `cloud-security-and-guardrails-guide`
7	Hands-on: a minimal, real, executed guardrail	Executed: a `grep` over `terraform show -json` that fails the job if the plan destroys `Shipments`
8	Project: Andes Cargo's safety net	Executed: the complete pipeline with the guardrail integrated, tested with a harmless change and a destructive one

What this module does NOT build

Three boundaries, declared now:

conftest/Open Policy Agent as a complete policy system (Rego rules, governance of what fails the build versus what only warns) gets named in lesson 6, exactly as terraform-and-iac-guide did in its Module 8 — the complete build-out lives in cloud-security-and-guardrails-guide. The guardrail you do build in lesson 7 is deliberately simpler than that: a hand-written grep, not a declarative policy.
Branch protection against a real GitHub.com repository is described with the exact click-path (lesson 4), but not executed: it's repository configuration, not a workflow, and there's no way for act to apply it.
Application rollback (rolling back to an earlier container image version, a Kubernetes rollout) gets named by contrast in lesson 2, without being built here — the boundary is with kubernetes-and-eks-in-production-guide and aws-serverless-and-containers-guide.

This specific module's execution honesty

The same pattern from the five previous modules, with a new nuance worth naming now. Lessons 3, 7, and 8 are genuinely executed, with two nuances that set them apart from everything before:

Lesson 3 reuses exactly the already-built ci.yml/apply.yml, without changing a single line — the only new thing is the commit that reverts, and the plan that revert produces. apply.yml's attempt, like in every previous module without a valid LOCALSTACK_AUTH_TOKEN, fails with the same honest connection refused you already know.
Lesson 7 needs something no previous module needed: a plan that proposes destroying a resource. And here's a direct consequence of a fact you already established in Module 3 and Module 5: without a real apply completed against LocalStack, this project's state is —and always has been— completely empty. An empty state can never produce a genuine delete action: there's nothing applied to destroy. Lesson 7 solves this with a legitimate, honest technique —seeding the local state with a test file (fixture) that declares the Shipments table already exists, exactly the same principle behind testing a security policy against a sample plan instead of depending on live infrastructure—. You're going to see the complete, verified mechanism in lesson 7.

None of this is hidden or explained after the fact: every code block in this module states, right at the point it appears, whether it really ran and what came out of the lab.

Common mistakes

Assuming this module teaches how to "undo" an apply with a Terraform command (conceptual, the mistake this module exists to prevent). What happens: someone looks for something like terraform undo or terraform rollback, commands that don't exist. How to spot it: if your first question opening this module is "what's the Terraform command to undo this?". How to fix it: Terraform doesn't have a rollback command — the complete mechanism is Git's (reverting the commit) combined with the same pipeline you already built (letting plan/apply run again, in reverse). Lesson 2 develops this with full precision.

Believing branch protection and lesson 7's guardrail do the same thing (scope-based). What happens: someone finishes this module thinking both mechanisms are interchangeable or redundant. How to fix it: branch protection controls who can merge and under what process condition (reviewed PR, green checks) — it doesn't look at the change's content. Lesson 7's guardrail specifically looks at the plan's content and blocks a concrete kind of change (destroying Shipments), regardless of who proposed it. They're complementary layers, not one substituting for the other — Module 8 runs them together, in the same pipeline.

Exercises

Exercise 1 — Explain, without using the word "revert," what an infrastructure rollback would do. In two sentences, describe the complete mechanism to a colleague who's never used Terraform in a pipeline.

See solution

A complete answer sounds, roughly, like this: "When an infrastructure change turns out to be a mistake, there's no 'earlier version' of the resource stored anywhere to simply reactivate — what does exist is the code that described the earlier state, in an older Git commit. The fix is creating a new commit that undoes exactly that code change, and letting the same automated pipeline —the one that already reviews and applies every change— process that new commit like any other: it reviews it, and if it's approved, it applies it."

Exercise 2 — Predict lesson 7's central problem before reading it. Based on everything you already know from Modules 3 and 5 about this project's state on this specific machine, why do you think testing a guardrail that detects "destroying Shipments" can't be done simply by deleting the table from the HCL?

See solution

Because, as you already confirmed in Module 3 (lesson 6) and Module 5 (lessons 6 and 7), this project's state on this machine never had a real apply completed against LocalStack — it is, and always has been, empty. A plan can only propose destroying a resource the state believes exists; if the state has no resource at all, removing that resource from the HCL simply means "one less thing to create," not "one thing to destroy." Genuinely testing the guardrail is going to need some way to make the state believe the table already exists, without depending on a real apply.

Exercise 3 — Locate this module's boundary with cloud-security-and-guardrails-guide before reaching lesson 6. Without reading it yet, what do you think conftest does that lesson 7's grep doesn't?

See solution

A reasonable answer: conftest evaluates the JSON plan against a set of declarative rules, written in a policy language (Rego), that can express arbitrarily complex conditions —about any resource, any attribute, combinations of conditions— and that an entire team can maintain as versioned policy code. A grep can only search for one specific text pattern, hand-written for a very particular case (this table, this action) — it works, but doesn't scale to "any policy the team needs" without rewriting the script every time. Lesson 6 confirms this intuition with complete precision.

Summary and next step

In this lesson you saw Module 6's complete map: why infrastructure rollback is, structurally, different from application rollback; the eight lessons that build the complete pattern, from git revert to a real automated guardrail; and this module's specific execution honesty, including the technique lesson 7 needs to be able to test a destruction guardrail without a real, completed apply.

Before moving on you should be able to: explain why Terraform doesn't have an "undo" command; name this module's three new pieces (rollback via git revert, branch protection, the guardrail); and anticipate, without surprises, what you'll be able to run for real and what stays representative.

Lesson 2 starts at the beginning: the exact mechanism of an infrastructure rollback, compared point by point against the application rollback you might already know from another context.

Resources

GitHub Docs — GitHub Actions — official documentation, the basis for this entire module.
HashiCorp Developer — Automate Terraform with GitHub Actions — the complete pattern, already cited since Module 3, that this module hardens with additional controls.
This guide's Module 3 (the-iac-pipeline-fmt-validate-plan) — the ci.yml this module reuses with no structural changes.
This guide's Module 5 (apply-on-merge-the-cd-half) — the apply.yml this module reuses, and the empty-state finding lesson 7 resolves.

← Previous8. Project: Andes Cargo's full plan-to-apply pipeline Next →2. What rollback means for infrastructure