Module 6: Rollback And Safety Nets
5. Would a pipeline have stopped the Claude Code incident?
Description
Module 1 (lesson 2) named the terraform destroy incident an AI agent ran against DataTalks.Club's production infrastructure, and left a preliminary answer: a pipeline doesn't prevent negligence, but it leaves an auditable record the real incident never had. This lesson returns to the same question, now with all the complete pipeline's pieces in hand —mandatory review (Module 3), automatic application (Module 5), rollback (lessons 2-3), branch protection (lesson 4)— and with one more piece, not yet fully built: lessons 6-7's guardrail. The answer, at this depth, has a nuance Module 1 couldn't give yet.
Connection to the module
This lesson directly connects this module's lessons 2, 3, and 4 with the incident already named in Module 1. It doesn't introduce any new technical mechanism — it's, deliberately, an analysis lesson, using everything you've already built to answer a concrete question as honestly as possible. Lesson 6 names the piece still missing to close the analysis completely: an automated guardrail, which lesson 7 genuinely builds.
The incident, again, with the exact facts
In March 2026, Claude Code —an AI agent operating with access to a real AWS account— ran terraform destroy against DataTalks.Club's production infrastructure (Alexey Grigorev), after the agent itself accidentally replaced the active state with an old copy archived in a .zip. The destroy deleted the VPC, the ECS cluster, the load balancers, the bastion host, the RDS database, and its automatic snapshots: close to 1,943,200 rows and 2.5 years of a complete course's history, with roughly ~24 hours of recovery, possible only via direct AWS support (Alexey Grigorev — How I Dropped Our Production Database · Hacker News #47278720 · incidentdatabase.ai — Incident 1424).
The detail that makes this incident more than a horror story, already named in Module 1: the agent had flagged the risk before executing the destroy, and the human supervising the session approved it anyway. It wasn't a destroy run blindly — there was a visible warning, and there was an explicit approval that overlooked it.
Running the incident, step by step, through each layer you built
This time, instead of answering in the abstract, follow the incident through each of this module's specific controls, one by one, and honestly ask yourself whether it would have stopped it.
① The agent replaces the active state with an old one (root cause)
│
▼
② The agent proposes terraform destroy, flagging the risk
│
▼
③ The supervising human approves anyway
│
▼
④ destroy runs against real production: VPC, ECS, RDS, snapshots
│
▼
⑤ ~24h of recovery via AWS supportStep ①, the state replacement. No piece of this module would have directly prevented it — it's not a change-review problem, it's a problem with handling the state itself (the kind of risk terraform-and-iac-guide, Module 4, already covered with locking and remote backends, outside this guide's scope). It's worth naming honestly: this module doesn't solve every risk in an infrastructure pipeline, only the ones within its declared scope.
Step ②, the proposal with the flagged risk. Here's where Module 3's HashiCorp/GitHub pattern does change something, though not in the way you'd expect at first: if that destroy had to go through a real ci.yml, with the plan published as evidence (Module 3, lesson 7) and branch protection requiring a PR to go through review before merging (this module's lesson 4), the risk the agent flagged would have ended up written down, in a public, persistent place —not just visible in a terminal session nobody else sees—. That doesn't prevent the wrong approval, but it radically changes who can see it, and when.
Step ③, the wrong approval. This is the step where you have to be most honest, without dressing up the answer: nothing you built in this module stops a human from approving something dangerous after having seen the warning. A production Environment with required approval (this guide's Module 4) wouldn't have changed the outcome if the same person, seeing the same plan with the same warning, approved it anyway. Branch protection doesn't either — it requires something to approve, not that the approval be correct. This is, precisely, the real limit of everything taught up to this point: it automates the process around a human decision, not the quality of that decision.
Step ④, running the destroy. Here's where the nuance shows up that Module 1 couldn't give yet, because this module's lessons 6-7 guardrail didn't exist yet at that point in the guide. An automated guardrail that reviews the JSON plan and blocks any destructive action on a protected resource —the exact mechanism you build in lesson 7— doesn't depend on human judgment at all. It doesn't matter if the human approving is tired, in a hurry, or simply trusts the agent asking too much: if the plan contains a delete action on a resource marked as protected, the job fails, automatically, before apply.yml even gets the chance to execute anything. This is the first layer in this entire module that acts without needing anyone to make the right decision at that moment — lesson 7 genuinely builds it, not just names it.
Step ⑤, the 24-hour recovery. Outside the scope of any piece of this module — disaster recovery, backups, automatic snapshots, is sre-and-incident-response-guide's territory, not this guide's.
The complete answer, with no dressing up
Putting the five steps together:
A complete pipeline —review, automatic application, rollback, branch protection— wouldn't have stopped the human from approving the destroy after seeing the warning. That's the uncomfortable part, and repeating it with the same honesty as Module 1 matters: automating the process around a decision doesn't change the decision itself.
But an automated guardrail, of the kind you build in lesson 7, would have changed the outcome —not because it made the human more careful, but because it doesn't need the human to be careful at all. If Andes Cargo's Shipments table (or, in the real incident, DataTalks.Club's RDS database) were marked as a resource protected by a guardrail that blocks any delete on it, the CI job would fail automatically while calculating the plan —long before any human got the chance to approve or reject anything—. This is the real difference between "leaving a record of the bad decision" (what you already saw in Module 1) and "making the bad decision technically impossible to execute for a specific resource marked as critical" (what this module's lessons 6-7 add).
With this, this module's complete thesis, in one sentence: branch protection and human review make the bad decision auditable; an automated guardrail, on a specific resource, makes some bad decisions impossible to execute, without depending on anyone noticing in time. Neither layer is "the complete solution" on its own — together, they're as close as this module gets to an honest answer to its title's question.
Common mistakes
Concluding that lesson 7's guardrail "solves" the incident entirely (expectation-based, this lesson's most important mistake). What happens: someone, after reading about the guardrail, thinks this module proves the Claude Code incident "couldn't happen anymore" in a modern pipeline. How to spot it: if your summary of this lesson is "with a guardrail, this would never happen" instead of "with a guardrail on THIS specific resource, THIS specific action would have been blocked." How to fix it: lesson 7's guardrail protects one concrete resource, explicitly marked ahead of time (Shipments, in Andes Cargo's case). A real incident can touch any resource — the guardrail only protects the ones someone thought, in advance, to protect. A complete policy system (conftest, named in lesson 6) scales that protection to more general rules, but isn't infallible either: it still depends on someone writing the correct rule before the incident happens.
Thinking this lesson contradicts Module 1 (conceptual). What happens: someone notices Module 1 said "a pipeline doesn't prevent negligence" and this lesson seems to say something different about the guardrail. How to fix it: there's no contradiction — Module 1 analyzed the pipeline without a content guardrail (everything that existed at that point in the guide was review and approval, both human). This lesson adds a piece Module 1 hadn't built yet: a control that doesn't depend on human judgment at all. The two lessons are consistent; this one simply has one more piece to work with.
Assuming the guardrail would have prevented the root cause (step ① of the incident) (scope-based). What happens: someone thinks lesson 7's guardrail would also have prevented the state from being replaced with an old copy. How to fix it: the guardrail acts on the plan's content, calculated after the state is already in whatever position it's in — it can't detect or prevent the state itself from having gotten corrupted before that comparison. That specific risk lives in state handling (locking, remote backends), a terraform-and-iac-guide topic, not this module's.
Exercises
Exercise 1 — Trace the incident through the five layers, without looking at this lesson. For each of the incident's five steps (state replacement, proposal with flagged risk, approval, execution, recovery), decide whether any piece of this module would have changed it, and which one.
See solution
① No piece of this module — it's a state-handling problem, outside this scope. ② The plan-in-PR pattern + branch protection means the agent's warning ends up written down in a public, persistent place, instead of getting lost in a terminal session. ③ No piece of this module changes the human decision itself — neither branch protection nor an approval Environment guarantees the approval is correct. ④ Lesson 7's guardrail, if the destroyed resource were marked as protected, would have automatically blocked the apply, without depending on any human. ⑤ No piece of this module — disaster recovery is sre-and-incident-response-guide's territory.
Exercise 2 — Answer Module 1's skeptical colleague, now with the new piece. The same colleague from Module 1's Exercise 2 (lesson 2) insists: "if a human can approve something dangerous anyway, no pipeline is worth anything." Answer them, incorporating what you learned in this lesson.
See solution
A complete answer sounds, roughly, like this: "You're right that no pipeline changes a human's decision to approve something dangerous having seen the warning — we already acknowledged that since Module 1. But that doesn't mean a pipeline 'isn't worth anything': an automated guardrail, on a specific resource marked as critical, doesn't ask anyone to make the right decision — it simply makes that action technically impossible to execute, no matter who approves it. It's not a universal solution (it only protects what someone thought to protect in advance), but for the resources that are protected, it completely eliminates the dependence on good human judgment at that specific moment."
Exercise 3 — Design the protection Andes Cargo would need, beyond Shipments. Based on Andes Cargo's four canonical resources (the bucket, the table, the two IAM roles), which others do you think would deserve a guardrail similar to lesson 7's, and why is the Shipments table the right choice for this guide?
See solution
A reasonable answer: the andes-cargo-shipment-docs bucket could also deserve a guardrail (it holds shipping documents, similar in criticality to the table's records), and the IAM roles could deserve a different one —not against destruction, but against changes that dangerously widen permissions—. The Shipments table is this guide's choice because it's the clearest, least ambiguous example of "data that, once destroyed, can't be reconstructed from the HCL" —unlike an IAM role, whose complete definition lives in the code itself and could be recreated with no information loss—, the same kind of reasoning that made the RDS database the real incident's most serious loss, not the VPC or the load balancers (which do get recreated identically from the HCL).
Summary and next step
In this lesson you returned to the Claude Code incident with this module's four pieces in hand —rollback, branch protection, and the guardrail still to be built— and traced, step by step, which of them would have changed which part of the real incident. The complete answer has two honest parts: no process control changes a bad human decision made with the correct information already visible; but an automated guardrail, on a specific resource marked as critical, eliminates the dependence on that decision entirely, for that resource.
Before moving on you should be able to: trace the complete incident through this lesson's five layers; precisely explain the difference between "leaving an auditable record" and "making an action technically impossible"; and anticipate, without having seen it yet, what kind of mechanism lessons 6 and 7 build.
Lesson 6 names the complete automated guardrail system —conftest, policy-as-code— as the next step beyond what this guide builds. Lesson 7 genuinely builds the minimal, honest version of that mechanism: the concrete guardrail this lesson already anticipated.
Resources
- Hacker News — incident discussion #47278720 — the original
destroyincident thread, already cited in Module 1. - incidentdatabase.ai — Incident 1424 — the structured record of the incident in the public AI incident database.
- This guide's Module 1 (
02-the-trouble-with-manual-apply.md) — the incident's first analysis, this lesson's direct basis. terraform-and-iac-guide, Module 8, lesson 5 — the incident's complete technical source, including how thestategot replaced.