Module 6: Operating The Claude Code Incident
6. Hands-on: the mitigation decision tree
Description
The incident is already declared (lesson 4), with roles assigned. This lesson reconstructs the most technical part of the real 24 hours: the path of decisions that led from "all infrastructure destroyed, with no visible recovery path" to "1,943,200 rows restored, intact." It's not a linear list of steps — it's a real decision tree, with branches where a different response would have led down a different path — mapped, stage by stage, against the five lifecycle phases Module 5, lesson 2 already quoted from Google SRE.
Connection to the module
This module's lesson 2 built TIMELINE.md with the facts in chronological order. This lesson doesn't repeat those facts — it reorganizes them as a decision tree, the shape that truly matters when someone asks "what if something had been different?" The tree this lesson builds integrates, without changing any fact already established, into the final version of TIMELINE.md lesson 8 (this module's project) delivers.
Step 1 — Why a tree, and not just a list of steps
A chronological list — what TIMELINE.md already has — answers "what happened, in what order?" A decision tree answers a different, more useful question for learning from the case: "at exactly what point was there a real alternative, and what determined which path the response took?" This incident's mitigation wasn't the linear execution of a pre-existing plan — there was no runbook written for "recover an entire infrastructure after an accidental destroy" — it was a series of decisions under uncertainty, each conditioned on the previous one's outcome. Reconstructing those branches, not just the final sequence, is what a decision tree contributes that a chronological list can't.
Step 2 — The complete tree
MITIGATION DECISION TREE -- CLAUDE CODE DESTROY INCIDENT
T+0: destroy executes -- VPC, ECS, load balancers, bastion host,
RDS + automated snapshots, all destroyed
│
▼
Is there anything to revert with Terraform's own state?
│
┌───────────────┴───────────────┐
NO (not applicable: the state
│ itself was the cause,
│ and the infra no longer exists)
▼
Is there an automated RDS snapshot visible
in AWS's own console?
│
┌───────────────┴───────────────┐
NO -- the automated (automated snapshots
snapshots were also do NOT survive an
destroyed along with the DB unprotected RDS
│ destroy)
│
▼
DECISION: escalate beyond self-service
(T+~1h -- open a ticket, upgrade to
AWS Business Support, ~+10% monthly cost)
│
▼
Does AWS confirm any recovery mechanism
on their side?
│
┌───────────────┴───────────────┐
YES (T+~1h30min) -- an internal (if the answer had
snapshot, NOT visible in the been "no," there
customer console would have been no
│ recovery path at
│ all -- the worst
│ possible outcome
│ for this incident)
▼
DECISION: escalate the restoration
(T+~2h to T+~3h -- phone call,
internal escalation within AWS)
│
▼
Wait for AWS's internal team to
execute the restoration -- outside
DataTalks.Club's direct control
from this point forward
│
▼
T+24h00m: snapshot restored.
VERIFICATION: courses_answer, 1,943,200
rows, confirmed intact
│
▼
INCIDENT RESOLVEDStep 3 — The tree, mapped against Module 5's five stages
| Stage (Module 5, lesson 2) | Where it fits in this tree |
|---|---|
| Detection | Immediate and total from T+0 on — every branch of the tree starts after detection has already happened (this module's lesson 3). |
| Declaration | T+~5min — the formal declaration (this module's lesson 4) happens before the mitigation decision tree starts branching. |
| Response | The tree's first two nodes ("is there anything to revert?", "is there a snapshot visible in our own console?") — the phase where the team coordinates what own options exist, before escalating outside its direct control. |
| Mitigation | From the first escalation decision (T+~1h, opening a ticket with AWS) through the internal snapshot confirmation (T+~1h30min) and the phone escalation (T+~2h to T+~3h) — the active work of limiting the damage and finding a restoration path, without the full root cause yet resolved. |
| Resolution | T+24h00m — the snapshot restored and verified intact; the condition that triggered the incident (lost data, no known recovery path) is no longer active. |
Notice an honest detail this mapping reveals: this incident's "Mitigation" phase was, at no point, a technical action the team itself executed with its own hands — unlike Module 5, lesson 2's hypothetical example (rolling back a broken process-shipment-manifest deployment). This incident's entire real mitigation consisted of escalating — asking for help outside the team's direct control — and waiting, because its own tools (the state, the AWS console, any snapshot visible on the customer side) had already been completely exhausted in the tree's first two nodes.
Step 4 — The branch that truly mattered
Of the four decisions marked in Step 2's tree, only one determined whether this incident ended in a complete recovery or in a permanent loss of 2.5 years of data: AWS's confirmation, at T+~1h30min, that a snapshot existed on their side, invisible from the customer console. The other three decisions — revert with Terraform (no, not applicable), look for an own snapshot (no, destroyed), escalate to support (yes, decision made) — each had only one reasonable path, given the system's state. This one, in contrast, was genuinely uncertain until AWS responded: if the answer had been "no, there's no recoverable snapshot anywhere," the entire tree would have ended right there, with no path toward resolution.
This is, honestly, the most important point this tree reveals about the incident itself: DataTalks.Club's recovery depended on a snapshot retention mechanism on AWS's side that the affected person themselves didn't know existed, and that wasn't guaranteed by any explicit configuration they had turned on. It wasn't the result of a safeguard the team designed — those safeguards (deletion protection, explicitly managed snapshots) are precisely what Grigorev implemented after the incident, as part of his preventive measures. It was, to a large extent, operational good fortune combined with the persistence to escalate until finding it.
Common mistakes
Presenting this tree as if the path to resolution had been obvious or guaranteed from the start (losing the real uncertainty Step 4 describes). What happens: someone reads the whole tree, start to finish, and concludes "the recovery was always going to work," because in hindsight they know the outcome. How to spot it: if your description of the tree treats the "does AWS confirm a recovery mechanism?" node as a formality, instead of as the genuine point of greatest uncertainty. How to fix it: this lesson's Step 4 is explicit — until AWS responded, at T+~1h30min, there was no guarantee any recovery path existed. Treating the favorable outcome as "expected from the start" is a real retrospective bias (hindsight bias), exactly the kind of distortion a blameless postmortem (Module 7) must avoid when reconstructing any incident.
Confusing "mitigation" with "a technical action the team executes with its own hands," after seeing this incident break that pattern (over-generalizing from a single case). What happens: someone concludes, after this lesson, that "mitigating" always means escalating and waiting, instead of applying a direct technical change. How to spot it: if your general definition of "mitigation," after this lesson, no longer includes Module 5's standard example (rolling back a broken deployment). How to fix it: this lesson's Step 3 explicitly noted that this incident is a case where mitigation consisted of escalating and waiting — not because that's mitigation's general definition, but because the team's own tools had already been exhausted. Mitigating process-shipment-manifest with a broken deployment (Module 5, lesson 2's example) is indeed a direct technical action; both cases are mitigation, with completely different mechanisms depending on which options remain available.
Assuming that, because AWS had an internal snapshot not listed in the console, any similar future incident would have the same safety net (generalizing a specific circumstance as a guarantee). What happens: someone concludes "AWS always has an internal backup snapshot, so an accidental destroy is never really catastrophic." How to spot it: if your disaster recovery plan depends, implicitly or explicitly, on AWS having some undocumented internal mechanism that rescues the data. How to fix it: this lesson's Step 4 was explicit that this mechanism wasn't guaranteed by any explicit configuration. Relying on the possibility that a similar mechanism exists, without explicitly turning it on (deletion protection, versioned own backups, point-in-time recovery), is exactly the error the preventive measures Grigorev adopted after the incident — cited in this guide's Module 1 — exist to correct. This guide's own Module 7 (07-hands-on-the-honest-backup-restore-attempt.md) builds, with LocalStack, a real, explicit DynamoDB backup/restore attempt for Andes Cargo — precisely so as not to depend on the same luck.
Exercises
Exercise 1 — Using Step 2's tree, identify the node where the incident could have resulted in a permanent data loss, with no possibility of recovery. Explain what condition, had it been different, would have changed that outcome.
See solution
The critical node is "does AWS confirm any recovery mechanism on their side?" (T+~1h30min). If the answer had been "no" — if the internal snapshot not listed in the console simply hadn't existed, or if AWS's retention policies had already purged it by that point — the entire tree would have ended right there, with no remaining path toward resolution: there was no other pending decision node offering an alternative. The condition that determined the outcome wasn't any decision made by DataTalks.Club's team — it was the existence, outside their control and knowledge, of that specific internal AWS mechanism.
Exercise 2 — Explain, using Step 3, why this incident's "Response" phase ended so quickly — only two nodes of the tree — compared to the "Mitigation" phase.
See solution
The "Response" phase, per Module 5's definition, is about coordination — what own options exist and who evaluates them — and in this incident those options ran out almost immediately: there was no Terraform state to revert (the infrastructure no longer existed), and there was no snapshot visible in the own console (destroyed along with the database). With both own options ruled out within minutes, there was no additional internal coordination work left to do — the only remaining path was escalating outside the team's direct control, which is exactly where this tree's "Mitigation" phase begins. The Response phase's brevity doesn't reflect a disorganized team — it reflects how quickly the own options ran out in an incident of this magnitude.
Exercise 3 — A colleague argues this decision tree "isn't practically useful," because every node describes a decision that can't be repeated the same way (AWS might not have the same internal mechanism next time). How do you respond, using this lesson's Step 4 and last "Common mistakes" entry?
See solution
This tree's value isn't in its specific nodes (AWS's internal snapshot) being reliably repeatable — Step 4 and the last "Common mistakes" entry already established that specific mechanism wasn't guaranteed. The real value is in the decision structure it reveals: exhaust the verifiable own options first (Terraform, own console), escalate quickly and without delay once they're exhausted, and never assume outside help will arrive on time or in the way it did this time. That structure — exhaust, escalate, verify, never assume — is the reusable part, not the specific AWS mechanism that resolved this particular case. It's the same reason this guide's Module 7 builds an explicit backup/restore attempt for Andes Cargo, instead of assuming a mechanism similar to this incident's would be available if something like it happened again.
Summary and next step
This lesson reconstructed the complete decision tree of the Claude Code incident's real mitigation: two own options exhausted within minutes (revert with Terraform, look for a visible snapshot), the decision to escalate to AWS support (T+~1h), the critical, uncertain confirmation of an internal snapshot not listed in the console (T+~1h30min), the phone escalation (T+~2h to T+~3h), and the wait until verified resolution (T+24h00m). You mapped every node against Module 5's five stages, and confirmed an uncomfortable honesty: this incident's mitigation wasn't a technical action performed by the team — it was escalating and waiting, because its own tools had already been exhausted, and the favorable outcome depended on an internal AWS mechanism that nobody on DataTalks.Club's side knew existed.
Before moving on you should be able to: recite the complete tree, from memory, in order; identify the node of greatest real uncertainty; and explain why this incident isn't a good example of the standard direct-technical-mitigation pattern.
Lesson 7 returns to this guide's formal vocabulary: the same 33.3x calculation from Module 1, now run with a real error_budget_calculator.py — the complete SLI, SLO, and error budget vocabulary behind the number.
Resources
- Alexey Grigorev — How I Dropped Our Production Database — the primary source for every node of this tree.
- This same repository, Module 5, lesson 2 (
02-the-incident-lifecycle.md) — the five stages this tree is mapped against. - This same repository, Module 6, lesson 2 (
02-hands-on-reconstructing-the-exact-timeline.md) —TIMELINE.md, the chronological source this tree reorganizes as decisions. - Google SRE Book — Managing Incidents — the definition of mitigation quoted in Module 5, applied here to a case where mitigation meant escalating, not executing.