Module 6: Operating The Claude Code Incident
2. Hands-on: reconstructing the exact timeline
Description
Every incident response document starts in the same place: an ordered sequence of facts, with timestamps, that anyone new can read and understand without needing to ask anyone what happened first. This lesson builds that sequence for the Claude Code incident, in incidents/2026-02-26-claude-code-destroy/TIMELINE.md — this module's central document, first version, with every fact verified against the primary source and, wherever the public record doesn't give an exact clock time, a relative timestamp (T+0, T+~5min) instead of an invented one.
Connection to the module
Lesson 1 promised this module produces real operational documents, not another narrative. This is the first one: a formal timeline that this module's lessons 3 through 6 are going to enrich, not rewrite — lesson 3's severity, lesson 4's roles, lesson 6's decision tree all get added on top of this same timeline, without changing a single fact already established here. Lesson 8, the module's final project, delivers this same file's complete, integrated version.
Step 1 — Why relative timestamps, and not the incident's real clock
An honesty worth resolving before writing a single line of the timeline: this incident's primary source — Alexey Grigorev's first-hand account — does include approximate time references ("around 10 PM," "close to midnight"). It would be technically possible to reconstruct a timeline with those approximate times converted into exact clock marks. This lesson, deliberately, doesn't do that, for two concrete reasons.
First reason: the source's times are approximate, not exact — the account itself uses "around" and "close to," never a minute-precise timestamp. Converting "around 10 PM" into an exact timestamp like 22:00:00 would fabricate a precision the source itself never had, exactly the kind of honesty this entire ecosystem has demanded since its first guide: never present as literal something that's actually an approximation.
Second, more important reason: what this timeline needs to communicate isn't "what clock time did each thing happen at" — it's "how much time passed between each stage." A real incident timeline, the kind an Incident Commander maintains while the incident is active, is almost always read and used in relative terms — "we've gone 40 minutes with no response from the vendor," not "it's 11:40 PM" — because the central operational question during an active response is never the clock time, it's how much time has passed since the incident began. This lesson adopts that same convention: T+0 marks the moment terraform destroy -auto-approve executes — the event that causes the data loss, the clearest, most verifiable reference point in the entire incident — and every other fact is marked with its offset relative to that point, with the ~ symbol wherever the exact offset isn't verified to minute precision.
Step 2 — The facts, verified against the primary source
Before building the timeline, the complete list of facts feeding it, each traceable to a cited source (already used in this guide's Module 1, verified again for this lesson):
| Fact | Source |
|---|---|
The agent replaced the active terraform.tfstate with an old version, extracted from an archived .zip | Grigorev, primary account |
terraform plan showed a long list of resources as "to create," not "to modify" — the first sign something was wrong | Grigorev: "My first clue that something was off was when I saw a long list of resources being created." |
| The human asked why; the agent explained that, according to the state it had loaded, the infrastructure "didn't exist" | Grigorev: Claude responded that "Terraform believed nothing existed" |
The apply stopped, but some resources had already been partially created before it stopped | Grigorev, primary account |
The agent proposed terraform destroy as the "cleanest" way to reverse what had already been created | Grigorev: "I cannot do it. I will do a terraform destroy. Since the resources were created through Terraform, destroying them through Terraform would be cleaner and simpler than through AWS CLI." |
| The human did not stop the agent | Grigorev: "So I didn't stop the agent from running terraform destroy." |
terraform destroy -auto-approve ran, destroying the VPC, the ECS cluster, load balancers, the bastion host, the RDS database, and all its automated snapshots | Grigorev, primary account; corroborated by Tom's Hardware and incidentdatabase.ai #1424 |
The courses_answer table (1,943,200 rows, 2.5 years of data) disappeared along with the rest of the infrastructure | Grigorev, primary account |
| The author opened an AWS support ticket, upgrading to Business Support (~10% more in monthly cost) | Grigorev: "Around midnight, I opened a support ticket... I upgraded, which added about 10% to my cloud costs." |
| AWS support confirmed a snapshot existed in their own systems, not visible in the customer console | Grigorev: "They found a snapshot on their end that I couldn't see in my console." |
| There was a phone call and an internal escalation within AWS to manage the restoration | Grigorev, primary account |
| Exactly 24 hours after the database was deleted, AWS restored the snapshot | Grigorev, verbatim quote: "Exactly 24 hours after the database had been deleted, AWS restored the snapshot." |
| The author published the complete account publicly on March 6, 2026 — about eight days after the incident | Article's publish date, verified |
Step 3 — The document
At the root of andes-cargo-infra/, create the directory incidents/2026-02-26-claude-code-destroy/ and, inside it, TIMELINE.md:
# TIMELINE.md — 2026-02-26 Claude Code `destroy` Incident
**Status:** Draft (this document) -> completed in Module 6, lesson 8
**Real incident, external:** happened to DataTalks.Club, not to Andes Cargo. Operated here as a
drill against `INCIDENT-RESPONSE-PLAN.md` (Module 5) -- see Module 6, lesson 1.
**Sources:** Alexey Grigorev, "How I Dropped Our Production Database" (alexeyondata.substack.com);
Hacker News #47278720; incidentdatabase.ai #1424.
## A note on timestamps
All timestamps in this document are **relative to T+0**, the moment `terraform destroy
-auto-approve` executed -- the clearest, most verifiable reference point in the entire incident.
The primary source gives only approximate clock times ("around 10 PM", "around midnight"); this
document never converts those into exact clock timestamps, which would fabricate a precision the
source itself does not have. Offsets marked `~` are inferred from the sequence the source
describes, not measured to the minute. The one offset that IS precise, stated verbatim by the
primary source: exactly 24 hours between T+0 and resolution.
## Timeline
| Offset | Event | Source |
|---|---|---|
| T-~1h | Deployment work begins: migrating a static site into the existing Terraform project (reusing it instead of a separate one, to save $5-10/month), without the correct state file -- it was on an old computer, never loaded | Grigorev |
| T-~15min | `terraform plan` output shows a long list of resources marked for **creation**, not modification -- first anomaly noticed | Grigorev: "My first clue that something was off..." |
| T-~10min | Human asks why; agent explains the loaded state file made it believe "nothing existed" | Grigorev: "Terraform believed nothing existed" |
| T-~10min | `apply` stopped -- but some resources had already been partially created before the stop | Grigorev |
| T-~5min | Agent proposes `terraform destroy` as the cleanest way to reverse what it had just created | Grigorev, verbatim quote in TIMELINE.md sources |
| T-~1min | Human does not stop the agent from proceeding | Grigorev: "So I didn't stop the agent..." |
| **T+0** | **`terraform destroy -auto-approve` executes.** VPC, ECS cluster, load balancers, bastion host, RDS database, and all automated snapshots -- destroyed. `courses_answer` (1,943,200 rows, 2.5 years of data) gone. | Grigorev; Tom's Hardware; incidentdatabase.ai #1424 |
| T+0 to T+~5min | Total outage, immediately and unambiguously visible -- every service down at once | Inferred directly from the scope of what T+0 destroyed |
| T+~1h | Support ticket opened with AWS; upgrade to Business Support tier (~+10% monthly cost) | Grigorev: "Around midnight, I opened a support ticket..." |
| T+~1h30min | AWS support confirms a snapshot exists on their end -- not visible in the customer console | Grigorev: "They found a snapshot on their end that I couldn't see in my console." |
| T+~2h to T+~3h | Phone call with AWS support; internal escalation within AWS for manual restoration | Grigorev |
| **T+24h00m** | **AWS restores the snapshot.** `courses_answer` restored intact, 1,943,200 rows confirmed. | Grigorev, verbatim: "Exactly 24 hours after the database had been deleted, AWS restored the snapshot." |
| (8 days after T+0) | Grigorev publishes the full incident writeup publicly | Publish date verified: Mar 06, 2026 |
## What this version does not include yet
Severity classification (Module 6, lesson 3), the declaration message and assigned roles (Module
6, lesson 4), and the mitigation decision tree (Module 6, lesson 6) are not part of this draft --
they are added, without changing any fact already established here, in the final version this
module's project (lesson 8) produces.Step 4 — Reading the timeline: what a single minute reveals
Look at one specific row: T+0 to T+~5min, with the note "total outage, immediately and unambiguously visible — every service down at once." This row doesn't come from a verbatim quote with an exact time — it comes from a direct inference, explicitly marked as such, about the scope of what T+0 destroyed. If the entire VPC, the ECS cluster, and the load balancers disappear in the same command, no additional data is needed to know the impact was immediate and total: it's a direct logical consequence of what got destroyed, not an independent measurement. Precisely distinguishing between "this is a verbatim cited fact" and "this is a reasonable inference from cited facts" is exactly the same discipline this entire ecosystem has demanded since cloud-security-and-guardrails-guide: never present an inference as if it were a quote.
Common mistakes
Converting the source's approximate times ("around 10 PM") into exact clock timestamps, "to make it look more professional" (fabricating false precision). What happens: someone, when copying this document, replaces T-~1h with an invented clock time like 21:00, assuming a "real" timeline needs exact times. How to spot it: if your version of TIMELINE.md includes any clock timestamp (HH:MM) in any row. How to fix it: this lesson's Step 1 explained the exact reason — the primary source never gives an exact time, only approximations ("around"), and converting an approximation into an exact number fabricates a precision that doesn't exist. An honest timeline uses ~ wherever the source is approximate, and never dresses up that approximation with a number that looks more precise than it really is.
Treating every ~ offset in this document as if it had the same certainty as T+24h00m (losing the confidence hierarchy between facts). What happens: someone cites "T+~1h: support ticket opened" with the same confidence as "T+24h00m: snapshot restored," without noticing the first is an approximate inference and the second is a verified verbatim quote. How to spot it: if your summary of the incident doesn't distinguish between facts with an exact verbatim quote and facts with an approximate offset. How to fix it: the document itself marks this difference with the ~ symbol — two rows in this timeline, T+0 and T+24h00m, are the only ones with a source giving an exact, verified duration (the command that ran, and the verbatim "exactly 24 hours" quote); everything else is a reasonable reconstruction, not an exact measurement, and the document flags it that way on purpose.
Rewriting the order of facts so the story "flows better," without verifying the reconstructed order matches the source (narrating instead of reconstructing). What happens: someone reorganizes the timeline so, for example, Claude Code's warning appears right before the destroy, without checking against the source whether that was really the exact sequence. How to spot it: if your timeline can't point, row by row, to what quote or reasonable inference backs that specific order. How to fix it: every row in Step 3's table has a "Source" column — before moving any row, confirm the order you're building follows the real sequence Grigorev describes, not a version reordered for narrative convenience. A reconstructed timeline's value depends entirely on it being traceable to the source, not on it reading well.
Exercises
Exercise 1 — Using only Step 2's fact table, calculate approximately how many minutes passed between the first noticed anomaly (terraform plan showing creations) and the destroy's execution. Explain why your answer must carry the ~ symbol.
See solution
Per Step 3's timeline, the first anomaly is marked at T-~15min and the destroy at T+0 — an approximate difference of 15 minutes. The answer must carry ~ because neither point has an exact clock timestamp in the primary source: both are reasonable inferences from the sequence Grigorev narrates, not direct measurements. Presenting "exactly 15 minutes" without the ~ symbol would imply a precision that neither the source nor this reconstruction actually has — exactly the error this lesson's first "Common mistakes" entry warns about.
Exercise 2 — Explain why T+0 is defined as the moment terraform destroy -auto-approve executes, and not, say, the moment the human decided "not to stop the agent." What makes the first point a better reference than the second?
See solution
terraform destroy -auto-approve is a discrete, verifiable technical event: a specific command that ran, with an observable, documented result (the destroyed infrastructure). "The moment the human decided not to stop the agent," in contrast, is an internal event, with no technical artifact marking it precisely — there's no log, no system timestamp, capturing exactly when that decision ended and the command's execution began. Using the verifiable technical event as T+0 gives the rest of the timeline a solid reference point, the same kind a real system would use (the moment a destructive command ran, not the moment someone "decided" something, which rarely leaves a technical trace).
Exercise 3 — A classmate argues this timeline "isn't useful for anything real" because it has no exact clock times, and that a real Incident Commander would need to know the exact time to coordinate with other teams. Do you agree? Use this lesson's Step 1 argument.
See solution
Disagree, with an important nuance. An Incident Commander responding to an ongoing incident, in real time, does need to coordinate with clock time (to agree on when to make the next call, for example) — but that's not this document's case: this TIMELINE.md is built after the incident, as a historical reconstruction for learning and portfolio purposes, not as the live coordination document an Incident Commander would maintain while the incident is happening. For that retrospective purpose, what matters isn't the exact clock time — which, this lesson already established, the primary source never gave precisely anyway — but the correct relative sequence and the verified total duration (T+24h00m, the whole document's most important figure). A retrospective timeline honest about what it knows with certainty is more useful than one that fakes a clock precision that never existed.
Summary and next step
In this lesson you built the first version of incidents/2026-02-26-claude-code-destroy/TIMELINE.md: fourteen facts verified against Grigorev's primary source, corroborated by Hacker News and incidentdatabase.ai #1424, ordered with timestamps relative to T+0 — the moment terraform destroy -auto-approve executed — instead of any real clock time, which the source itself never gave precisely. You confirmed the entire incident's one genuinely exact reference point: exactly 24 hours between the data loss and the complete restoration.
Before moving on you should be able to: explain why this timeline uses T+0 instead of a real clock; distinguish, in Step 3's table, which rows are verbatim quotes and which are reasonable inferences; and recite from memory the complete sequence, from T-~1h to T+24h00m.
Lesson 3 takes this same timeline and asks it the question INCIDENT-RESPONSE-PLAN.md's severity matrix (Module 5) already knows how to answer: how serious was this?
Resources
- Alexey Grigorev — How I Dropped Our Production Database — the primary source for every fact in this lesson.
- Hacker News — the incident's original thread (#47278720) — independent public corroboration.
- incidentdatabase.ai — Incident 1424 — an independent structured record, already cited in this guide's Module 1.
- Google SRE — Incident Management Guide — the phase framework (prepare, respond, learn) this timeline starts populating.
- This same repository, Module 1, lesson 5 (
05-two-real-incidents-two-different-questions.md) — the incident's original narration this lesson reformats, without repeating, as a formal timeline.