Module 1: What Is Sre And Reliability As A Feature
5. Two real incidents, two different questions
Description
If you came from terraform-and-iac-guide, cicd-and-gitops-on-aws-guide, or cloud-security-and-guardrails-guide, this is the fourth time you've run into the same incident: a Claude Code agent ran terraform destroy against DataTalks.Club's production infrastructure and deleted 2.5 years of data. This lesson doesn't narrate it from scratch again just to raise the alarm once more — three guides already did that, each with its own angle — it picks it back up with the question none of the three has asked yet, because none of the three had, until this guide, the vocabulary to ask it: what would an error budget have said about this? Alongside that case, this lesson names — without operating it in depth — a second real incident, of a completely different nature: the us-east-1 outage of October 2025, a failure that had nothing to do with a poorly approved command.
Connection to the module
Lesson 3 gave you the error budget vocabulary. Lesson 4 identified, in Andes Cargo's real inventory, that the entire system lives in a single region — exactly the kind of exposure this lesson's second case makes tangible. This lesson connects both pieces with the two real incidents that carry the rest of this guide, and leaves open the question lesson 6 answers with a real number.
The first case, in the exact figures you already know (recapped, not repeated)
Alexey Grigorev, founder of DataTalks.Club — the free-courses platform — was using Claude Code to migrate a static site into the platform's existing Terraform infrastructure, reusing the same project instead of creating a separate one, to save between five and ten dollars a month in duplicate infrastructure. The agent replaced the active terraform.tfstate with an old version, extracted from an archived .zip. With that stale state as a reference, it interpreted a large part of the real production infrastructure as orphaned resources, and ran terraform destroy -auto-approve. The result: the entire VPC, the ECS cluster, the load balancers, the bastion host, the RDS database and its automatic snapshots, all deleted. 2.5 years of data from the platform disappeared with a single command; the courses_answer table, one of the affected ones, had 1,943,200 rows. Recovery took close to 24 hours of work with AWS enterprise support, until they found an internal snapshot that wasn't even listed in the console. The detail you already know if you came from the sibling guides: Claude Code had flagged the risk of the destroy before running it, and the human in charge approved it anyway.
Exact date, verified against Grigorev's primary source: February 26, 2026. If you came from terraform-and-iac-guide or cicd-and-gitops-on-aws-guide, their original DISEÑO.md files mentioned "March 2026" — a discrepancy already corrected in those guides' written lessons, and in this one's design.
The new angle: what would an error budget have said about this?
terraform-and-iac-guide (Module 8, lesson 5) answered "what happened, and what rule keeps it from happening again?" — the discipline of reading the full plan. cloud-security-and-guardrails-guide (Module 1, lesson 6) answered "what automated control would have stopped it?" — the most direct answer: conftest evaluating the plan before the apply, blocking without relying on human attention. Neither question is this guide's, and it's worth saying precisely why: an error budget isn't a preventive control. It wouldn't have blocked the destroy — that function is served by conftest, not by an error budget calculator. What an error budget does, that no preventive control does, is turn damage that already happened into an immediate, objective number, without needing to reconstruct the whole story for someone to understand the magnitude.
Think of it this way: if DataTalks.Club had had, before the incident, a monthly SLO defined for its platform — any reasonable number, not necessarily 99.9% — and someone asked, the very same day of the incident, "how bad was this, in terms the business understands?", the answer with an error budget wouldn't be a narration of 24 hours of work with AWS support. It would be a multiple: "this consumed X times the whole month's downtime budget, in a single event." Lesson 6 of this module calculates that exact multiple, by hand, with the full arithmetic in view — and the result isn't "a bit more than expected." It's a number that, with no additional context needed, immediately communicates that this wasn't a bad day — it was an event that ate, many times over, an entire month's margin.
The honesty this section can't skip: an error budget measures after something happened. It's not a substitute for conftest, nor for reading the plan, nor for any of the preventive safeguards the sibling guides already built. It's a different instrument, answering a different question — not "how do we avoid it?" but "how bad was it, in a number anyone can understand without having lived through the 24 hours?" — and that question, until this guide, no module in this ecosystem had answered with real arithmetic.
The second case, named by contrast: the us-east-1 outage, October 2025
If the DataTalks.Club incident is a case of cause inside the system — a command approved without enough attention, over infrastructure the team itself controlled — this second case is the opposite category: a failure that occurred entirely outside the control of any team whose infrastructure lived in the affected region, with no conftest, no policy, no plan-review discipline able to have done anything about it.
On October 19 and 20, 2025, AWS's us-east-1 region — the same region where every Andes Cargo resource lives today, confirmed in lesson 4 of this module — suffered an outage of roughly 15 hours, impacting dozens of AWS services and thousands of applications depending on it across the entire internet. The root cause, according to verified technical coverage of the incident: a race condition in DynamoDB's automated DNS management system, between two internal components — a DNS Planner, which monitors the state of load balancers, and a DNS Enactor, which applies DNS changes through Route 53. At some point, while one Enactor was experiencing a delay, another Enactor began a cleanup process that removed a DNS plan that was actually still valid, wiping out all the IP addresses for DynamoDB's regional endpoint at once. That failure cascaded: services that internally depend on DynamoDB — including EC2, Lambda, ECS, EKS, and Fargate — also started failing, without any of those services having, on their own, any technical problem of their own.
TWO INCIDENTS, TWO COMPLETELY DIFFERENT ORIGINS
CAUSE INSIDE THE SYSTEM CAUSE OUTSIDE THE SYSTEM
(DataTalks.Club, Feb 2026) (us-east-1, Oct 2025)
An agent with legitimate access An internal race condition
runs a destructive command at AWS, in its own DNS
over infrastructure the automation for DynamoDB
team directly controls
│ │
▼ ▼
Preventable with discipline and NOT preventable by any
the team's own controls: reading client-side control —
the plan, `conftest`, the cause lives inside
`prevent_destroy` (Modules 2 AWS itself, not in
and 4 of `cloud-security-and- `andes-cargo-infra/`
guardrails-guide`)
│ │
▼ ▼
The right question: "which The right question: "what
controls of ours failed?" do we do when the provider
itself fails, and the
cause isn't ours at all?"Why this guide names this case, but doesn't operate it in depth like DataTalks.Club's. Three honest reasons, not an excuse: first, this case didn't happen to Andes Cargo or to any project in this ecosystem — it's a cited market case, not an incident with its own data verified by three sibling guides, the way the Claude Code incident is. Second, the real answer to "what do we do when all of us-east-1 fails?" is multi-region architecture — active failover between regions, real-time data replicas — something that is, by design, outside the $0, single-region scope of this guide and of this entire ecosystem. Third, and most important: the discipline this case demands — SLI, SLO, error budget, an incident lifecycle with roles and communication — is exactly the same one this guide already builds with the DataTalks.Club case. The difference isn't in the response tools, it's in the root cause, and this second case's root cause simply doesn't admit a $0 solution.
src/paths/aws-cloud-ecosystem/VALIDACION.md is precise about why this case matters for the market, beyond whether Andes Cargo can operate it in depth: "the mandatory case study already exists: the us-east-1 outage (Oct 2025, 2,057 HN comments) was a DNS race condition between the Planner and the Enactor, with no versioning or compare-and-swap — not a capacity problem". It is, in practice, the second reference incident any serious SRE conversation in 2026 needs to be able to name with technical precision, not just as "AWS went down that day."
Common mistakes
Looking, in the second case, for an operational lesson this guide doesn't build (expectation mismatch). What happens: someone finishes this lesson expecting a later module to teach configuring multi-region failover for Andes Cargo, or some mechanism that would have protected against the us-east-1 outage. How to spot it: if you look, in the rest of this guide, for some cross-region replication HCL resource. How to fix it: this case is named, with technical precision, so you can recognize and explain it in a real conversation — an interview, an architecture discussion — but this guide, honest about its $0 scope declared since lesson 1, doesn't build multi-region DR. It's exactly the same discipline finops-and-cost-guardrails-guide applied to Cost Explorer: name a real limit with precision, don't fake a solution the guide's scope doesn't allow.
Concluding an error budget would have prevented the DataTalks.Club incident (confusing the function). What happens: someone, after reading this lesson's new-angle section, believes that if DataTalks.Club had had an error budget calculated beforehand, the destroy would never have run. How to spot it: if your summary of this lesson is "an error budget would have prevented this." How to fix it: this lesson was explicit about the opposite point — an error budget measures, it doesn't prevent. This ecosystem's preventive piece already exists, and lives in cloud-security-and-guardrails-guide (conftest, prevent_destroy). What an error budget contributes, that those preventive controls don't, is the ability to quantify the damage after something happened, with an immediate number — a different instrument, not a better version of the same one.
Treating the two cases as if they had the same root cause just because both affect availability (over-simplification). What happens: someone lumps both incidents under the generic label "things that can go wrong on AWS," without distinguishing that one has a completely controllable cause (a poorly approved human/agent command) and the other a cause completely outside the customer's control (an internal AWS failure). How to spot it: if your explanation of "what to do so it doesn't happen again" is the same for both cases. How to fix it: this lesson's diagram exists exactly for this distinction — the right question facing the first case is "which controls of ours failed?"; facing the second, it's "what do we do when the provider itself fails, and the cause isn't ours at all?" These are different questions, with different engineering answers, even though both end up measured with the same SLI/SLO/error budget vocabulary.
Exercises
Exercise 1 — Explain, without using the word "prevent," what an error budget contributes to the DataTalks.Club case. A teammate asks what the point is of calculating an error budget over an incident that already happened, if it can't undo the damage either way. Answer with the real function this lesson established.
See solution
An error budget, applied after an incident, turns a long, complex narration (24 hours of work with AWS support, a corrupted state, a snapshot not listed in the console) into a single, objective, comparable number: how many times the full monthly downtime budget was consumed in that one event. That number lets anyone — technical or not — understand the incident's real magnitude without needing to reconstruct the whole story, and gives the team an objective basis for deciding how urgent it is to invest in additional safeguards. It doesn't undo the damage or prevent it from happening again — that's the job of conftest and preventive policies — but it measures the damage with a precision no qualitative narration matches.
Exercise 2 — Classify a hypothetical incident using this lesson's diagram. If an Andes Cargo engineer, by mistake, ran awslocal dynamodb delete-table --table-name Shipments in a real production environment (not LocalStack) with no additional confirmation, does that incident belong to the "cause inside the system" or "cause outside the system" category? Justify with the diagram's criterion.
See solution
Cause inside the system, same as the DataTalks.Club case. The diagram's criterion isn't "was a destructive command involved?" — both of this lesson's cases have that at some point — it's "is the root cause within the team's direct control, or does it depend on AWS's own internal infrastructure?" A delete-table run by an Andes Cargo engineer (or agent), with no safeguard in place, is a failure of the team's own discipline — exactly the kind of failure conftest, reading the plan, or deletion protection could have stopped — not an AWS infrastructure failure outside anyone's control.
Exercise 3 — Explain why this guide names the us-east-1 outage instead of ignoring it, despite not being able to operate it with this ecosystem's $0 tools. Using this lesson's VALIDACION.md quote, argue why omitting this case would be a worse design decision than naming it with honesty about its scope.
See solution
VALIDACION.md is explicit: this outage is "the mandatory case study" that none of the 18 competing syllabi analyzed cover with technical precision — most treat it, when they mention it at all, as "AWS went down that day," without naming the real root cause (the DNS race condition between the Planner and the Enactor). Omitting it entirely would leave whoever completes this guide unable to take part, with technical credibility, in a market conversation where this case is already a standard reference. Naming it with precision — the exact cause, the duration, the scope — without pretending this guide builds a multi-region solution outside its declared $0 scope, is the same honesty discipline that carries every representative case in this ecosystem: a precisely declared limit beats a silent omission or a promise the guide can't keep.
Summary and next step
In this lesson you picked the Claude Code destroy incident back up with an angle none of the three sibling guides used yet: what an error budget contributes — not as a preventive control, but as an instrument for immediate, objective measurement of damage that already happened. You named, by contrast, 2026's second mandatory reference case for the market: the us-east-1 outage of October 2025, caused by a DNS race condition between DynamoDB's DNS Planner and DNS Enactor, with ~15 hours of impact — a case with a root cause completely different from DataTalks.Club's, cited with technical precision but not operated in depth, for this lesson's three honest reasons.
Before moving on you should be able to: explain what an error budget contributes to the DataTalks.Club case without saying it "prevents" it; name the exact technical root cause of the us-east-1 outage (the DNS race condition); and distinguish, with this lesson's diagram, when an incident has a cause inside the system and when it has one outside it.
Lesson 6 does what this lesson only promised: the real calculation, by hand, of how many times the monthly error budget was consumed in the 24 hours it took to recover from the Claude Code incident.
Resources
- Alexey Grigorev — How I Dropped Our Production Database — the firsthand account from the person it happened to; the primary source for the exact figures and the date (February 26, 2026), verified for this lesson.
- Hacker News — original Claude Code incident thread (#47278720) — the public discussion that brought the case to light.
- incidentdatabase.ai — Incident 1424 — the independent, structured record of the Claude Code incident.
- The Register — A single DNS race condition brought AWS to its knees — verified technical coverage of the
us-east-1outage, root cause and scope, cited in this lesson. src/paths/aws-cloud-ecosystem/VALIDACION.md— the source of the quote about theus-east-1outage as a mandatory market case study.terraform-and-iac-guide, Module 8, lesson 5;cloud-security-and-guardrails-guide, Module 1, lesson 6 — this ecosystem's two previous narrations of the Claude Code incident, which this lesson doesn't repeat but reframes with error budget vocabulary.