Module 1: What Is Sre And Reliability As A Feature

4. Hands-on: reading Andes Cargo like an SRE would

Description

cloud-security-and-guardrails-guide already walked through andes-cargo-infra/ asking "is this dangerous?" finops-and-cost-guardrails-guide walked the same ground asking "does this generate a charge every month?" This lesson makes the third pass, with a question different from the previous two that no guide in this ecosystem has asked yet: what can fail here, and who would find out that it failed? This isn't a security audit — that's already done — nor a cost inventory — also done. It's Andes Cargo's first reading through an SRE lens: every resource, every configuration, evaluated for its exposure to failure, not for its vulnerability or its price.

This lesson has two parts with two different levels of certainty, with the same honesty that carries the rest of this ecosystem. Reading the real configuration — the exact values for concurrency, timeout, capacity mode, region — is completely literal: these are the numbers the previous guides left the system with. Confirming those numbers with awslocal is representative: this writing environment doesn't have LOCALSTACK_AUTH_TOKEN exported, so the LocalStack container doesn't start. The output you'll read is what those commands would produce against the infrastructure already applied, reconstructed field by field from what aws-core-services-guide and aws-serverless-and-containers-guide already confirmed running for real — never invented.

Connection to the module

Lesson 3 gave you the error budget vocabulary. Before you can define a real SLI (Module 2) or instrument observability (Module 3), you need to know which parts of the system can fail — without that knowledge, any SLI you pick would be a guess. This lesson builds that risk inventory. Lessons 5 and 6 are going to show, with the Claude Code incident, what happens when exactly this kind of question wasn't asked in time.

Check 1 — The Lambda function: `get-function`, read for exposure to failure

awslocal lambda get-function --function-name process-shipment-manifest

What to expect (representative — reconstructed field by field from the real state aws-serverless-and-containers-guide, Module 2, project, already confirmed running; CodeSha256, LastModified, and RevisionId are your variable value, the rest is literal):

{
    "Concurrency": {
        "ReservedConcurrentExecutions": 5
    },
    "Configuration": {
        "FunctionName": "process-shipment-manifest",
        "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:process-shipment-manifest",
        "Runtime": "python3.13",
        "Role": "arn:aws:iam::000000000000:role/LambdaManifestProcessorRole",
        "Handler": "handler.lambda_handler",
        "Timeout": 10,
        "MemorySize": 128,
        "Environment": {
            "Variables": {
                "CARRIER_API_KEY_SECRET_NAME": "andes-cargo/carrier-api-key"
            }
        },
        "Layers": [
            {
                "Arn": "arn:aws:lambda:us-east-1:000000000000:layer:shipment-utils-layer:1"
            }
        ],
        "State": "Active",
        "LastUpdateStatus": "Successful"
    }
}

A security audit reads this JSON asking "does the role have excess permissions?" A cost audit reads "how much does each GB-second cost at 128 MB?" An SRE reading asks three different questions:

ReservedConcurrentExecutions: 5 — what happens on the sixth simultaneous invocation? This is a real design decision from aws-serverless-and-containers-guide, meant to protect both the function's availability and an external carrier API from a traffic spike. But a concurrency ceiling is, at the same time, an availability limit: if Andes Cargo ever receives six manifests at the same instant, the sixth one gets throttled — it doesn't fail with a code error, it fails with a capacity error, a completely different kind of failure that needs its own signal to be detected.
Timeout: 10 — what happens if a manifest takes more than 10 seconds to process? The ceiling protects against a function hanging indefinitely, but it's also a hard limit: if some future manifest, bigger or more complex, needs 11 seconds, the invocation gets force-killed, no matter how close it was to finishing.
The S3 trigger invokes this function asynchronously — where does a failed event go? The official AWS Lambda documentation is precise on this exact point: "By default, Lambda retries a failed asynchronous invocation up to two times." — two automatic retries, with nobody having to configure anything. But after the second failed retry, without a dead-letter queue configured specifically for this asynchronous invocation, the event simply gets discarded. None of this JSON's fields confirms whether that queue exists for this specific invocation path — it's exactly the kind of question this lesson leaves open, and that Module 3's observability will be able to answer with evidence, not inference.

Check 2 — The DynamoDB table: `describe-table`, read for exposure to failure

awslocal dynamodb describe-table --table-name Shipments

What to expect (representative; ItemCount and TableSizeBytes are your variable value — and, as you already confirmed in aws-serverless-and-containers-guide, ItemCount isn't even a real-time count — the rest is literal):

{
    "Table": {
        "TableName": "Shipments",
        "KeySchema": [
            {
                "AttributeName": "shipmentId",
                "KeyType": "HASH"
            }
        ],
        "TableStatus": "ACTIVE",
        "ItemCount": 6,
        "TableSizeBytes": 912,
        "TableArn": "arn:aws:dynamodb:us-east-1:000000000000:table/Shipments",
        "BillingModeSummary": {
            "BillingMode": "PAY_PER_REQUEST"
        }
    }
}

The SRE question about this same JSON: PAY_PER_REQUEST means DynamoDB scales automatically, with nobody reserving capacity — but "automatic" doesn't mean "no limit or condition." The official DynamoDB documentation is specific about exactly when that autopilot stops keeping up:

"On-demand capacity mode instantly accommodates up to double the previous peak traffic on a table. [...] However, throttling can occur if you exceed double your previous peak within 30 minutes."

— Amazon DynamoDB Developer Guide — On-demand capacity mode

Translated to Andes Cargo's case: if Shipments' normal traffic runs around, say, 10 writes per second at its historical peak, the table can instantly absorb up to 20 writes per second with no problem at all. But if an unexpected event — a promotion, a data migration, a poorly designed mass retry — pushes traffic to 50 writes per second within minutes, more than double the previous peak, in less than 30 minutes, the table can start throttling requests, returning capacity errors despite being in on-demand mode. No resource in andes-cargo-infra/, today, measures whether this has ever happened, nor would alert if it happened again — another question this lesson leaves identified, not resolved.

Check 3 — The S3 bucket: `get-bucket-location`, read for exposure to failure

awslocal s3api get-bucket-location --bucket andes-cargo-shipment-docs

What to expect (representative — literal: for a bucket created in us-east-1, the official AWS documentation is explicit that the correct value is null, not the string "us-east-1"):

{
    "LocationConstraint": null
}

This is the smallest result of this lesson's three commands, and the SRE question that follows from it is, at the same time, the biggest one in this whole module: all of Andes Cargo — the bucket, the function, the table — lives in a single region. No resource replicates to another region, no failover mechanism exists if all of us-east-1 stops responding. This isn't a design flaw in the previous guides — a multi-region architecture for an early-stage shipment-tracking system would be, following exactly lesson 3's logic in this module, a completely unjustified engineering cost against the real risk — but it's a fact an SRE needs to have precisely named, not discovered by surprise. Lesson 5 of this module is going to name, with real numbers, exactly what happens when all of us-east-1 has a bad day.

The complete risk inventory: what was identified, and who resolves it

Resource	What the command confirms	The SRE risk it exposes	Who measures/resolves it in this guide
`process-shipment-manifest`	`ReservedConcurrentExecutions: 5`	A sixth simultaneous invocation gets throttled — with no alert today	Module 3 (`Throttles` metrics), Module 4 (alert)
`process-shipment-manifest`	`Timeout: 10`	An invocation exceeding 10s gets force-killed	Module 3 (real duration, measured)
`process-shipment-manifest` (S3 trigger, asynchronous)	Automatic retry ×2, no confirmed DLQ on this path	An event that fails twice is silently lost, with no evidence	Module 3 (logs), Module 7 (runbook)
`Shipments`	`BillingModeSummary.BillingMode: PAY_PER_REQUEST`	Possible throttling if traffic exceeds double the previous peak in under 30 minutes	Module 2 (an availability SLI would measure it)
`Shipments`	No confirmed `deletion protection`	Nothing today prevents an accidental `delete-table` or `destroy` — the exact topic of the incident in lessons 5 and 6	Module 7 (backup/restore attempt)
`andes-cargo-shipment-docs` (via the function)	`LocationConstraint: null` (`us-east-1`)	A single region, no failover — the same kind of risk the `us-east-1` outage (lesson 5) made real for half the internet	Named, not resolved — outside this guide's $0 scope

No row in this table is a security vulnerability — that inventory was already done by cloud-security-and-guardrails-guide — nor a cost driver — that one was already done by finops-and-cost-guardrails-guide. Every row is, specifically, a question about what happens when something fails and who finds out. That is, in one table, the real difference between the three readings this ecosystem has now done of the same system.

Common mistakes

Repeating the security or cost audit under another name (overlap). What happens: someone, doing this reading, ends up writing down findings like "the bucket doesn't have public access block" or "the table on PAY_PER_REQUEST could get expensive" — valid findings, but ones that already belong to the other two guides. How to spot it: if this lesson's inventory mentions security or cost, instead of availability. How to fix it: this lesson's single question is "what happens when this fails, and who finds out?" — never "is this insecure?" nor "is this expensive?" If a finding doesn't have a clear answer to "how would the failure be detected," it doesn't belong in this inventory.

Concluding a concurrency ceiling of 5 is "too low" or "misconfigured" with no context (intuition without data). What happens: someone sees ReservedConcurrentExecutions: 5 and assumes it's an arbitrarily low number, with no evidence at all of Andes Cargo's real traffic. How to spot it: if your conclusion is "they should raise that number" without having measured how many simultaneous invocations actually occur. How to fix it: this lesson identifies the risk — what happens on the sixth invocation — it doesn't resolve or judge it. aws-serverless-and-containers-guide already justified that number with an explicit design reason (protecting both the function and an external carrier API). This guide's Module 3 is the one that's going to bring real data on how many simultaneous invocations actually occur, before anyone decides whether 5 is the right number.

Treating this reading as the end of the work, not the start (expectation mismatch). What happens: someone finishes this lesson thinking they already "covered" Andes Cargo's reliability, because they identified the risks. How to spot it: if you can't name, for every row in this lesson's risk table, which specific module of this guide measures or resolves it. How to fix it: this lesson is exactly what lesson 4 of finops-and-cost-guardrails-guide was for cost, or lesson 4 of cloud-security-and-guardrails-guide was for security — an inventory, not a solution. This lesson's table exists, specifically, as a map toward the rest of the guide: every row has a destination module, no row closes here.

Exercises

Exercise 1 — Classify a hypothetical finding. If you noticed that andes-cargo-app-server (the EC2 instance inherited from aws-core-services-guide) has no CloudWatch alarm configured on its health status, does that finding belong to this lesson (SRE), to cloud-security-and-guardrails-guide (security), or to finops-and-cost-guardrails-guide (cost)? Justify.

See solution

It belongs to this lesson (SRE). The absence of a health alarm exposes no security risk (it doesn't open unauthorized access) nor cost risk (it generates no additional or avoidable charge) — it exposes exactly the kind of risk this lesson looks for: if the instance stops responding, nobody finds out until a human notices by accident. It's the same pattern as this lesson's table rows: "what happens when it fails, and who finds out?" is an availability question, SRE's exclusive territory.

Exercise 2 — Explain why ReservedConcurrentExecutions: 5 isn't, by itself, either good or bad. A teammate claims any reserved concurrency ceiling is "a bad practice" because it limits the system's availability. Do you agree?

See solution

Disagree, with nuance. A concurrency ceiling does introduce an availability limit — the sixth simultaneous invocation gets throttled, as this lesson identified — but it also protects against the opposite scenario: with no ceiling at all, an uncontrolled traffic spike on process-shipment-manifest could consume all the account's available concurrency, affecting other Lambda functions sharing the same account and region. aws-serverless-and-containers-guide documented this exact reason when setting the number to 5: protecting both this function's availability and an external carrier API's from a spike. The right question isn't "should a ceiling exist?" — it almost always should — but "does this specific number reflect the real expected traffic, measured with data, not assumed?" — exactly the question this guide's Module 3 will be able to answer with evidence.

Exercise 3 — Design the SRE question for a new resource, without running anything. If Andes Cargo added an SQS queue tomorrow between S3 and Lambda (to decouple manifest upload from processing), what SRE question — following this lesson's pattern, not security's or cost's — would you ask about that new queue?

See solution

Following the same pattern as this lesson's three checks, the right question would be about the redrive policy and the queue's visibility timeout: how many times is a message Lambda can't process retried before it moves to a dead-letter queue? and if that dead-letter queue exists, is anything or anyone watching it, or do messages pile up there with nobody finding out? The security question would be about permissions to access the queue; the cost one, about the price per million requests. The SRE one, following this lesson's exact pattern, is always the same shape: "what happens when this fails, and who finds out?"

Summary and next step

In this lesson you read three central Andes Cargo resources — the process-shipment-manifest function, the Shipments table, the andes-cargo-shipment-docs bucket — with a question no previous guide in this ecosystem asked: not security, not cost, but exposure to failure. You identified six concrete risks — a concurrency ceiling with no alert, a timeout with no real-duration metric, asynchronous retries with no confirmed dead-letter queue, a capacity mode with a specific throttling condition, no confirmed deletion protection, and a single region with no failover — and located, for each one, exactly which module of this guide is going to measure or resolve it.

Before moving on you should be able to: explain the single question that distinguishes this reading from security's and cost's; name the six identified risks and in which exact JSON field of each command they were discovered; and explain why this lesson is an inventory, not a solution.

Lessons 5 and 6 bring the case that makes this inventory stop feeling theoretical: the Claude Code destroy incident, and the first time, in this whole guide, it gets measured with a real number.

Resources

AWS Lambda Developer Guide — Understanding retry behavior in Lambda — the exact source for the two default automatic retries on asynchronous invocation, cited in Check 1.
Amazon DynamoDB Developer Guide — On-demand capacity mode — the exact source for the throttling condition at double the previous peak within 30 minutes, cited in Check 2.
AWS CLI — s3api get-bucket-location Command Reference — the exact source for the null value for buckets in us-east-1, cited in Check 3.
aws-serverless-and-containers-guide, Module 2, project — the real, verified state of process-shipment-manifest (concurrency, layer, alias) this lesson reconstructs.
cloud-security-and-guardrails-guide, Module 1 and finops-and-cost-guardrails-guide, Module 1 — the two previous readings of the same system, with questions different from this lesson's.

← Previous3. Reliability as a feature, not an accident Next →5. Two real incidents, two different questions

4. Hands-on: reading Andes Cargo like an SRE would

Description

Connection to the module

Check 1 — The Lambda function: get-function, read for exposure to failure

Check 2 — The DynamoDB table: describe-table, read for exposure to failure

Check 3 — The S3 bucket: get-bucket-location, read for exposure to failure

The complete risk inventory: what was identified, and who resolves it

Common mistakes

Exercises

Summary and next step

Resources

Check 1 — The Lambda function: `get-function`, read for exposure to failure

Check 2 — The DynamoDB table: `describe-table`, read for exposure to failure

Check 3 — The S3 bucket: `get-bucket-location`, read for exposure to failure