Module 7: Blameless Postmortems And Runbooks

7. Hands-on: the honest backup/restore attempt

Next →

Description

🔶 This is the whole guide's lesson with the highest technical uncertainty, declared as such in DISENO.md before it was even written. POSTMORTEM.md's action item #4 (lesson 4) asks for an evaluation of a Shipments backup owned by the team, instead of depending — as happened to DataTalks.Club — on a recovery mechanism the team itself didn't know existed. This lesson actually attempts awslocal dynamodb create-backup and restore-table-from-backup on a disposable copy of Shipments, and documents the real result, whatever it is, instead of promising in advance that it would work.

Connection to the module

This lesson follows, point by point, the same pattern cloud-security-and-guardrails-guide already used with CloudTrail in its Module 7, lesson 5: really attempt it, document the real outcome exactly as it came out, and precisely distinguish what that outcome confirms from what it doesn't. The honest difference from that precedent, which this lesson declares from the first step: CloudTrail had explicitly confirmed API coverage in LocalStack's documentation; DynamoDB's backup/restore doesn't have that same clarity — a higher level of uncertainty, declared as such since DISENO.md.

Step 1 — Why this lesson is more uncertain than its CloudTrail precedent

Before attempting anything, it's worth precisely reading what LocalStack's official documentation confirms and doesn't confirm about DynamoDB:

"Included in Plans: Hobby, Base, Ultimate" — with "Persistence Supported" flagged at the top of DynamoDB's service page.

— LocalStack Docs — DynamoDB

DynamoDB, as a complete service, is indeed confirmed on the free Hobby plan — the same source this guide's Module 1 already used for describe-table. What that same page does not do, unlike the CloudWatch page this guide's Module 3 already precisely cited ("Included in Plans: Hobby, Base, Ultimate", with an explicit list of limitations — no Logs Insights, no composite alarms), is list with the same clarity which specific backup API operations (CreateBackup, DescribeBackup, RestoreTableFromBackup, ListBackups) are covered and which aren't. This is, precisely, the difference between the CloudTrail case (cloud-security-and-guardrails-guide's Module 7: "create-trail, start-logging, and lookup-events" explicitly confirmed) and this one: here, the parent service is confirmed, but the specific family of operations this lesson needs doesn't have the same textual confirmation.

   TWO LEVELS OF UNCERTAINTY -- WHY THIS LESSON DECLARES IT DIFFERENT

   CLOUDTRAIL (cloud-security M7.5)          DYNAMODB BACKUP/RESTORE (this lesson)
   ─────────────────────────────────          ──────────────────────────────────────
   Service: confirmed on Hobby                 Service (DynamoDB): confirmed on Hobby
   Specific operations (create-trail,           Specific operations (create-backup,
   lookup-events): CONFIRMED by name             restore-table-from-backup): NOT listed
   in official documentation                     with the same clarity
        │                                            │
        ▼                                            ▼
   Uncertainty: only whether the real            Uncertainty: whether the service EVEN
   event reaches the trail in this               RESPONDS at all to these specific
   specific token-less environment                operations, on top of the same
                                                    token-less environment limit

Step 2 — The disposable copy, never the real table

A backup/restore attempt isn't practiced against Shipments directly — not even in a $0 lab, the correct discipline is to never experiment on a production resource when a disposable copy is an option. Before the attempt, Shipments-backup-drill gets created, with the exact same schema Shipments already confirmed since this guide's Module 1 (shipmentId as the partition key, PAY_PER_REQUEST):

awslocal dynamodb create-table \
  --table-name Shipments-backup-drill \
  --attribute-definitions AttributeName=shipmentId,AttributeType=S \
  --key-schema AttributeName=shipmentId,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

What to expect (representative — the exact same schema Shipments already confirmed in this guide's Module 1, lesson 4; with no LOCALSTACK_AUTH_TOKEN, the LocalStack container doesn't start in this authoring environment):

{
    "TableDescription": {
        "TableName": "Shipments-backup-drill",
        "TableStatus": "ACTIVE",
        "KeySchema": [{ "AttributeName": "shipmentId", "KeyType": "HASH" }],
        "BillingModeSummary": { "BillingMode": "PAY_PER_REQUEST" },
        "TableArn": "arn:aws:dynamodb:us-east-1:000000000000:table/Shipments-backup-drill"
    }
}

Step 3 — The real attempt: verifying the network condition first

Before the awslocal command itself, this lesson does something no earlier lesson in this ecosystem did with the same level of detail: confirm, with a direct, minimal network check, exactly why any awslocal attempt is going to fail in this specific authoring environment — not as an assumption, but as a fact verified at the moment this lesson was written:

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(3)
try:
    s.connect(("127.0.0.1", 4566))
except Exception as e:
    print(type(e).__name__, e)

What actually happened, literally, running it in this environment (verified while writing this lesson):

ConnectionRefusedError [Errno 61] Connection refused

This result confirms, as directly as possible, the same root cause every representative attempt in this guide has named since Module 1: no process is listening on port 4566 in this authoring environment, because the LocalStack container never started without LOCALSTACK_AUTH_TOKEN. awslocal is, at bottom, a thin wrapper over the AWS CLI pointing at http://localhost:4566 — any command it runs, no matter which, fails at this exact network point, before DynamoDB's specific logic even comes into play.

With that root cause directly confirmed, the real create-backup attempt:

awslocal dynamodb create-backup \
  --table-name Shipments-backup-drill \
  --backup-name shipments-backup-drill-test-1

What happened, expected (reconstructed from botocore/AWS CLI's standard connection-error format for this same port, already documented literally by cloud-security-and-guardrails-guide, Module 7, lesson 5, for the same environment with no active container):

aws: [ERROR]: Could not connect to the endpoint URL: "http://localhost:4566/"

And the restore attempt, with the same root cause:

awslocal dynamodb restore-table-from-backup \
  --backup-arn arn:aws:dynamodb:us-east-1:000000000000:table/Shipments-backup-drill/backup/PLACEHOLDER \
  --target-table-name Shipments-backup-drill-restored

What happened, expected (same reason):

aws: [ERROR]: Could not connect to the endpoint URL: "http://localhost:4566/"

This second command's --backup-arn uses PLACEHOLDER on purpose: since create-backup never completed, there's no real backup ARN to cite — inventing one that looked real would be exactly the same error this ecosystem already named in this guide's Module 4 about the PagerDuty integration key: a value that appears real without being real is less honest than an explicit placeholder.

Step 4 — What this attempt does confirm, and what it doesn't

  • It confirms, with a direct network check, the exact root cause — no process is listening on 4566 in this authoring environment, the same limit named since this guide's Module 1, now verified with a minimal test independent of awslocal itself.
  • It does not confirm, in this specific environment, whether create-backup and restore-table-from-backup are implemented in LocalStack's free tier. Unlike the CloudTrail case, the official documentation doesn't give that explicit, by-operation-name confirmation — Step 1 already declared this.
  • With a real LOCALSTACK_AUTH_TOKEN and the container running, the exact shape a successful create-backup response would have, per AWS's official API reference (never invented field by field):
{
    "BackupDetails": {
        "BackupArn": "<ARN generated by DynamoDB when the backup is created>",
        "BackupName": "shipments-backup-drill-test-1",
        "BackupStatus": "AVAILABLE",
        "BackupType": "USER",
        "BackupCreationDateTime": "<timestamp of the moment the command ran>"
    }
}

— AWS DynamoDB API Reference — CreateBackup, with BackupArn and BackupCreationDateTime explicitly marked with no fixed value: both depend on when and in which account you run the command, exactly the kind of data DISENO.md's honesty table forbids presenting as a fixed literal.

And the shape restore-table-from-backup would have, with the same discipline — just the RestoreSummary section, the specific part of the response confirming the restore started from a real backup:

{
    "TableDescription": {
        "TableName": "Shipments-backup-drill-restored",
        "TableStatus": "CREATING",
        "RestoreSummary": {
            "SourceBackupArn": "<the same BackupArn from create-backup>",
            "SourceTableArn": "<ARN of Shipments-backup-drill>",
            "RestoreDateTime": "<timestamp of the moment of restoration>",
            "RestoreInProgress": true
        }
    }
}

— AWS DynamoDB API Reference — RestoreTableFromBackup. AWS's own documentation adds an operational honesty worth quoting here, because it changes what "restored" means in practice: "You must manually set up the following on the restored table: Auto scaling policies [...] IAM policies [...] Amazon CloudWatch metrics and alarms [...] Tags [...] Stream settings [...] Time to Live (TTL) settings." — a table restored from a backup does not automatically inherit Module 4's CloudWatch alarm, nor any IAM or TTL configured on the original table; restoring the data is only the first step, not the complete recovery process.

Step 5 — Fulfilling action item #4, without faking a result that didn't happen

Lesson 4's action item #4 demanded "document the real outcome, whatever it is" — not "confirm backup/restore works in LocalStack." With this lesson's evidence, the documented outcome is precise: the network cause is directly confirmed (Step 3); the specific backup API's coverage on LocalStack's Hobby plan still lacks explicit textual confirmation (Step 1); and the exact shape a successful response would have is documented, field by field, against AWS's official reference, with no invented value that depended on execution time (Step 4). This is exactly the kind of result — uncertain at a specific point, honest about exactly which one — this guide's design anticipated since DISENO.md: "Documented exactly as it comes out, never promised in advance."

Common mistakes

Interpreting Step 1's absence of explicit confirmation as "backup/restore definitely doesn't work in LocalStack" (turning uncertainty into a denial). What happens: someone reads that the documentation doesn't list create-backup with the same clarity as create-trail, and concludes the operation surely fails even with the token exported. How to spot it: if your summary of this lesson states with certainty that DynamoDB backup/restore "isn't supported" in LocalStack. How to fix it: Step 1 is precise about what kind of uncertainty this is — absence of explicit confirmation, not confirmation that the operation fails. It's honest not to know for certain; it isn't honest to turn "unconfirmed" into "confirmed not to work," a claim this lesson has no evidence to support.

Inventing a real-looking BackupArn or BackupCreationDateTime to fill out Step 4's example (repeating the same error already named with Module 4's PagerDuty key). What happens: someone, uncomfortable leaving a field marked <ARN generated...>, replaces it with an ARN that looks authentic. How to spot it: if your version of Step 4's JSON has an ARN or timestamp that looks like a real value, instead of the explicit marker. How to fix it: Step 4, on purpose, leaves those two fields with no fixed value — they depend on when and in which account you actually run the command, and DISENO.md's honesty table forbids presenting that kind of data as a fixed literal. An explicit, readable marker is more honest than a value that looks real without being real.

Assuming that, because this lesson's attempt didn't confirm success, action item #4 is still "pending" or "unmet" (confusing an uncertain result with an incomplete task). What happens: someone, seeing create-backup didn't manage to run successfully in this environment, concludes lesson 4's action item #4 is still open. How to spot it: if your evaluation of this lesson treats it as a failure instead of the exact fulfillment of what the action item asked for. How to fix it: this lesson's Step 5 clarifies it — the action item asked to "investigate and document the real outcome," exactly what this lesson did with verifiable evidence (Step 3's network test, Step 4's official AWS reference); an uncertain result, documented with complete honesty, fulfills that action item just as well as a successful result would have.

Exercises

Exercise 1 — Run Step 3's network check yourself (the three-line Python script) on your own machine, with no LocalStack running, and confirm you get the same type of connection error (the exact message may vary depending on your operating system).

See solution

On a machine with no process listening on port 4566, Step 3's socket.connect() should fail with some error from the "connection refused" family — on Unix-based systems (macOS, Linux), typically ConnectionRefusedError, with an errno number that can vary slightly between operating systems (on macOS, Errno 61; on Linux, often Errno 111) but with the same meaning: the operating system actively confirmed no process is listening on that port, unlike a timeout (which would instead indicate the packet was lost on the network with no response of any kind). This check is independent of awslocal — it confirms the network root cause directly, without depending on the AWS CLI even being installed.

Exercise 2 — Explain, using Step 4, why AWS's quote about what does not get automatically restored ("Auto scaling policies [...] CloudWatch metrics and alarms [...]") is relevant to Andes Cargo specifically, beyond being a generic warning.

See solution

It's relevant because it connects directly to the real alarm this guide's Module 4, lesson 5 already built: andes-cargo-manifest-error-budget-burn-rate doesn't watch Shipments-backup-drill-restored or any automatically restored table — that alarm is tied, by name, to the process-shipment-manifest Lambda, not to any specific DynamoDB table, so in this particular case the alarm would keep working unchanged. But if Andes Cargo ever had an alarm directly on the Shipments table itself (say, on its consumed capacity), restoring that table from a backup to a new name would leave that hypothetical alarm orphaned, watching a resource that's no longer the one in production. AWS's quote isn't an abstract warning — it's the exact reason a real restore runbook (outside this specific lesson's scope) would need an explicit additional step: reconnecting any monitoring to the new table name, not assuming "restored" means "exactly as before in every sense."

Exercise 3 — A classmate argues this lesson "contributes nothing" because it failed to confirm, in any way, that backup/restore works in LocalStack. Do you agree, using Step 4's "What this attempt does confirm, and what it doesn't" section?

See solution

Disagree. Step 4 precisely lists three things this lesson does contribute, none of which depends on LocalStack responding successfully: first, a direct, independent confirmation of the network root cause in this environment (Step 3's Python script, which depends on neither awslocal nor any assumption); second, the exact shape — field by field, against AWS's official reference — a real response from both operations would have, useful for anyone running this same attempt with a real token who needs to know what to expect; third, an operational honesty quoted from AWS itself (what doesn't get automatically restored) that no earlier lesson in this ecosystem had mentioned. An attempt that doesn't confirm success isn't automatically a worthless attempt — the difference between "contributes nothing" and "precisely documents the limits of what can be confirmed here" is, again, the same honesty discipline governing every lesson in this guide.

Summary and next step

This lesson really attempted awslocal dynamodb create-backup and restore-table-from-backup on Shipments-backup-drill, a disposable copy of Shipments's real schema — never the production table itself. You confirmed, with a direct network check independent of awslocal, the exact root cause of the failure in this authoring environment, and documented, field by field against AWS's official API reference, the exact shape a successful response would have, without inventing any ARN or timestamp that depended on execution time. You declared, from the first step, why this lesson carries more uncertainty than its CloudTrail precedent: DynamoDB is confirmed on the Hobby plan, but the specific backup API doesn't have the same textual confirmation.

Before moving on you should be able to: explain the uncertainty difference between this case and CloudTrail's; reproduce Step 3's network check; and defend why an uncertain result, honestly documented, fulfills action item #4 just as well as a confirmed success.

Lesson 8, this module's final project, integrates the four artifacts — POSTMORTEM.md, the SMART action items, the runbook, and this honest attempt's result — into Andes Cargo's complete portfolio package.

Resources

  1. LocalStack Docs — DynamoDB — the source confirming the Hobby plan and the absence of explicit detail about the backup API, cited in Step 1.
  2. AWS DynamoDB API Reference — CreateBackup — the exact source for the BackupDetails fields reconstructed in Step 4.
  3. AWS DynamoDB API Reference — RestoreTableFromBackup — the exact source for RestoreSummary and the quote on what doesn't get automatically restored.
  4. cloud-security-and-guardrails-guide, Module 7, lesson 5 (05-hands-on-cloudtrail-as-far-as-localstack-goes.md) — the direct precedent for the honest-attempt pattern this lesson follows.
  5. This same repository, Module 1, lesson 4 (04-hands-on-reading-andes-cargo-like-an-sre.md) — Shipments's real schema, reused for Shipments-backup-drill.
← Previous6. Hands-on: Andes Cargo's first real runbookNext →8. Project: Andes Cargo's postmortem and runbooks package