Module 8: Capstone The Andes Cargo Reliability Package

3. Hands-on: introducing a deterministic synthetic incident

Next →

Description

This lesson builds this whole module's only genuinely new data: observability/upload_synthetic_incident_batch.py, a fixed sequence of 25 manifests uploaded to the real bucket — the same real S3 trigger Module 3 already established, never lambda invoke — with 10 malformed on purpose, all for the same reason: carrier omitted. Never random. Lesson 4 is going to take the two numbers this batch produces — the aggregate count and the two-window burn rate — and run them, with no code change, against Module 4's three alert engines.

Connection to the module

This batch is deliberately different from Module 3, lesson 3's batch in two concrete ways, neither accidental: the size (25, not 20), and above all the shape of the failure. Module 3 scattered three different reasons across three isolated positions (5, 12, 17) — the "background noise" pattern the Module 7 runbook classified as Branch B (no code action needed). This batch concentrates ten consecutive failures, all with the exact same reason, in the last ten positions — the signature of a code change that broke something, Branch A of the same decision tree, the one Module 7 never got to demonstrate with a worked example. This module's lesson 5 is going to confirm that classification by running the real runbook.

Step 1 — Why the failure's position, not just its count, tells the story

Before the code, the design decision. A batch with ten malformed manifests scattered among twenty-five, at random positions, would read the same as Module 3's batch, just noisier — the same Branch B, a bit bigger. This batch doesn't do that: the first fifteen positions are complete, valid manifests, cycling through the three already-known shipments (4471, 4472, 4473); starting at position 16, each of the remaining ten manifests has exactly the same missing field, carrier, with no exception. That concentration — clean, clean, clean... and suddenly, ten identical failures in a row — is the real signature of a deployment that changed something in the system generating manifests, at some point between position 15 and 16, not of individual manifests that each, on their own, arrived malformed.

   TWO WAYS TO FAIL -- THE SAME COUNT, TWO DIFFERENT STORIES

   MODULE 3, LESSON 3 (20 manifests)          THIS BATCH (25 manifests)
   ────────────────────────────────           ─────────────────────────
   Positions 5, 12, 17: broken                Positions 16-25: broken
   Each for a DIFFERENT reason                 All 10 for the SAME reason
   Scattered among good manifests               Concentrated at the end, in a block
        │                                            │
        ▼                                            ▼
   Signature: noise from individually          Signature: a code change that
   malformed manifests                          broke something, at one exact
   (Runbook M7.6, Branch B)                     point (Runbook M7.6, Branch A)

Step 2 — The complete script

Create observability/upload_synthetic_incident_batch.py in andes-cargo-infra/:

# upload_synthetic_incident_batch.py
# A FIXED, deterministic sequence of 25 shipment manifests uploaded to
# andes-cargo-shipment-docs, through the real S3 trigger path (never a manual
# `lambda invoke`) -- the same discipline Module 3, lesson 3 established.
#
# Different from Module 3, lesson 3's batch on purpose: positions 1-15 are
# well-formed (shipments 4471/4472/4473, cycling); positions 16-25 (10 of 25)
# are ALL missing the same field, carrier -- a concentrated, single-reason
# failure signature, not scattered noise. No random, no datetime.now().

import subprocess
from pathlib import Path

BUCKET = "andes-cargo-shipment-docs"
KEY_PREFIX = "manifests/year=2026/month=03/incident-drill"
OUT_DIR = Path("manifests-synthetic-incident")

SHIPMENTS = {
    "4471": {"originCountry": "Peru", "destinationCountry": "Chile", "carrier": "AndesExpress", "weightKg": "120"},
    "4472": {"originCountry": "Colombia", "destinationCountry": "Ecuador", "carrier": "AndesExpress", "weightKg": "85"},
    "4473": {"originCountry": "Chile", "destinationCountry": "Peru", "carrier": "RutaSur", "weightKg": "200"},
}

GOOD_CYCLE = ["4471", "4472", "4473"]

# Positions 1-15: well-formed. Positions 16-25: ALL missing "carrier" -- the
# concentrated failure signature this lesson needs (contrast with Module 3,
# lesson 3's three scattered, single-position failures).
BREAK_STARTS_AT = 16
TOTAL = 25


def manifest_lines(shipment_id, skip_field=None):
    fields = {"shipmentId": shipment_id, **SHIPMENTS[shipment_id]}
    if skip_field:
        fields.pop(skip_field, None)
    return "\n".join(f"{k}={v}" for k, v in fields.items()) + "\n"


def build_batch():
    batch = []
    for i in range(1, TOTAL + 1):
        shipment_id = GOOD_CYCLE[(i - 1) % 3]
        if i >= BREAK_STARTS_AT:
            text = manifest_lines(shipment_id, skip_field="carrier")
            batch.append((i, "bad", shipment_id, text))
        else:
            text = manifest_lines(shipment_id)
            batch.append((i, "good", shipment_id, text))
    return batch


def upload_one(index, outcome, shipment_id, text):
    OUT_DIR.mkdir(parents=True, exist_ok=True)
    filename = f"{index:02d}-shipment-{shipment_id}-manifest.txt"
    filepath = OUT_DIR / filename
    filepath.write_text(text)
    key = f"{KEY_PREFIX}/{filename}"
    subprocess.run(
        ["awslocal", "s3api", "put-object", "--bucket", BUCKET, "--key", key, "--body", str(filepath)],
        check=True,
    )
    return key


if __name__ == "__main__":
    for index, outcome, shipment_id, text in build_batch():
        key = upload_one(index, outcome, shipment_id, text)
        print(f"[{index:02d}/25] {outcome:>4} shipment={shipment_id} -> s3://{BUCKET}/{key}")

The only structural difference from upload_manifest_batch.py (Module 3) is in build_batch(): instead of a MALFORMED dictionary with scattered positions and different reasons, a single constant (BREAK_STARTS_AT = 16) decides, with a simple comparison (i >= BREAK_STARTS_AT), from which position everything breaks the same way. It's less code, not more — the script's simplicity reflects the simplicity of the story it tells: one change, at one point, with the same effect repeated ten times.

Step 3 — Running the upload

python3 observability/upload_synthetic_incident_batch.py

What to expect (representative — the exact same pattern Module 3, lesson 3 already confirmed for s3api put-object triggering process-shipment-manifest asynchronously; trimmed at the positions where the pattern changes):

[01/25] good shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/01-shipment-4471-manifest.txt
[02/25] good shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/02-shipment-4472-manifest.txt
...
[15/25] good shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/15-shipment-4473-manifest.txt
[16/25]  bad shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/16-shipment-4471-manifest.txt
[17/25]  bad shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/17-shipment-4472-manifest.txt
[18/25]  bad shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/18-shipment-4473-manifest.txt
[19/25]  bad shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/19-shipment-4471-manifest.txt
[20/25]  bad shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/20-shipment-4472-manifest.txt
[21/25]  bad shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/21-shipment-4473-manifest.txt
[22/25]  bad shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/22-shipment-4471-manifest.txt
[23/25]  bad shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/23-shipment-4472-manifest.txt
[24/25]  bad shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/24-shipment-4473-manifest.txt
[25/25]  bad shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=03/incident-drill/25-shipment-4471-manifest.txt

Fifteen good lines, ten bad lines in a continuous block — exactly the visual pattern Step 1 anticipated.

Step 4 — Two windows, two readings of the same batch

This is this lesson's central step. The batch, on its own, occurs in a short interval — the last minutes of an hour that, until then, had had completely healthy traffic. Just as BAD_WEEK in Module 2 needed a short window (the last day) and a long one (the full week) to reveal something the final balance alone didn't show, this incident needs the same two windows:

Short window — the batch itself (the last few minutes, awslocal cloudwatch get-metric-statistics with --period 300):

Invocations (Sum): 25.0
Errors (Sum):       10.0

Long window — the full hour (--period 3600; includes 375 healthy invocations from normal traffic already recorded before this batch was uploaded, added to this batch's 25):

Invocations (Sum): 400.0
Errors (Sum):       10.0

What to expect (representative — same reason declared since Module 3: with no LOCALSTACK_AUTH_TOKEN in this authoring environment, the LocalStack container doesn't start; the batch's 25 invocations are Step 3's direct arithmetic, the 375 from prior normal traffic are traffic already underway that hour, not re-uploaded in this lesson):

--- Synthetic M8.3 -- short window (the batch, ~5 min) ---
error_rate = 10 / 25 = 0.4000 (40.00%)
burn_rate  = 0.4000 / 0.001 = 400.00x

--- Synthetic M8.3 -- long window (the full hour) ---
error_rate = 10 / 400 = 0.0250 (2.50%)
burn_rate  = 0.0250 / 0.001 = 25.00x

0.001 is the same allowed_error_rate from SLO.md that has governed every burn rate calculation in this guide since Module 2. Both numbers — 400.00x in the short window, 25.00x in the long one — are, unambiguously, well above Table 5-8's most urgent threshold (Page (fast), 14.4x). This batch isn't a borderline case designed to graze a threshold — it's, deliberately, a clear case, so lesson 4 can focus on whether the machine reacts correctly, not on whether the case itself is ambiguous.

Step 5 — Why the long window's number (25.00x) isn't a coincidence with CloudWatch's

Notice something lesson 4 is going to use directly: error_rate = 10 / 400 = 2.5% in the long window is exactly the same ratio aws_cloudwatch_metric_alarm.manifest_error_budget_burn_rate (Module 4, lesson 5) is going to evaluate with its own errors / invocations metric_query over the same one-hour period — that alarm, declared in observability.tf, operates with a single window, exactly this calculation's long window. The short window's number (400.00x), in contrast, is only visible to the engines that do implement two windows — scripts/burn_rate_evaluator.py and Prometheus/Alertmanager — the same real CloudWatch limitation Module 4, lesson 5 already honestly declared. This coincidence isn't chance: all three engines calculate the same math over the same data, with different window coverage — exactly the same pattern ALERTING-POLICY.md already tested with bad_week and normal.

Common mistakes

Designing this lesson's batch with scattered positions, "so it looks different" from Module 3, without realizing that repeats the same failure signature (changing the number without changing the story). What happens: someone, building their own version of this batch, picks ten random (though fixed) positions instead of a continuous block, thinking it's enough for the size to be different. How to spot it: if your version of the batch, run through Module 7's runbook in lesson 5, lands in the same Branch B as the original example, instead of the Branch A this lesson aims to demonstrate. How to fix it: this lesson's Step 1 is explicit — what makes this batch genuinely different isn't the size (25 instead of 20), it's the failure's concentration into a continuous block, with a single repeated reason. A scattered batch, no matter how many positions it has, is still a variation of Module 3's same case.

Calculating the long window's burn rate using only the batch's 25 invocations, ignoring the 375 already recorded that hour from normal traffic (forgetting the long window isn't "just the new stuff"). What happens: someone, reading Step 4, uses 10 / 25 for the "long window" row too, instead of 10 / 400. How to spot it: if your two Step 4 rows show the same number. How to fix it: the short and long windows measure different time periods by definition — the short one is the recent batch; the long one is the full hour, which includes healthy traffic from before the batch. If both windows gave the same number, there'd be no reason to have two windows — this lesson's entire distinction, and Module 4's multi-window pattern in general, depends on them being genuinely different.

Assuming a 400x burn rate in the short window means the incident is "four hundred times worse" than one of 14.4x (reading the multiplier without the scale that accompanies it). What happens: someone compares 400.00x directly against 14.4x and concludes this incident is proportionally much more serious than any borderline case from Table 5-8. How to spot it: if your description of the incident uses a magnitude comparison ("28 times worse than the Page threshold") without connecting it to what that means in real budget minutes. How to fix it: Module 5, lesson 5, Exercise 2 already worked through this distinction with the 1,000x case — past a certain point, what matters isn't how much higher the multiplier is, but that any value above the most urgent threshold already justifies the most urgent possible response; the exact number (400.00x versus 25.00x) is useful information for diagnosing the incident's shape, not an indicator that the response should be "four hundred times more urgent."

Exercises

Exercise 1 — Calculate by hand what this lesson's short window would have shown if, instead of 10 of 25 malformed manifests, the batch had only 1 of 25 malformed (with the same reason, missing carrier).

See solution

error_rate = 1 / 25 = 0.04 (4%). burn_rate = 0.04 / 0.001 = 40.0x. Even with only one malformed manifest out of 25, the short window's burn rate would still be well above Table 5-8's most urgent threshold (14.4x) — the reason is that the short window, by definition, is a small batch: any error fraction reads, proportionally, much higher in a small sample than in a large one. It's the same lesson the Module 7 runbook already warned about: "one error out of five invocations breaches threshold = 0.001 exactly as surely as three hundred errors out of a hundred thousand" — you always need to read Invocations alongside Errors, never one without the other.

Exercise 2 — Explain why this lesson's build_batch() uses a single constant (BREAK_STARTS_AT) instead of the MALFORMED dictionary upload_manifest_batch.py (Module 3) used. What would it cost this script if you instead wanted to exactly reproduce Module 3's scattered pattern?

See solution

BREAK_STARTS_AT works because this batch has a property Module 3's didn't: every broken position shares the same reason (missing carrier) and forms a continuous range — a single comparison (i >= BREAK_STARTS_AT) is enough to decide, with no additional data structure. Reproducing Module 3's scattered pattern — non-consecutive positions, each with a different reason — would indeed need going back to that lesson's MALFORMED dictionary (position → specific reason), because there's no simple comparison rule that captures "positions 5, 12, and 17, each broken a different way." This lesson's code simplicity is a direct reflection of the simplicity of the story the batch tells, not an implementation coincidence.

Exercise 3 — Step 5 connects this lesson's long window's error_rate = 2.5% to Module 4's real CloudWatch alarm. Without running anything, predict: would the andes-cargo-manifest-error-budget-burn-rate alarm move to ALARM state with these numbers?

See solution

Yes, with a wide margin. observability.tf's alarm fires when errors / invocations >= threshold (0.001), with comparison_operator = "GreaterThanOrEqualToThreshold". With error_ratio = 10 / 400 = 0.025 (2.5%), well above the 0.001 (0.1%) the threshold requires, the condition is met unambiguously — the alarm would move from INSUFFICIENT_DATA/OK to ALARM in the next evaluation cycle. This module's lesson 4 confirms exactly this, along with the other two engines.

Summary and next step

This lesson built and ran observability/upload_synthetic_incident_batch.py: a fixed batch of 25 manifests, fifteen good and ten broken in a continuous block (positions 16-25), all for the exact same reason — carrier omitted — uploaded through the real S3 trigger. From that batch, you calculated two burn rate windows: a short one (the batch itself, 400.00x) and a long one (the full hour including prior healthy traffic, 25.00x), both well above Table 5-8's most urgent threshold. You confirmed that the long window (2.5% error rate) is exactly the number the CloudWatch alarm, with no change, is going to evaluate in lesson 4.

Before moving on you should be able to: explain why the failure's concentration, not just its count, is what distinguishes this batch from Module 3's; calculate by hand the burn rate for any window given its invocation/error count; and predict, without running anything, whether a 0.001-threshold alarm would fire with a given error_ratio.

Lesson 4 takes these two numbers — 400.00x and 25.00x — and runs them, with no code change, against Module 4's three alert engines: the Python evaluator, the real Alertmanager rule, and the real CloudWatch alarm. If all three agree this fires, the complete machine has just passed its first test with data it never saw before.

Resources

  1. This same repository, Module 3, lesson 3 (03-hands-on-real-metrics-from-the-inherited-lambda.md) — the direct precedent for the S3-upload pattern, and the batch this one deliberately contrasts with.
  2. This same repository, Module 2, lesson 6 (06-hands-on-burn-rate-not-just-the-balance.md) — the two-window (short/long) pattern this lesson applies to a new incident.
  3. This same repository, Module 7, lesson 6 (06-hands-on-the-first-real-andes-cargo-runbook.md) — the decision tree (Branch A/B/C) this module's lesson 5 is going to apply to this same batch.
  4. Google SRE Workbook — Alerting on SLOs — the source of the burn rate formula applied in Step 4.
← Previous2. Architecture review: the full reliability machineNext →4. End-to-end: the alert fires, the incident gets declared