3. Hands-on: real metrics from the inherited Lambda

Description

This lesson builds lesson 2's "punctuality board": a fixed, deterministic batch of 20 manifests, really uploaded to andes-cargo-shipment-docs through the real S3 trigger — never with awslocal lambda invoke, the manual path SLO.md explicitly excludes from the SLI — and then read as two aggregate numbers with awslocal cloudwatch get-metric-statistics over AWS/Lambda/Invocations and AWS/Lambda/Errors. Three of the 20 manifests are deliberately broken, at fixed positions, each one violating exactly one of the rules validate_manifest() — the real function, inherited from aws-serverless-and-containers-guide — already checks. Never random.

This lesson carries an honesty you need to read before the first code block: in this specific writing environment, with no LOCALSTACK_AUTH_TOKEN exported, the LocalStack container doesn't start, so neither the real upload nor the CloudWatch query ran against a live LocalStack. The official documentation confirms CloudWatch — logs, metrics, and alarms — is on LocalStack's Hobby plan, with native integration for the two Lambda metrics this lesson needs. Everything you'll read in this lesson's "What to expect" blocks is reconstructed field by field from that confirmed behavior and from process-shipment-manifest's real code — never invented. If you run this lesson with your own LOCALSTACK_AUTH_TOKEN, the result should be exactly this.

Connection to the module

Lesson 2 promised metrics answer "how many valid events were there, and how many were good?" — compute_sli()'s exact numerator and denominator. This lesson produces those two numbers for the first time with data that isn't a hand-committed dataset. Lessons 4 and 5 are going to read this same batch — this lesson's same Errors: 3.0 — with logs and traces, respectively.

---

Step 1 — The fixed batch: 20 manifests, 3 deliberately broken

process-shipment-manifest validates five required fields (shipmentId, originCountry, destinationCountry, carrier, weightKg) and that weightKg is a positive number — the real logic of validate_manifest(), inherited unchanged from aws-serverless-and-containers-guide, Module 2. This batch uses exactly those rules to fail three times, each for a different reason:

#	Status	Shipment	What's missing or wrong	Error validate_manifest() produces
5	broken	4471	weightKg omitted entirely	missing required field: weightKg
12	broken	4472	carrier omitted entirely	missing required field: carrier
17	broken	4473	weightKg=heavy (non-numeric)	weightKg must be numeric

The remaining 17 are complete, valid manifests, cycling among the three already-familiar shipments — 4471 (Peru→Chile, AndesExpress, 120 kg), 4472 (Colombia→Ecuador, AndesExpress, 85 kg), 4473 (Chile→Peru, RutaSur, 200 kg).

---

Step 2 — The complete script

Create observability/upload_manifest_batch.py in andes-cargo-infra/:

# upload_manifest_batch.py
# Uploads a FIXED sequence of 20 shipment manifests to andes-cargo-shipment-docs, one at a
# time, through `awslocal s3api put-object` -- the real trigger path for
# process-shipment-manifest (S3 ObjectCreated -> Lambda), never a manual `lambda invoke`.
# SLO.md excludes manual test invocations from the SLI denominator; every one of these 20
# uploads is meant to count as real traffic through the real path.
#
# 17 well-formed manifests (shipments 4471/4472/4473, cycling); 3 malformed on purpose, at
# fixed positions 5, 12 and 17 -- each missing or corrupting exactly the field that
# validate_manifest() (aws-serverless-and-containers-guide, Module 2) already checks for.
# No random, no datetime.now() -- same 20 files, same order, every run.

import subprocess
from pathlib import Path

BUCKET = "andes-cargo-shipment-docs"
KEY_PREFIX = "manifests/year=2026/month=08/batch"
OUT_DIR = Path("manifests-batch")

SHIPMENTS = {
    "4471": {"originCountry": "Peru", "destinationCountry": "Chile", "carrier": "AndesExpress", "weightKg": "120"},
    "4472": {"originCountry": "Colombia", "destinationCountry": "Ecuador", "carrier": "AndesExpress", "weightKg": "85"},
    "4473": {"originCountry": "Chile", "destinationCountry": "Peru", "carrier": "RutaSur", "weightKg": "200"},
}

GOOD_CYCLE = ["4471", "4472", "4473"]

# index -> (shipmentId, field_to_break, broken_value_or_None)
MALFORMED = {
    5: ("4471", "weightKg", None),      # omitted entirely -> "missing required field: weightKg"
    12: ("4472", "carrier", None),      # omitted entirely -> "missing required field: carrier"
    17: ("4473", "weightKg", "heavy"),  # present but not numeric -> "weightKg must be numeric"
}


def manifest_lines(shipment_id, skip_field=None, override_field=None, override_value=None):
    fields = {"shipmentId": shipment_id, **SHIPMENTS[shipment_id]}
    if override_field:
        fields[override_field] = override_value
    if skip_field:
        fields.pop(skip_field, None)
    return "\n".join(f"{k}={v}" for k, v in fields.items()) + "\n"


def build_batch():
    batch = []
    good_index = 0
    for i in range(1, 21):
        if i in MALFORMED:
            shipment_id, field, broken_value = MALFORMED[i]
            if broken_value is None:
                text = manifest_lines(shipment_id, skip_field=field)
            else:
                text = manifest_lines(shipment_id, override_field=field, override_value=broken_value)
            batch.append((i, "bad", shipment_id, text))
        else:
            shipment_id = GOOD_CYCLE[good_index % 3]
            good_index += 1
            text = manifest_lines(shipment_id)
            batch.append((i, "good", shipment_id, text))
    return batch


def upload_one(index, outcome, shipment_id, text):
    OUT_DIR.mkdir(parents=True, exist_ok=True)
    filename = f"{index:02d}-shipment-{shipment_id}-manifest.txt"
    filepath = OUT_DIR / filename
    filepath.write_text(text)
    key = f"{KEY_PREFIX}/{filename}"
    subprocess.run(
        ["awslocal", "s3api", "put-object", "--bucket", BUCKET, "--key", key, "--body", str(filepath)],
        check=True,
    )
    return key


if __name__ == "__main__":
    for index, outcome, shipment_id, text in build_batch():
        key = upload_one(index, outcome, shipment_id, text)
        print(f"[{index:02d}/20] {outcome:>4} shipment={shipment_id} -> s3://{BUCKET}/{key}")

manifest_lines() builds the real text file parse_manifest() expects — key=value lines, never JSON; skip_field simulates an omitted field; override_field/override_value simulates a field that's present but invalid. build_batch() is the fixed 20-position sequence, with the three broken ones always at 5, 12, and 17, no matter how many times you run the script.

---

Step 3 — Running the upload

python3 observability/upload_manifest_batch.py

What to expect (representative — the exact s3api put-object pattern, each upload triggers process-shipment-manifest asynchronously via the S3 trigger, exactly as aws-core-services-guide Module 6 already confirmed running for real):

[01/20] good shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/01-shipment-4471-manifest.txt
[02/20] good shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/02-shipment-4472-manifest.txt
[03/20] good shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/03-shipment-4473-manifest.txt
[04/20] good shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/04-shipment-4471-manifest.txt
[05/20]  bad shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/05-shipment-4471-manifest.txt
[06/20] good shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/06-shipment-4472-manifest.txt
[07/20] good shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/07-shipment-4473-manifest.txt
[08/20] good shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/08-shipment-4471-manifest.txt
[09/20] good shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/09-shipment-4472-manifest.txt
[10/20] good shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/10-shipment-4473-manifest.txt
[11/20] good shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/11-shipment-4471-manifest.txt
[12/20]  bad shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/12-shipment-4472-manifest.txt
[13/20] good shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/13-shipment-4472-manifest.txt
[14/20] good shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/14-shipment-4473-manifest.txt
[15/20] good shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/15-shipment-4471-manifest.txt
[16/20] good shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/16-shipment-4472-manifest.txt
[17/20]  bad shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/17-shipment-4473-manifest.txt
[18/20] good shipment=4473 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/18-shipment-4473-manifest.txt
[19/20] good shipment=4471 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/19-shipment-4471-manifest.txt
[20/20] good shipment=4472 -> s3://andes-cargo-shipment-docs/manifests/year=2026/month=08/batch/20-shipment-4472-manifest.txt

Twenty lines, twenty new objects in andes-cargo-shipment-docs, twenty asynchronous Lambda invocations triggered — three of which are going to end in FunctionError: "Unhandled", exactly as the already-confirmed aws-serverless-and-containers-guide precedent documents for an uncaught ValueError.

---

Step 4 — Reading the aggregate count with CloudWatch

awslocal cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Invocations \
  --dimensions Name=FunctionName,Value=process-shipment-manifest \
  --start-time 2026-08-14T14:00:00Z \
  --end-time 2026-08-14T15:00:00Z \
  --period 3600 \
  --statistics Sum

What to expect (representative — CloudWatch, Lambda metrics, confirmed on LocalStack's Hobby plan):

{
    "Label": "Invocations",
    "Datapoints": [
        {
            "Timestamp": "2026-08-14T14:00:00+00:00",
            "Sum": 20.0,
            "Unit": "Count"
        }
    ]
}

awslocal cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Errors \
  --dimensions Name=FunctionName,Value=process-shipment-manifest \
  --start-time 2026-08-14T14:00:00Z \
  --end-time 2026-08-14T15:00:00Z \
  --period 3600 \
  --statistics Sum

What to expect (representative, same reason):

{
    "Label": "Errors",
    "Datapoints": [
        {
            "Timestamp": "2026-08-14T14:00:00+00:00",
            "Sum": 3.0,
            "Unit": "Count"
        }
    ]
}

Two numbers, exactly the ones lesson 2's table promised: total_valid = 20 (every invocation triggered by the real S3 trigger, none manual), total_good = 20 - 3 = 17. Lesson 7 of this module is going to feed compute_sli() with exactly these two numbers.

---

Common mistakes

Using awslocal lambda invoke to generate the batch, instead of uploading real files to S3 (unknowingly breaking the SLI definition). What happens: someone, for convenience, invokes the Lambda directly with a test payload instead of uploading real manifests to the bucket. How to spot it: if your command uses lambda invoke with a --payload file://... instead of s3api put-object. How to fix it: SLO.md, in its SLI section, explicitly excludes "Manual test invocations (console/CLI, synthetic payloads used only to verify a deployment)" from the denominator — not because the technical result is different, but because those invocations don't represent real customer traffic. This lesson uploads files to the real bucket, triggering the real S3 trigger, precisely so the 20 invocations count as legitimate traffic under the definition already written in SLO.md.

Interpreting Errors: 3.0 as if it were three bugs of the same type (not distinguishing metrics from logs). What happens: someone sees the number 3 and assumes the three invocations failed the same way. How to spot it: if your explanation of the error uses "the" in singular ("the error that happened three times") instead of plural. How to fix it: Step 1's table already shows three different reasons — a missing field, another missing field, a non-numeric value; Errors: 3.0 is an aggregate count, exactly the limitation lesson 2 named for this pillar. This module's lesson 4 is going to confirm, with real logs, that they're three different error messages.

Assuming 20 invocations are enough to measure SLO.md's monthly SLI (confusing a verification batch with real production traffic). What happens: someone, seeing Invocations: 20 and Errors: 3, calculates an 85% SLI and compares it directly against SLO.md's 99.9%, concluding the system is severely off its target. How to spot it: if your conclusion from this lesson is "Andes Cargo is way below its SLO" with no mention at all of sample size. How to fix it: 20 invocations in a minutes-wide window, with 3 failures deliberately injected to verify the observability pipeline detects them, aren't a representative sample of a full month of real traffic — this module's lesson 7 handles this distinction with the care it deserves, showing what happens if you treat this batch as if it were the whole month, and what happens if you treat it correctly, as one more real day among Module 2's 30.

---

Exercises

Exercise 1 — Calculate, without running anything, what Errors would have shown if only position 17 had been broken (positions 5 and 12 fixed). What about Invocations?

See solution

Invocations would still be 20.0 — the invocation count doesn't depend on whether they succeeded or not, it counts every triggered invocation. Errors would drop to 1.0 — only position 17 (weightKg=heavy) would still fail. This confirms something important about the Invocations metric: it measures traffic, not success; success or failure lives exclusively in Errors, a separate metric, exactly as Module 2's four golden signals table already distinguished traffic from errors as two different signals.

Exercise 2 — Explain why position 5 (missing weightKg) and position 17 (weightKg=heavy) produce the same type of exception (ValueError) but different error messages. Use validate_manifest()'s real code.

See solution

validate_manifest() first checks that every field in REQUIRED_FIELDS is present — if weightKg is missing entirely (position 5), the error is "missing required field: weightKg". Only if the field is present does the function try to convert it with float(fields["weightKg"]) inside a try/except block: if that conversion fails (position 17, "heavy" isn't convertible to a number), the error is "weightKg must be numeric". Both cases end in the same final ValueError — lambda_handler always re-raises with raise ValueError(f"manifest validation failed for {key}: {errors}") — but the errors list that message includes is different in each case, because it comes from a different validation branch.

Exercise 3 — Design a fourth broken position, without writing code yet. Which position of the batch (between 1 and 20, other than 5, 12, and 17) would you choose, and what validate_manifest() error would you trigger, if you specifically wanted to test the non-positive weightKg rule (for example, weightKg=-5)?

See solution

Any new position would work mechanically — for example, position 9 — as long as it's added to MALFORMED with a tuple like ("4472", "weightKg", "-5"). The error it would produce: validate_manifest() does manage to convert "-5" with float() with no exception, but the next check (if weight <= 0) does fire, adding "weightKg must be a positive number" to the error list — the third and last type of error this function can produce, different from both "missing field" and "non-numeric," and one this lesson's batch, deliberately, doesn't cover yet. This exercise shows that this batch's three failures don't exhaust every way of breaking a manifest — they cover three, out of a total of four possible in validate_manifest().

---

Summary and next step

This lesson built and "ran" — representative, with the exact technical reason declared — this module's first observability pillar: a fixed batch of 20 manifests, uploaded to andes-cargo-shipment-docs through the real S3 trigger (never a manual lambda invoke, SLO.md's explicit exclusion), with 3 deliberately broken at fixed positions, each violating a different validate_manifest() rule, the real function inherited from aws-serverless-and-containers-guide. You read the aggregate result with awslocal cloudwatch get-metric-statistics: Invocations: 20.0, Errors: 3.0 — the two exact numbers compute_sli() needs.

Before moving on you should be able to: explain why this batch uses s3api put-object and not lambda invoke; name the batch's three failure reasons, with their exact position; and explain why 20 invocations aren't, on their own, a representative sample of SLO.md's monthly SLI.

Lesson 4 takes that same Errors: 3.0 and opens it up: which specific requestId corresponds to each of the three failures, and what error message each one left in /aws/lambda/process-shipment-manifest's real logs.

Resources

1. LocalStack Docs — CloudWatch — confirmation of the Hobby plan and the native Lambda metrics this lesson reads.
2. AWS CLI — cloudwatch get-metric-statistics Command Reference — this lesson's command's official reference.
3. aws-serverless-and-containers-guide (NIEVA), Module 2, lesson 7 — validate_manifest() and the real precedent of awslocal lambda invoke with a synthetic S3 payload, the source of this lesson's "manual" vs. "real" distinction.
4. This same repository, Module 2, lesson 8 (08-project-andes-cargos-slo-md.md) — SLO.md, SLI section, the exact source for the manual-invocation exclusion.
5. aws-core-services-guide (NIEVA), Module 6, lesson 6 — the real confirmation, run in that environment, that an S3 upload triggers process-shipment-manifest asynchronously.