Module 3: Observability As Sli Input

4. Hands-on: real logs with `jq`

Description

Errors: 3.0 is all lesson 3 could say about the batch's failures — a count, with no detail at all. This lesson opens up those three failures: which specific requestId corresponds to each one, and what real error message validate_manifest() left in each case. The CloudWatch Logs query that brings that data is, like the previous lesson, representative — the reason is identical: with no LOCALSTACK_AUTH_TOKEN, LocalStack doesn't start in this environment. What changes in this lesson, and is the central piece: once that output (representative, but faithful to the real format CloudWatch Logs produces) gets saved to a file, jq processes it for real — the binary ran, in this very environment, to produce every "What to expect" block in this lesson's second half.

Connection to the module

Lesson 2 promised logs answer "which ones, specifically, were the bad events, and why?" — this lesson demonstrates it with the three failed invocations from lesson 3's batch, identified by requestId, each with its real error message. Lesson 5 is going to take one of those three requestIds — the one from position 17 — and follow it with a complete trace in Jaeger.

Step 1 — Querying the real logs for the three failures

Every process-shipment-manifest invocation writes to /aws/lambda/process-shipment-manifest (the log group name aws-core-services-guide, Module 6, already established as Lambda's automatic convention: /aws/lambda/<function-name>). Two lines per failed invocation are the ones that matter here: lambda_handler's real print(f"Invalid manifest {key}: {errors}") right before re-raising the exception, and the REPORT line the Lambda runtime automatically appends at the end of every invocation, with Status: error\tError Type: Unhandled when the invocation didn't catch its exception — exactly the behavior aws-serverless-and-containers-guide already confirmed for this same Lambda.

A filter pattern with two alternatives (?"..." ?"...", CloudWatch Logs Insights' OR syntax) brings both lines back in a single query:

awslocal logs filter-log-events \
  --log-group-name /aws/lambda/process-shipment-manifest \
  --filter-pattern '?"Invalid manifest" ?"Status: error"' \
  --start-time 2026-08-14T14:00:00Z \
  --end-time 2026-08-14T15:00:00Z

What to expect (representative — CloudWatch Logs, confirmed on LocalStack's Hobby plan; save this output exactly as observability/manifest-log-events.json, you'll need it in Step 2):

{
    "events": [
        {
            "logStreamName": "2026/08/14/[$LATEST]4a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d",
            "timestamp": 1786732801005,
            "message": "Invalid manifest manifests/year=2026/month=08/batch/05-shipment-4471-manifest.txt: ['missing required field: weightKg']\n",
            "ingestionTime": 1786732801512,
            "eventId": "38234501234567890123456789012345678901"
        },
        {
            "logStreamName": "2026/08/14/[$LATEST]4a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d",
            "timestamp": 1786732801041,
            "message": "REPORT RequestId: a47f3e21-8b6a-4c9d-9f12-3d8e7b1a2c44\tDuration: 38.47 ms\tBilled Duration: 39 ms\tMemory Size: 128 MB\tMax Memory Used: 44 MB\tStatus: error\tError Type: Unhandled\n",
            "ingestionTime": 1786732801530,
            "eventId": "38234501234567890123456789012345678902"
        },
        {
            "logStreamName": "2026/08/14/[$LATEST]4a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d",
            "timestamp": 1786732808211,
            "message": "Invalid manifest manifests/year=2026/month=08/batch/12-shipment-4472-manifest.txt: ['missing required field: carrier']\n",
            "ingestionTime": 1786732808698,
            "eventId": "38234501234567890123456789012345678903"
        },
        {
            "logStreamName": "2026/08/14/[$LATEST]4a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d",
            "timestamp": 1786732808253,
            "message": "REPORT RequestId: f3c91a08-2e4d-4b7f-8a3c-5e9d1f6b8a72\tDuration: 41.02 ms\tBilled Duration: 42 ms\tMemory Size: 128 MB\tMax Memory Used: 45 MB\tStatus: error\tError Type: Unhandled\n",
            "ingestionTime": 1786732808710,
            "eventId": "38234501234567890123456789012345678904"
        },
        {
            "logStreamName": "2026/08/14/[$LATEST]4a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d",
            "timestamp": 1786732814977,
            "message": "Invalid manifest manifests/year=2026/month=08/batch/17-shipment-4473-manifest.txt: ['weightKg must be numeric']\n",
            "ingestionTime": 1786732815488,
            "eventId": "38234501234567890123456789012345678905"
        },
        {
            "logStreamName": "2026/08/14/[$LATEST]4a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d",
            "timestamp": 1786732815012,
            "message": "REPORT RequestId: c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58\tDuration: 35.88 ms\tBilled Duration: 36 ms\tMemory Size: 128 MB\tMax Memory Used: 44 MB\tStatus: error\tError Type: Unhandled\n",
            "ingestionTime": 1786732815499,
            "eventId": "38234501234567890123456789012345678906"
        }
    ],
    "searchedLogStreams": [
        {
            "logStreamName": "2026/08/14/[$LATEST]4a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d",
            "searchedCompletely": true
        }
    ]
}

Six events, two for each of the three failed invocations — the validation line that explains the "why," and the REPORT line that carries the requestId — in the same order they happened (positions 5, 12, 17 of lesson 3's batch).

Step 2 — `jq`, really running, in three steps

From here on, everything ran for real: jq doesn't depend on LocalStack, it processes the file you just saved with none of this environment's limitations.

Step 2a — The three requestIds, extracted from the REPORT lines with a regular expression:

jq -r '.events[] | select(.message | contains("Status: error")) | .message | capture("RequestId: (?<requestId>[a-f0-9-]+)") | .requestId' observability/manifest-log-events.json

What to expect (literal — run with jq in this environment):

a47f3e21-8b6a-4c9d-9f12-3d8e7b1a2c44
f3c91a08-2e4d-4b7f-8a3c-5e9d1f6b8a72
c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58

Step 2b — The three validation reasons, extracted from the Invalid manifest lines:

jq -r '.events[] | select(.message | contains("Invalid manifest")) | .message' observability/manifest-log-events.json

What to expect (literal):

Invalid manifest manifests/year=2026/month=08/batch/05-shipment-4471-manifest.txt: ['missing required field: weightKg']

Invalid manifest manifests/year=2026/month=08/batch/12-shipment-4472-manifest.txt: ['missing required field: carrier']

Invalid manifest manifests/year=2026/month=08/batch/17-shipment-4473-manifest.txt: ['weightKg must be numeric']

(The blank lines are the line break print() itself leaves at the end of each message — a real detail of the log format, not an error in this command.)

Step 2c — Combining both into a single object per failed invocation:

jq '
  (.events | map(select(.message | contains("Invalid manifest"))) | map(.message | rtrimstr("\n"))) as $reasons
  | (.events | map(select(.message | contains("Status: error"))) | map(.message | capture("RequestId: (?<id>[a-f0-9-]+)").id)) as $ids
  | [range(0; $ids | length) | {requestId: $ids[.], reason: $reasons[.]}]
' observability/manifest-log-events.json

What to expect (literal):

[
  {
    "requestId": "a47f3e21-8b6a-4c9d-9f12-3d8e7b1a2c44",
    "reason": "Invalid manifest manifests/year=2026/month=08/batch/05-shipment-4471-manifest.txt: ['missing required field: weightKg']"
  },
  {
    "requestId": "f3c91a08-2e4d-4b7f-8a3c-5e9d1f6b8a72",
    "reason": "Invalid manifest manifests/year=2026/month=08/batch/12-shipment-4472-manifest.txt: ['missing required field: carrier']"
  },
  {
    "requestId": "c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58",
    "reason": "Invalid manifest manifests/year=2026/month=08/batch/17-shipment-4473-manifest.txt: ['weightKg must be numeric']"
  }
]

This is, literally, the detail Errors: 3.0 couldn't give: three different requestIds, three different reasons, each traceable to the exact validate_manifest() line that produced it. Save requestId c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58 — position 17 — for lesson 5, which follows it with a complete trace.

Common mistakes

Confusing "representative" (the CloudWatch query) with "representative" (the jq result) — they're two different things in this lesson. What happens: someone, reading that Step 1 is representative, assumes Step 2 is too. How to spot it: if your explanation of this lesson says "all of this is representative" without distinguishing step 1 from step 2. How to fix it: Step 1 (the CloudWatch query) is representative because it depends on LocalStack, which doesn't start in this environment. Step 2 (jq over the already-saved file) is completely real — jq doesn't depend on any AWS service, it ran for real, and the manifest-log-events.json file Step 1 produces is the only connection point between the two.

Writing a different jq expression and expecting this lesson's exact same result (not understanding what capture does). What happens: someone writes their own expression to extract the requestId — for example, with split(": ") instead of capture() — and the result doesn't match this lesson's. How to spot it: if your Step 2a output has extra text (like RequestId: included) instead of just the UUID. How to fix it: capture("RequestId: (?<requestId>[a-f0-9-]+)") uses a regular expression with a named group (?<requestId>) that extracts only the part matching [a-f0-9-]+ — hexadecimal characters and hyphens — after the literal text "RequestId: ". Any expression that doesn't isolate exactly that pattern is going to include too much or too little text.

Treating the three extracted requestIds as if they were arbitrary, with no connection to lesson 3 (losing the fixed-batch thread). What happens: someone processes this lesson in isolation, without noticing the three requestIds correspond exactly to positions 5, 12, and 17 of the previous lesson's batch. How to spot it: if you can't say, from memory, which position of the 20-invocation batch corresponds to c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58. How to fix it: lesson 1 of this module already warned about this — this module runs over a single fixed batch, read through three instruments. The requestId c8e42d15-... isn't new data; it's the same failed invocation number 17 from lesson 3, now precisely identified.

Exercises

Exercise 1 — Write, without running it yet, a jq command that counts how many failed invocations are in the file, extracting no detail at all. Use the same manifest-log-events.json file.

See solution

jq '[.events[] | select(.message | contains("Status: error"))] | length' observability/manifest-log-events.json

This command produces 3 — the same number Errors: 3.0 already gave in lesson 3, now confirmed by directly counting the REPORT lines with Status: error in the logs, instead of reading a CloudWatch aggregate. It's a useful way to independently verify the metric and the log tell the same story.

Exercise 2 — Modify Step 2b's command so that, instead of printing the full message, it extracts only the list of errors inside brackets (for example, ['missing required field: weightKg']). Hint: use capture() with a regular expression that captures everything between [ and ].

See solution

jq -r '.events[] | select(.message | contains("Invalid manifest")) | .message | capture("(?<errors>\\[.*\\])") | .errors' observability/manifest-log-events.json

With this lesson's file, it produces:

['missing required field: weightKg']
['missing required field: carrier']
['weightKg must be numeric']

The pattern \\[.*\\] (the brackets escaped with \\, because they're special characters in a regular expression) captures everything between the first [ and the last ] in the line — exactly the text representation of the Python errors list validate_manifest() built.

Exercise 3 — Explain why Step 2c uses range(0; $ids | length) instead of, for example, zip($ids; $reasons) (if your jq version has it) or any other way of combining two lists. What guarantees $ids[.] and $reasons[.] always correspond to the same invocation, at the same index?

See solution

The guarantee doesn't come from range() itself — it comes from the fact that both lists, $ids and $reasons, get built by filtering the same .events array in the same order it appears in the file, and that file, in turn, preserves the real chronological order CloudWatch Logs received each line in (position 5 before 12, before 17). Since each failed invocation wrote exactly one Invalid manifest line followed, shortly after, by exactly one REPORT line with Status: error, filtering each type of line separately preserves the same relative order in both resulting lists — the element at index 0 of $ids corresponds, by construction, to the element at index 0 of $reasons. This positional correspondence is fragile in general (it would break if some invocation failed without leaving both lines, or if chronological order weren't preserved) — one more reason not to treat this trick as a universal jq technique, but as something valid specifically for this file's structure.

Summary and next step

This lesson opened up the three failures lesson 3 could only count: with a representative CloudWatch Logs query (same reason as always: with no LOCALSTACK_AUTH_TOKEN, LocalStack doesn't start here) you saved the real detail of the six relevant lines, and with jq — really run, in three progressive steps — you extracted the three requestIds, the three validation reasons, and combined them into a single object per failed invocation. You also confirmed the error-line count (3) matches exactly the previous lesson's Errors: 3.0 — the metric and the log tell the same story, at different levels of detail.

Before moving on you should be able to: explain the difference between what's representative (Step 1) and what's real (Step 2) in this lesson; recite the three extracted requestIds, with their matching failure reason; and modify a simple capture() expression to extract a different fragment of a log message.

Lesson 5 takes position 17's requestId — c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58 — and follows it with a real OpenTelemetry trace, showing at what exact step of the upload → Lambda → DynamoDB flow that invocation broke.

Resources

jq Manual — capture — this lesson's function for extracting data with named regular expressions.
jq Manual — select — the filter that separates the REPORT lines from the Invalid manifest lines in this lesson.
AWS CLI — logs filter-log-events Command Reference — the official reference for Step 1's representative command, including the ?"..." ?"..." pattern syntax.
LocalStack Docs — CloudWatch Logs — confirmation of the Hobby plan for the logs service.
This same repository, Module 3, lesson 3 (03-hands-on-real-metrics-from-the-inherited-lambda.md) — the fixed batch whose three failures this lesson opens up in detail.

← Previous3. Hands-on: real metrics from the inherited Lambda Next →5. Hands-on: traces with OpenTelemetry and Jaeger

4. Hands-on: real logs with `jq`

Description

Connection to the module

Step 1 — Querying the real logs for the three failures

Step 2 — jq, really running, in three steps

Common mistakes

Exercises

Summary and next step

Resources

Step 2 — `jq`, really running, in three steps