Module 3: Observability As Sli Input

5. Hands-on: traces with OpenTelemetry and Jaeger

Next →

Description

This lesson follows "the car" from lesson 2's analogy: batch invocation number 17 — the same one lesson 4 already identified by requestId with jq — seen end to end through its flow's three steps: the S3 upload, processing in process-shipment-manifest, and the attempt (or not) to write to Shipments. Unlike the previous two lessons, everything that follows ran for real: a real Jaeger v2 container, brought up with docker run in this very environment; a real Python script, with the OpenTelemetry SDK, that sent two complete traces over OTLP/HTTP; and Jaeger's API, really queried, confirming both traces arrived with the exact structure the script produced.

Connection to the module

Lesson 2 promised traces answer "at what exact step of the flow did a specific invocation break?" This lesson demonstrates it with two real traces: one from the successful path (batch position 1, shipment 4471), and one from the invocation that failed at position 17 — the same requestId c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58 lesson 4 extracted from the logs. Lesson 6 extends the same docker-compose.yml this lesson creates, adding Prometheus and Grafana.

Step 1 — Why Jaeger v2, not all-in-one, and why not X-Ray

Before the code, two decisions worth understanding, not just copying.

Jaeger v2, image jaegertracing/jaeger, never jaegertracing/all-in-one. Jaeger v1 reached its end of life on December 31, 2025 — the all-in-one image still exists on Docker Hub, but it's discontinued, and the container itself warns about it on startup. The correct, current image for a complete local deployment is jaegertracing/jaeger (v2), built on the OpenTelemetry Collector framework — confirmed in this environment, version 2.20.0.

X-Ray, named by contrast, never run in this guide. AWS X-Ray would be AWS's native alternative for distributed tracing — but LocalStack's official documentation is explicit: "Included in Plans: Ultimate," absent from the Hobby plan the rest of this guide uses. This lesson uses OpenTelemetry + Jaeger instead, precisely because it teaches the same concept — a distributed trace, with parent-child spans — with an open-source tool that does run, with no dependency on a paid tier.

Step 2 — Bringing up Jaeger v2 for real

Create observability/docker-compose.yml in andes-cargo-infra/ (this lesson opens it with a single service; lesson 6 adds two more):

services:
  jaeger:
    image: jaegertracing/jaeger:2.20.0
    container_name: andes-cargo-jaeger
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
cd observability
docker compose up -d jaeger

What to expect (literal — verified in this environment):

 Network observability_default  Created
 Container andes-cargo-jaeger  Started

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:16686/

What to expect (literal):

200

Jaeger's UI responds before any trace exists — the next step is what's going to fill it.

Step 3 — The instrumentation script

Create observability/instrument_manifest_flow.py:

# instrument_manifest_flow.py
# Sends two real traces of the upload -> process-shipment-manifest -> DynamoDB flow to
# Jaeger v2 over OTLP/HTTP. Trace 1 is the successful path (shipment 4471). Trace 2 is
# invocation #17 from the fixed batch (lessons 3.3/3.4) -- a manifest with a non-numeric
# weightKg, rejected by validate_manifest() -- shown as a real ERROR span.
# Span content (which shipment, which request id, which error) is fixed and literal;
# only the timestamps Jaeger assigns on ingest vary by run.

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import Status, StatusCode

JAEGER_OTLP_ENDPOINT = "http://localhost:4318/v1/traces"


def build_tracer(service_name):
    provider = TracerProvider(resource=Resource.create({"service.name": service_name}))
    provider.add_span_processor(
        BatchSpanProcessor(OTLPSpanExporter(endpoint=JAEGER_OTLP_ENDPOINT))
    )
    return provider


app_provider = build_tracer("andes-cargo-app-server")
lambda_provider = build_tracer("process-shipment-manifest")
dynamodb_provider = build_tracer("dynamodb")

app_tracer = app_provider.get_tracer("andes-cargo.upload")
lambda_tracer = lambda_provider.get_tracer("andes-cargo.manifest-processor")
dynamodb_tracer = dynamodb_provider.get_tracer("andes-cargo.shipments-table")


def trace_successful_manifest():
    """Trace 1: shipment 4471 (batch position 1), the full upload -> Lambda -> DynamoDB
    path succeeds."""
    with app_tracer.start_as_current_span("shipment-manifest-upload") as upload_span:
        upload_span.set_attribute("shipment.id", "4471")
        upload_span.set_attribute("s3.bucket", "andes-cargo-shipment-docs")
        upload_span.set_attribute(
            "s3.key", "manifests/year=2026/month=08/batch/01-shipment-4471-manifest.txt"
        )

        with lambda_tracer.start_as_current_span("process-shipment-manifest") as fn_span:
            fn_span.set_attribute("faas.name", "process-shipment-manifest")
            fn_span.set_attribute("aws.region", "us-east-1")
            fn_span.set_attribute("request.id", "req-0001-success")

            with dynamodb_tracer.start_as_current_span("dynamodb-put-item") as db_span:
                db_span.set_attribute("db.system", "dynamodb")
                db_span.set_attribute("db.name", "Shipments")
                db_span.set_attribute("aws.dynamodb.table_names", "Shipments")
                db_span.set_status(Status(StatusCode.OK))

            fn_span.set_status(Status(StatusCode.OK))
        upload_span.set_status(Status(StatusCode.OK))


def trace_malformed_manifest():
    """Trace 2: invocation #17 of the fixed batch (lessons 3.3/3.4) -- the manifest for
    shipment 4473 has a non-numeric weightKg ('heavy' instead of a number), which
    validate_manifest() (aws-serverless-and-containers-guide, Module 2) rejects. The
    Lambda span ends in ERROR before any DynamoDB span is opened -- the trace shows
    exactly where the flow broke, the same way the log line in lesson 3.4 does."""
    with app_tracer.start_as_current_span("shipment-manifest-upload") as upload_span:
        upload_span.set_attribute("shipment.id", "4473")
        upload_span.set_attribute("s3.bucket", "andes-cargo-shipment-docs")
        upload_span.set_attribute(
            "s3.key", "manifests/year=2026/month=08/batch/17-shipment-4473-manifest.txt"
        )

        with lambda_tracer.start_as_current_span("process-shipment-manifest") as fn_span:
            fn_span.set_attribute("faas.name", "process-shipment-manifest")
            fn_span.set_attribute("aws.region", "us-east-1")
            fn_span.set_attribute("request.id", "c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58")
            error_message = "manifest validation failed: ['weightKg must be numeric']"
            error = ValueError(error_message)
            fn_span.record_exception(error)
            fn_span.set_status(Status(StatusCode.ERROR, error_message))

        upload_span.set_status(Status(StatusCode.ERROR, "manifest processing failed"))


if __name__ == "__main__":
    trace_successful_manifest()
    trace_malformed_manifest()
    for provider in (app_provider, lambda_provider, dynamodb_provider):
        provider.force_flush()
        provider.shutdown()
    print("2 traces sent to", JAEGER_OTLP_ENDPOINT)

Notice trace 2: it never opens a dynamodb-put-item span — exactly like the real flow, where write_shipment_record() never runs if validate_manifest() already rejected the manifest. That third span's absence is, in itself, the evidence of where the flow broke — no additional field is needed to see it.

Step 4 — Running the script

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
python3 observability/instrument_manifest_flow.py

What to expect (literal — run in this environment, with OpenTelemetry Python SDK 1.44.0):

2 traces sent to http://localhost:4318/v1/traces

Step 5 — Confirming the traces in Jaeger, for real

curl -s http://localhost:16686/api/services

What to expect (literal):

{"data":["process-shipment-manifest","dynamodb","jaeger","andes-cargo-app-server"],"total":4,"limit":0,"offset":0,"errors":null}

Three services of Andes Cargo's own (andes-cargo-app-server, process-shipment-manifest, dynamodb) plus jaeger, the collector's own internal service — proof the script's three span sources all arrived, each with its own service name.

curl -s "http://localhost:16686/api/traces?service=andes-cargo-app-server&limit=10"

What to expect (literal summary of the response — the traceIDs and durations in microseconds are your variable value, assigned by Jaeger at ingest time; the structure, span names, attributes, and statuses are literal):

=== traceID: 64441cce3ff83431f92061452f78f715 ===
  service=andes-cargo-app-server      op=shipment-manifest-upload     status=ERROR desc="manifest processing failed"
    attrs: s3.bucket=andes-cargo-shipment-docs, s3.key=manifests/year=2026/month=08/batch/17-shipment-4473-manifest.txt, shipment.id=4473
  service=process-shipment-manifest   op=process-shipment-manifest    status=ERROR desc="manifest validation failed: ['weightKg must be numeric']"
    attrs: aws.region=us-east-1, faas.name=process-shipment-manifest, request.id=c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58

=== traceID: d7717881159f0132c7b71704b930da16 ===
  service=andes-cargo-app-server      op=shipment-manifest-upload     status=OK
    attrs: s3.bucket=andes-cargo-shipment-docs, s3.key=manifests/year=2026/month=08/batch/01-shipment-4471-manifest.txt, shipment.id=4471
  service=process-shipment-manifest   op=process-shipment-manifest    status=OK
    attrs: aws.region=us-east-1, faas.name=process-shipment-manifest, request.id=req-0001-success
  service=dynamodb                    op=dynamodb-put-item            status=OK
    attrs: aws.dynamodb.table_names=Shipments, db.name=Shipments, db.system=dynamodb

Two traces, exactly as the script built them. Trace 64441cce... has two spans, both ERROR — none from DynamoDB; trace d7717881... has three, all three OK. The error trace's request.id, c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58, is exactly the same one lesson 4 extracted with jq from the logs — the same invocation, now seen end to end.

If you open http://localhost:16686 in a browser and search by service andes-cargo-app-server, the UI shows both traces with the same detail: the successful trace with three nested horizontal bars (one per span, with DynamoDB's completely inside the Lambda's), and the error trace with only two, the second one marked in red.

Common mistakes

Using the jaegertracing/all-in-one image because "it's the one that shows up first in an old tutorial" (outdated knowledge). What happens: someone searches "Jaeger docker" and finds examples from a few years ago using all-in-one. How to spot it: if your docker-compose.yml has jaegertracing/all-in-one instead of jaegertracing/jaeger. How to fix it: all-in-one corresponds to Jaeger v1, which reached its end of life on December 31, 2025 — the container still starts (which is why the mistake is easy to miss), but it prints an explicit end-of-life warning. This lesson uses jaegertracing/jaeger (v2) precisely to avoid building on a discontinued base.

Expecting to see the error trace also under dynamodb when querying api/services about that specific trace (assuming every planned span always opens). What happens: someone looks for trace 64441cce... in the DynamoDB view and doesn't find it, and assumes something failed in the script. How to spot it: if your question is "why doesn't the error trace show up when I filter by dynamodb service?" How to fix it: this is the correct behavior, not an error — trace_malformed_manifest() never opens a dynamodb-put-item span, precisely because the real flow never gets to attempt the write when validation fails first. That trace not appearing under the dynamodb service is, itself, the evidence of where the flow broke.

Comparing this lesson's traceID with the one you get running the script yourself, and assuming something's wrong if they don't match (not distinguishing what's literal from what varies). What happens: someone runs the script, gets a different traceID than this lesson's, and thinks they did something wrong. How to spot it: if your worry is that the traceID doesn't match this lesson's digit for digit. How to fix it: the traceID is generated by Jaeger (or the SDK, depending on configuration) at ingest time — it's, by design, random, just like a timestamp. What's literal and should match exactly is the structure: two traces, the first with two spans in ERROR, the second with three spans in OK, the same service names, the same attributes, the same request.id.

Exercises

Exercise 1 — Explain, using the script's code, why trace_malformed_manifest() calls fn_span.record_exception(error) before fn_span.set_status(...), and not the other way around. Would anything change functionally if the order were reversed?

See solution

Functionally, the order between these two specific calls doesn't change the final result — both are independent operations on the same span, and neither depends on the other's result. What does matter is that both happen inside the with lambda_tracer.start_as_current_span(...) block, before the span closes — an already-closed span can't receive either a recorded exception or a status change. The order chosen in the script (record_exception first, set_status after) follows the most common convention in OpenTelemetry examples: recording the specific event (the exception, with its message and type) before declaring the span's aggregate result (its final status) — a logical sequence of "this is what happened" followed by "that's why the result is this," even though the SDK itself doesn't require that order.

Exercise 2 — Design, without writing code yet, a third trace for an invocation that does pass validation but fails when writing to DynamoDB (for example, if the table were throttling writes). How many spans would it have, and which one would end in ERROR?

See solution

It would have three spans — shipment-manifest-upload, process-shipment-manifest, dynamodb-put-item — the same number as the successful trace, because the flow does get to open the DynamoDB span (unlike this lesson's error trace, where validate_manifest() breaks the flow earlier). The difference would be in the status: dynamodb-put-item would end in ERROR (with its own recorded exception, for example a capacity throttle), and that error status would propagate upward — process-shipment-manifest and shipment-manifest-upload would also end in ERROR, even though their own internal code didn't directly raise the exception. This exercise shows that a trace's span count says as much as its status: a three-span trace with the third one red tells a different story than a two-span trace, even though both end up being, in aggregate, a failed invocation.

Exercise 3 — Explain, in one sentence, what evidence from this lesson would be missing if you'd only run lesson 3 (metrics) and lesson 4 (logs), without this lesson 5. Be specific about what question would go unanswered.

See solution

The evidence of where, within the multi-step flow, invocation 17 stopped would be missing. Lesson 3 (metrics) confirms that there were 3 errors; lesson 4 (logs) confirms which ones and why, with the exact validation message — but neither, on its own, shows the time and hierarchical relationship between the flow's steps: that the S3 upload did happen, that the Lambda did start processing, and that the DynamoDB write attempt never even opened. That absence — a third step that simply isn't there, instead of being there and having failed — is exactly the kind of evidence only a trace can show with precision, the question lesson 2 specifically assigned to this third pillar.

Summary and next step

This lesson ran, end to end, this module's third observability pillar: a real Jaeger v2 container (jaegertracing/jaeger:2.20.0, never all-in-one, discontinued), an instrumentation script with the OpenTelemetry Python SDK (1.44.0) that sent two real traces over OTLP/HTTP, and a confirmation, via Jaeger's API, that both arrived with the exact structure the script defined: a three-span trace, all OK (the successful path), and a two-span trace, both ERROR (invocation 17, the same one lesson 4 already identified by requestId), with no DynamoDB span at all — visual proof the flow broke before any write attempt.

Before moving on you should be able to: explain why this guide uses jaegertracing/jaeger and not all-in-one; explain why the error trace has two spans instead of three; and connect this lesson's request.id with the requestId lesson 4 extracted from the logs.

Lesson 6 extends the same observability/docker-compose.yml this lesson created, adding Prometheus and Grafana — the stack the market asks for more often than native CloudWatch, per the evidence that lesson is going to cite.

Resources

  1. Jaeger — Getting Started — the official Jaeger v2 guide, including the warning about all-in-one.
  2. GitHub — jaegertracing/jaeger — the official repository, the source for the 2.20.0 version used in this lesson.
  3. OpenTelemetry — Python — the official documentation for the SDK instrument_manifest_flow.py uses.
  4. LocalStack Docs — X-Ray — confirmation that X-Ray requires the Ultimate plan, the exact reason this lesson doesn't run it.
  5. This same repository, Module 3, lesson 4 (04-hands-on-real-logs-with-jq.md) — the requestId c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58 this lesson follows with a complete trace.
← Previous4. Hands-on: real logs with `jq`Next →6. Hands-on: Prometheus and Grafana, the stack the market asks for