6. Hands-on: Prometheus and Grafana, the stack the market asks for

Description

Up to this lesson, every observability pillar in this module read the same fixed batch with an AWS tool — CloudWatch (representative) for metrics and logs, with Jaeger as an open-source alternative only for traces (because X-Ray isn't available on the Hobby plan). This lesson adds a second, complete, real, running alternative for the metrics pillar: Prometheus and Grafana, the stack this ecosystem's market research found cited by name, more often than native CloudWatch, in real job postings.

Everything that follows ran for real: two more containers (Prometheus v3.13.2, Grafana OSS 13.1.3) added to the same docker-compose.yml lesson 5 already created, a real Python exporter that exposes the same 20/3 batch as Prometheus metrics, and a Grafana panel, created with Grafana's real API, that queries those metrics and returns the same numbers you already know from lesson 3.

Connection to the module

Lesson 2 made clear metrics are the only pillar that feeds compute_sli() directly. This lesson builds a second real source for those same metrics — not CloudWatch this time, but Prometheus — and lesson 7 is going to use exactly these numbers, extracted with a real PromQL query, as Module 2's calculator's input.

Step 1 — The market evidence: why this tool, not just CloudWatch

This ecosystem's market validation (src/paths/aws-cloud-ecosystem/VALIDACION.md) is specific about this, with figures, not an opinion:

"The market asks for a concrete infrastructure tool in ~7 of 13 postings (MediaStream 'Prometheus, Grafana, and ELK stack'; Dev.Pro 'New Relic, Datadog'; Randstad 'CloudWatch'; EarnIn 'Datadog')."

Of the four tools explicitly named in those postings, Prometheus/Grafana shows up once — the same as CloudWatch (Randstad) — but alongside New Relic and Datadog (Dev.Pro, EarnIn) it confirms a broader pattern: almost no posting in this market expects you to know only your cloud provider's native tool. Knowing how to read a metric in CloudWatch (lessons 3 and 4 of this module) and knowing how to read the same metric in an open-source stack that's portable across providers (this lesson) are two skills the market asks for separately, not one that replaces the other.

Step 2 — Extending the docker-compose.yml

Lesson 5 left observability/docker-compose.yml with a single service, jaeger. This lesson adds two more:

services:
  jaeger:
    image: jaegertracing/jaeger:2.20.0
    container_name: andes-cargo-jaeger
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP

  prometheus:
    image: prom/prometheus:v3.13.2
    container_name: andes-cargo-prometheus
    extra_hosts:
      - "host.docker.internal:host-gateway"   # linux: maps the host inside the container
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:13.1.3
    container_name: andes-cargo-grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=andescargo

extra_hosts: host.docker.internal:host-gateway is the line that makes this docker-compose.yml work the same on Linux as on Docker Desktop (macOS/Windows) — without it, host.docker.internal only resolves automatically on Docker Desktop.

Also create observability/prometheus.yml, the file the volume above mounts inside the container:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "manifest-processor"
    static_configs:
      - targets: ["host.docker.internal:8000"]

The second job_name is the one that matters: it tells Prometheus to scrape an exporter running on the host's port 8000 — exactly the one you build in the next step.

Step 3 — The exporter: the same 20/3 batch, as Prometheus metrics

# manifest_metrics_exporter.py
# Exposes the SAME fixed batch of process-shipment-manifest invocations from the M3.3
# CloudWatch walkthrough as real Prometheus metrics. Deterministic: no random, no datetime.now()
# driving any counted value -- the batch below is a fixed constant, replayed once at startup.

import time
from prometheus_client import Counter, start_http_server

# --- The fixed batch: 20 invocations, indices 5, 12 and 17 malformed on purpose. ---
# Same 20 events as observability/upload_manifest_batch.py (lesson 3.3) and the same 3
# failed request IDs as observability/manifest-log-events.json (lesson 3.4).
BATCH = [
    (1, "good"), (2, "good"), (3, "good"), (4, "good"), (5, "bad"),
    (6, "good"), (7, "good"), (8, "good"), (9, "good"), (10, "good"),
    (11, "good"), (12, "bad"), (13, "good"), (14, "good"), (15, "good"),
    (16, "good"), (17, "bad"), (18, "good"), (19, "good"), (20, "good"),
]

manifest_invocations_total = Counter(
    "manifest_invocations_total",
    "Total invocations of process-shipment-manifest observed by this exporter",
)
manifest_errors_total = Counter(
    "manifest_errors_total",
    "Invocations of process-shipment-manifest that ended in an unhandled exception",
)


def replay_batch():
    for _index, outcome in BATCH:
        manifest_invocations_total.inc()
        if outcome == "bad":
            manifest_errors_total.inc()


if __name__ == "__main__":
    start_http_server(8000)
    replay_batch()
    print("manifest_metrics_exporter listening on :8000/metrics")
    print(f"Replayed {len(BATCH)} invocations, {sum(1 for _, o in BATCH if o == 'bad')} errors.")
    while True:
        time.sleep(3600)

This exporter doesn't call awslocal or depend on LocalStack at all — it's a real Python HTTP server, with two real Prometheus counters, exposing lesson 3's same deterministic batch from a completely different source. It runs on the host, not inside Docker, precisely so Prometheus (inside Docker) can scrape it through host.docker.internal.

Step 4 — Bringing everything up, for real

pip install prometheus_client
python3 observability/manifest_metrics_exporter.py &

What to expect (literal):

manifest_metrics_exporter listening on :8000/metrics
Replayed 20 invocations, 3 errors.

cd observability
docker compose up -d

What to expect (literal — verified in this environment):

 Container andes-cargo-jaeger  Running
 Container andes-cargo-prometheus  Running
 Container andes-cargo-grafana  Started

curl -s http://localhost:9090/-/ready
curl -s http://localhost:3000/api/health

What to expect (literal):

Prometheus Server is Ready.

{"database":"ok","version":"13.1.3","commit":"45a27d64b64a82d666b06aa5c5bb3521587edb0d"}

Step 5 — Confirming the scraping, with real PromQL

curl -s 'http://localhost:9090/api/v1/query?query=manifest_invocations_total'

What to expect (literal — the value 1786737874.434 is the query's Unix timestamp, your variable value; "20" is literal):

{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "manifest_invocations_total",
                    "instance": "host.docker.internal:8000",
                    "job": "manifest-processor"
                },
                "value": [1786737874.434, "20"]
            }
        ]
    }
}

curl -s 'http://localhost:9090/api/v1/query?query=manifest_errors_total/manifest_invocations_total'

What to expect (literal):

{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {"instance": "host.docker.internal:8000", "job": "manifest-processor"},
                "value": [1786737874.51, "0.15"]
            }
        ]
    }
}

0.15 — a 15% error rate, exactly 3/20, the same proportion Errors: 3.0 over Invocations: 20.0 already gave in lesson 3, now confirmed with a real PromQL query against a source completely different from CloudWatch.

Step 6 — The Grafana panel, really created with the API

First, the data source:

curl -s -u admin:andescargo -X POST http://localhost:3000/api/datasources \
  -H "Content-Type: application/json" \
  -d '{"name": "Prometheus", "type": "prometheus", "url": "http://prometheus:9090", "access": "proxy", "isDefault": true}'

What to expect (literal, "id" and "uid" are assigned by Grafana when the resource is created):

{"datasource": {"id": 1, "uid": "afv6sh2yl2ww0a", "name": "Prometheus", "type": "prometheus", "url": "http://prometheus:9090", "isDefault": true, ...}, "id": 1, "message": "Datasource added", "name": "Prometheus"}

Then, the panel — success/error for the same batch, plus an error-rate gauge:

curl -s -u admin:andescargo -X POST http://localhost:3000/api/dashboards/db \
  -H "Content-Type: application/json" \
  -d '{
    "dashboard": {
      "uid": "andes-cargo-manifest-sli",
      "title": "Andes Cargo -- process-shipment-manifest SLI input",
      "panels": [
        {
          "id": 1, "type": "stat", "title": "Invocations vs errors (fixed batch)",
          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
          "targets": [
            {"expr": "manifest_invocations_total - manifest_errors_total", "legendFormat": "good", "refId": "A"},
            {"expr": "manifest_errors_total", "legendFormat": "errors", "refId": "B"}
          ]
        },
        {
          "id": 2, "type": "gauge", "title": "Observed error rate",
          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
          "targets": [{"expr": "manifest_errors_total / manifest_invocations_total", "legendFormat": "error rate", "refId": "A"}],
          "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "max": 1}}
        }
      ],
      "schemaVersion": 39
    },
    "overwrite": true
  }'

What to expect (literal):

{"folderUid": "", "id": 3189596175294464, "slug": "andes-cargo-process-shipment-manifest-sli-input", "status": "success", "uid": "andes-cargo-manifest-sli", "url": "/d/andes-cargo-manifest-sli/andes-cargo-process-shipment-manifest-sli-input", "version": 1}

"status": "success" — the dashboard was really created, on the real Grafana instance running in this environment, with two panels: one showing "good" and "errors" as separate series, another showing the error rate as a gauge between 0 and 1.

Confirming the data the panel shows, by querying the data source the same path Grafana's own UI uses internally:

curl -s -u admin:andescargo "http://localhost:3000/api/datasources/proxy/uid/afv6sh2yl2ww0a/api/v1/query?query=manifest_invocations_total%20-%20manifest_errors_total"

What to expect (literal):

{"status": "success", "data": {"resultType": "vector", "result": [{"metric": {"instance": "host.docker.internal:8000", "job": "manifest-processor"}, "value": [1786737907.6, "17"]}]}}

"17" — Grafana's "Invocations vs errors" panel shows exactly the same number of good events lesson 3 already calculated by hand (20 - 3 = 17), this time read end to end through a Python exporter, a Prometheus scrape, and a Grafana query — three layers, same data.

Common mistakes

Forgetting extra_hosts: host.docker.internal:host-gateway on Linux (assuming host.docker.internal always behaves the same way). What happens: on Docker Desktop (macOS/Windows), host.docker.internal resolves automatically with no extra configuration; on Linux (plain Docker Engine), it doesn't. How to spot it: if Prometheus reports the manifest-processor target as down, with a name-resolution error, on a Linux machine. How to fix it: docker-compose.yml's extra_hosts line explicitly maps host.docker.internal to the host's real gateway, without relying on Docker Desktop to do it for you — that's why this docker-compose.yml includes it from the start, instead of assuming a single operating system.

Running the exporter after bringing up Prometheus, and expecting the scrape to recover on its own (not understanding the scraping interval). What happens: someone runs docker compose up first, and only afterward starts manifest_metrics_exporter.py; Prometheus's first scrape attempt fails because port 8000 isn't responding yet. How to spot it: if the first curl to /api/v1/query returns no result, or returns an empty value. How to fix it: it's not a permanent error — with scrape_interval: 15s, Prometheus automatically retries on the next cycle, and the target flips to up as soon as the exporter responds. This lesson presents the correct order (exporter first, docker compose up after) precisely to avoid this wait.

Confusing the "id" Grafana assigns to the data source with the "uid" when building the proxy URL (mixing up two identifiers). What happens: someone copies the number "id": 1 from Step 6's response and uses it in the URL /api/datasources/proxy/uid/1/..., and the command fails. How to spot it: if your query to the data source's proxy returns a "datasource not found" error. How to fix it: Grafana assigns two different identifiers to every resource — a numeric internal "id" and an alphanumeric "uid", meant to be stable across instances; the /api/datasources/proxy/uid/<uid>/... path specifically needs the second one, not the first. In this lesson, that value is afv6sh2yl2ww0a — the one Step 6's first command's response already showed.

Exercises

Exercise 1 — Without running anything, write the PromQL expression that would show the batch's percentage of good events (not errors). Verify that, with this lesson's numbers, it gives 85 (if you multiply by 100) or 0.85 (if you leave it as a fraction).

(manifest_invocations_total - manifest_errors_total) / manifest_invocations_total

Exercise 2 — Explain why this exporter uses Counter (a counter that only goes up) and not Gauge (a value that can go up or down) for manifest_invocations_total. What would happen if Gauge were used instead, by mistake?

Exercise 3 — Compare, in a short table of your own making, this data's complete path in CloudWatch (lesson 3) versus its complete path in Prometheus/Grafana (this lesson). Name each layer the number 17 (good events) passes through on each path.

Summary and next step

This lesson ran, end to end, a second real metrics source for the same fixed 20-invocation batch: Prometheus v3.13.2 and Grafana 13.1.3, added to lesson 5's docker-compose.yml, a real Python exporter (manifest_metrics_exporter.py) exposing the same counters CloudWatch already showed representatively, and a Grafana panel — created with Grafana's real API, not described in prose — confirming the same 17 good events you already knew. Step 1's market quote justified why this second path matters: Prometheus/Grafana shows up explicitly named in real postings, alongside other third-party observability tools, at the same frequency as native CloudWatch.

Before moving on you should be able to: explain the difference between extra_hosts: host.docker.internal:host-gateway and Docker Desktop's automatic behavior; write from memory this batch's error-rate PromQL expression; and name the four layers of the data's complete path on the Prometheus/Grafana side.

Lesson 7 closes this module's thread: it takes exactly this lesson's numbers — 20 invocations, 3 errors, extracted with real PromQL — and feeds them to error_budget_calculator.py, Module 2's calculator, for the first time with real telemetry instead of a sample dataset.

Resources

1. src/paths/aws-cloud-ecosystem/VALIDACION.md — the exact market quote ("Prometheus, Grafana, and ELK stack") that justifies this lesson.
2. Prometheus — Docker Hub — the official image, version v3.13.2, used in this lesson.
3. Grafana — Download OSS — the official image, version 13.1.3.
4. Prometheus — Querying basics (PromQL) — the official reference for this lesson's PromQL expressions.
5. Grafana HTTP API — Data source and Dashboard — the official reference for the two endpoints used in Step 6.
6. prometheus/client_python — Counter — the official documentation for the metric type manifest_metrics_exporter.py uses.