Module 4: Alerting On Error Budget Burn Rate

4. Hands-on: the real rule in Alertmanager

Next →

Description

Lesson 3 prototyped the alert decision in pure Python, with no infrastructure around it. This lesson builds this module's second engine: the same decision — Table 5-8's three severities, the same two scenarios ("bad week," "normal") — but this time really evaluated by Prometheus, against metrics scraped from a real exporter, with alerts routed to a real receiver by Alertmanager. Everything that follows ran for real, in Docker, in this very environment: Prometheus v3.13.2 and Alertmanager v0.33.1, brought up with docker compose, evaluating a real alert rule, delivering a real notification to a real receiver.

Connection to the module

This lesson extends observability/docker-compose.yml (Module 3, lesson 6) with a third service, alertmanager — sibling of prometheus and grafana, already in that same file. It reuses the manifest_invocations_total/manifest_errors_total metric from Module 3, lesson 6 only as a pattern precedent (this lesson's exporter exposes a new metric, manifest_burn_rate, with the numbers lesson 3 of this module already produced). Lesson 5 does, with CloudWatch, what this lesson does with Prometheus.

Step 1 — The exporter: lesson 3's four numbers, as real Prometheus Gauges

# burn_rate_exporter.py
# Exposes the SAME four literal burn-rate numbers that burn_rate_evaluator.py (M4.3) already
# computed, as real Prometheus Gauges, for BOTH scenarios at once (label "scenario"), each
# with its short-window and long-window value (label "window"). One real exporter, one real
# Prometheus scrape, one real Alertmanager evaluation deciding both cases side by side.
# Deterministic: the four values below are copy-pasted from M4.3's literal stdout -- set once
# at startup, never random, never datetime.now().

import time
from prometheus_client import Gauge, start_http_server

manifest_burn_rate = Gauge(
    "manifest_burn_rate",
    "Burn rate of process-shipment-manifest's error budget, per scenario and window",
    ["scenario", "window"],
)

# From scripts/burn_rate_evaluator.py (M4.3), literal output:
manifest_burn_rate.labels(scenario="bad_week", window="short").set(17.54)
manifest_burn_rate.labels(scenario="bad_week", window="long").set(43.41)
manifest_burn_rate.labels(scenario="normal", window="short").set(0.00)
manifest_burn_rate.labels(scenario="normal", window="long").set(0.90)

if __name__ == "__main__":
    start_http_server(8001)
    print("burn_rate_exporter listening on :8001/metrics")
    print("bad_week: short=17.54x long=43.41x | normal: short=0.00x long=0.90x")
    while True:
        time.sleep(3600)

A Gauge (not a Counter, like Module 3 lesson 6's exporter) because burn rate is a value that goes up and down freely, not a cumulative total — the same distinction that lesson's exercise 2 already worked through. The scenario label (bad_week/normal) lets a single exporter, on a single port, expose both of this module's scenarios at once; the window label (short/long) lets the alert rule request both values separately, exactly as lesson 2's two-window pattern demands.

Step 2 — Extending docker-compose.yml, prometheus.yml, and the alert rules

observability/docker-compose.yml (Module 3, lesson 6 left jaeger, prometheus, grafana; this lesson adds alertmanager and mounts a new rules file):

services:
  prometheus:
    image: prom/prometheus:v3.13.2
    container_name: andes-cargo-prometheus
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml:ro
    ports:
      - "9090:9090"

  alertmanager:
    image: prom/alertmanager:v0.33.1
    container_name: andes-cargo-alertmanager
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    ports:
      - "9093:9093"

observability/prometheus.yml, with two new blocks (alerting, rule_files) and a new job_name pointed at this lesson's exporter:

global:
  scrape_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "manifest-burn-rate"
    static_configs:
      - targets: ["host.docker.internal:8001"]

observability/alert_rules.yml, Table 5-8's three severities, as real PromQL rules:

groups:
  - name: manifest-burn-rate
    rules:
      - alert: ManifestErrorBudgetBurnRatePageFast
        expr: |
          manifest_burn_rate{window="short"} >= 14.4
          and ignoring(window)
          manifest_burn_rate{window="long"} >= 14.4
        for: 0m
        labels:
          severity: page
        annotations:
          summary: "process-shipment-manifest burn rate >= 14.4x (1h/5m window), scenario={{ $labels.scenario }}"

      - alert: ManifestErrorBudgetBurnRatePageSlow
        expr: |
          manifest_burn_rate{window="short"} >= 6
          and ignoring(window)
          manifest_burn_rate{window="long"} >= 6
        for: 0m
        labels:
          severity: page
        annotations:
          summary: "process-shipment-manifest burn rate >= 6x (6h/30m window), scenario={{ $labels.scenario }}"

      - alert: ManifestErrorBudgetBurnRateTicket
        expr: |
          manifest_burn_rate{window="short"} >= 1
          and ignoring(window)
          manifest_burn_rate{window="long"} >= 1
        for: 0m
        labels:
          severity: ticket
        annotations:
          summary: "process-shipment-manifest burn rate >= 1x (3d/6h window), scenario={{ $labels.scenario }}"

The piece that makes all of this work is and ignoring(window). manifest_burn_rate{window="short"} >= 14.4 selects, out of the four series the exporter exposes, only the short-window ones that cross the threshold; PromQL's and operator, by default, demands all labels match between the two sides of the comparison — but the left side has window="short" and the right window="long", labels that will never match. ignoring(window) tells PromQL to specifically ignore that label when matching, and pair up by whatever should actually match: scenario. The result is, for each scenario, "does the short window cross the threshold, and does the same scenario's long window too?" — the two-window AND condition, expressed in pure PromQL, with no logic external to Prometheus.

observability/alertmanager.yml, with a real webhook receiver:

route:
  receiver: "andes-cargo-reliability-team"
  group_by: ["alertname", "scenario"]
  group_wait: 5s
  group_interval: 30s
  repeat_interval: 1h

receivers:
  - name: "andes-cargo-reliability-team"
    webhook_configs:
      - url: "http://host.docker.internal:9099/alerts"
        send_resolved: true

And the receiver itself — a real HTTP server, in pure Python, standing in for the Slack/PagerDuty channel a real team would connect here:

# alert_webhook_receiver.py
# A tiny, real HTTP server standing in for the "team channel" -- Alertmanager POSTs its alert
# payload here, in the same JSON shape it would send to Slack/PagerDuty/Opsgenie webhooks.
# $0, no external dependency. Prints each alert group it receives, so the loop
# Prometheus -> Alertmanager -> receiver is verifiable end to end, not just described.

import json
from http.server import BaseHTTPRequestHandler, HTTPServer


class AlertHandler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(length))
        for alert in body.get("alerts", []):
            print(
                f"[alert_webhook_receiver] status={alert['status']:<8} "
                f"alertname={alert['labels'].get('alertname')} "
                f"scenario={alert['labels'].get('scenario')} "
                f"severity={alert['labels'].get('severity')}",
                flush=True,
            )
        self.send_response(200)
        self.end_headers()

    def log_message(self, fmt, *args):
        pass


if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 9099), AlertHandler)
    print("alert_webhook_receiver listening on :9099/alerts")
    server.serve_forever()

Step 3 — Bringing everything up, for real

python3 observability/burn_rate_exporter.py &
python3 observability/alert_webhook_receiver.py &

What to expect (literal):

burn_rate_exporter listening on :8001/metrics
bad_week: short=17.54x long=43.41x | normal: short=0.00x long=0.90x
alert_webhook_receiver listening on :9099/alerts

cd observability
docker compose up -d prometheus alertmanager

What to expect (literal — verified in this environment):

 Container andes-cargo-prometheus  Started
 Container andes-cargo-alertmanager  Started

docker exec andes-cargo-alertmanager alertmanager --version
curl -s http://localhost:9090/-/ready
curl -s http://localhost:9093/-/ready

What to expect (literal):

alertmanager, version 0.33.1 (branch: HEAD, revision: 2c8da51e03f3dbbed24f9711ca2d76aab4eef9c5)
Prometheus Server is Ready.
OK

Step 4 — Confirming the scraping, with real PromQL

curl -s 'http://localhost:9090/api/v1/query?query=manifest_burn_rate'

What to expect (literal — the four series, exactly the exporter's four values):

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {"metric": {"scenario": "bad_week", "window": "short"}, "value": [1786739904.286, "17.54"]},
      {"metric": {"scenario": "bad_week", "window": "long"},  "value": [1786739904.286, "43.41"]},
      {"metric": {"scenario": "normal",   "window": "short"}, "value": [1786739904.286, "0"]},
      {"metric": {"scenario": "normal",   "window": "long"},  "value": [1786739904.286, "0.9"]}
    ]
  }
}

Four series (each value's Unix timestamp is your variable value; the labels and numbers are literal), confirming Prometheus really scraped this lesson's exporter.

curl -s -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=manifest_burn_rate{window="short"} >= 14.4 and ignoring(window) manifest_burn_rate{window="long"} >= 14.4'

What to expect (literal — the most urgent severity's complete expression result):

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {"metric": {"scenario": "bad_week", "window": "short"}, "value": [1786739926.172, "17.54"]}
    ]
  }
}

A single result: scenario="bad_week". The same query, evaluated over normal, would return an empty list (0.90x never crosses 14.4x on any window) — confirmation, in pure PromQL, of the same contrast lesson 3 already showed in Python.

Step 5 — The complete cycle: the rule fires, Alertmanager receives it, the receiver logs it

curl -s http://localhost:9090/api/v1/rules

What to expect (literal, after Prometheus's first rule-evaluation cycle, ~1 minute after bringing up the containers):

ManifestErrorBudgetBurnRatePageFast  state=firing  alerts=[('bad_week', 'firing')]
ManifestErrorBudgetBurnRatePageSlow  state=firing  alerts=[('bad_week', 'firing')]
ManifestErrorBudgetBurnRateTicket    state=firing  alerts=[('bad_week', 'firing')]

All three rules turn firing, and all three have exactly one active scenario: bad_week. No normal result shows up in any of the three — the same conclusion from lesson 3, this time produced by Prometheus's real rules engine, not a Python function.

curl -s http://localhost:9093/api/v2/alerts

What to expect (literal):

ManifestErrorBudgetBurnRatePageFast bad_week page active
ManifestErrorBudgetBurnRateTicket   bad_week ticket active
ManifestErrorBudgetBurnRatePageSlow bad_week page active

Prometheus delivered all three alerts to Alertmanager (confirmed with curl http://localhost:9090/api/v1/alertmanagers, which lists alertmanager:9093 as an active target), and Alertmanager has them registered as active, grouped by alertname+scenario per alertmanager.yml's group_by.

And, closing the loop, the receiver — the stand-in for a real Slack/PagerDuty channel — really received them:

cat observability/webhook.log

What to expect (literal — the complete log, line by line, of what Alertmanager really delivered):

alert_webhook_receiver listening on :9099/alerts
[alert_webhook_receiver] status=firing   alertname=ManifestErrorBudgetBurnRatePageFast scenario=bad_week severity=page
[alert_webhook_receiver] status=firing   alertname=ManifestErrorBudgetBurnRateTicket scenario=bad_week severity=ticket
[alert_webhook_receiver] status=firing   alertname=ManifestErrorBudgetBurnRatePageSlow scenario=bad_week severity=page

Three lines, one per severity, all with scenario=bad_week — none with scenario=normal. This is the complete result, end to end: Prometheus evaluated the rule → decided to fire only for bad_week → Alertmanager received it and grouped it → the receiver logged it, with no step simulated in prose.

   THIS LESSON'S COMPLETE CYCLE, VERIFIED END TO END

   burn_rate_exporter.py (Gauge, 4 fixed values)
          |
          | scrape every 15s
          v
   Prometheus (evaluates alert_rules.yml every ~1min)
          |
          | POST /api/v2/alerts (only scenario=bad_week crosses the threshold)
          v
   Alertmanager (groups by alertname+scenario, routes)
          |
          | POST http://host.docker.internal:9099/alerts
          v
   alert_webhook_receiver.py -- logs 3 lines, all bad_week

Common mistakes

Omitting ignoring(window) in the PromQL expression, and not understanding why the rule never fires (this lesson's central mistake). What happens: someone writes manifest_burn_rate{window="short"} >= 14.4 and manifest_burn_rate{window="long"} >= 14.4, with no ignoring(window), and the rule stays inactive forever, even though the exporter's numbers are correct. How to spot it: if your direct query at /api/v1/query for the complete expression returns an empty list, even though both halves separately do return results. How to fix it: PromQL's and operator demands matching on all labels by default — since the left side has window="short" and the right window="long", there's never a match without explicitly telling PromQL which label to ignore when pairing up. ignoring(window) is, literally, the PromQL implementation of "both windows of the same scenario," not a cosmetic detail.

Bringing up docker compose up before starting the exporter, and expecting the scraping to fail forever (repeated from Module 3, lesson 6, with the same fix). What happens: someone brings up Prometheus/Alertmanager first, and the manifest-burn-rate job's first scrape attempt fails because port 8001 isn't responding yet. How to spot it: curl http://localhost:9090/api/v1/targets shows the target with health: "unknown" or a connection error. How to fix it: it's not a permanent error — with scrape_interval: 15s, Prometheus automatically retries, and the target flips to up on the next cycle, as soon as the exporter responds. This lesson presents the correct order (exporter first) precisely to avoid this unnecessary wait.

Expecting the rule to fire at the exact instant docker compose up finishes (not accounting for Prometheus's evaluation interval). What happens: someone runs curl http://localhost:9090/api/v1/rules immediately after bringing up the containers and sees state: "inactive" on all three rules, and concludes something failed. How to spot it: if your check happens seconds, not minutes, after docker compose up. How to fix it: Prometheus evaluates alert rules on its own interval (1 minute by default, independent of the 15-second scrape_interval) — the first evaluation can happen before the exporter has even been scraped once. Wait at least one full evaluation cycle (~1 minute) before confirming the rules' state, exactly what this lesson did to produce Step 5's output.

Exercises

Exercise 1 — Rewrite the Ticket row's PromQL expression (threshold 1x) using this lesson's same and ignoring(window) pattern, without looking at the file. Mentally verify that, applied to the exporter's four values, it only fires for bad_week.

See solution 
manifest_burn_rate{window="short"} >= 1
and ignoring(window)
manifest_burn_rate{window="long"} >= 1

Applied to the exporter's values: for bad_week, short window 17.54x ≥ 1 (true) and long window 43.41x ≥ 1 (true) → fires. For normal, short window 0.00x ≥ 1 (false) — the and condition already fails on the first term, with no need to evaluate the second — → doesn't fire. Same exact pattern as the other two severities, just with a different threshold and alert name.

Exercise 2 — Explain what would happen to this lesson's three rules if the exporter only exposed the long window's value (window="long"), with no short-window value at all. Could any of the three alerts ever fire?

See solution

None of the three alerts would ever fire. Each expression demands both sides of and ignoring(window) return at least one result meeting its condition — if no series with window="short" exists, the expression's left side (manifest_burn_rate{window="short"} >= X) always returns an empty vector, no matter how high the long window's value is. An empty vector on either side of an and makes the complete result also empty — the rule stays permanently inactive. This illustrates, very concretely, why the short window isn't optional: without it, the whole rule loses the ability to fire, no matter how serious the real situation is.

Exercise 3 — Explain the difference between an alert's state in Prometheus (firing) and its state in Alertmanager (active). Are they synonyms, or do they represent different layers of the same cycle?

See solution

They represent two different, sequential layers of the same cycle. firing in Prometheus (/api/v1/rules) is the state of the alert rule: the PromQL condition was met, evaluated by Prometheus's rules engine. active in Alertmanager (/api/v2/alerts) is the state of the notification: Alertmanager received that alert (via POST /api/v2/alerts, which Prometheus sends automatically when a rule turns firing) and has it registered as pending routing or already routed to a receiver. An alert can be firing in Prometheus without Alertmanager having received it yet (if the connection between them fails, for example) — they're two separate systems, with two different responsibilities: one decides whether something should alert, the other decides who to notify and how to group that alert with related ones.

Summary and next step

This lesson built and ran, end to end, this module's second engine: a real Python exporter exposing burn rate as Prometheus Gauges, a real alert rule with Table 5-8's three severities (using and ignoring(window) to express the two-window condition in pure PromQL), and Alertmanager delivering those alerts to a real webhook receiver. The result, verified at every layer: all three severities fire for scenario="bad_week", none for scenario="normal" — Prometheus confirms it at /api/v1/rules, Alertmanager at /api/v2/alerts, and the receiver logs it in its own log, three layers, the same result.

Before moving on you should be able to: explain what ignoring(window) does and why it's necessary; describe the complete cycle's four layers (exporter → Prometheus → Alertmanager → receiver); and reproduce, by running your own copy of this environment, this lesson's same result.

Lesson 5 builds this module's third engine: the same decision, this time as a real CloudWatch alarm, declared in Terraform over Andes Cargo's real Lambda.

Resources

  1. Prometheus — Alerting rules — the official reference for the syntax used in alert_rules.yml.
  2. Prometheus — Querying: Operators — the official reference for and/ignoring(), this lesson's central operator.
  3. Alertmanager — Configuration — the official reference for route, group_by, and webhook_configs.
  4. This same repository, Module 3, lesson 6 (06-hands-on-prometheus-and-grafana-the-stack-the-market-asks-for.md) — the docker-compose.yml and exporter pattern this lesson extends.
  5. This same repository, Module 4, lesson 3 (03-hands-on-the-burn-rate-evaluator.md) — the Python prototype of the same decision this lesson reimplements in PromQL.
← Previous3. Hands-on: the burn rate evaluatorNext →5. Hands-on: a real CloudWatch alarm on the Lambda