Module 7: Analyzing Results And Ci

8. Mini-project: export and detect a regression

Next →

Overview

This mini-project brings the whole analyzing half of the module together with your own hands, from start to finish, and connects it with the automating one. You'll bring up the Reservo API with the declared /quote_slow endpoint, run two loads —a baseline against the fast endpoint /quote and a degraded one against the slow /quote_slow—, export each to a results.json, and run a regression check that compares the two p95s and fails with an exit code because the performance got worse than the limit. As a close, you write the CI .github/workflows/load.yml (content) that would automate all this. By the end you'll have the complete journey executed: produce the result, save it, compare it, catch the regression with an exit code, and know how it would be put in a pipeline.

Connection to the module: this is module 7's capstone. It uses the server from lesson 1 (with /quote_slow), the exporter from lesson 3, the regression check from lesson 4, and the load.yml from lesson 6. It's the synthesis: if you do this journey on your own, you master the module's analysis half. What comes next is module 8, the capstone of the whole guide, where this is integrated into a complete Reservo load test (from smoke to stress, with thresholds and checks).

The assignment

You're responsible for the Reservo API's performance. A colleague opened a change that touches the quote endpoint, and you want to make sure it didn't degrade the performance before it merges. Your task:

  1. Bring up Reservo with /quote_slow (which models the endpoint after the bad change).
  2. Measure the baseline (the previous performance, hitting /quote) and export it.
  3. Measure the current run (the after performance, hitting /quote_slow) and export it.
  4. Run a regression check that compares the two p95s and fails if it got worse by more than +20%.
  5. Write the load.yml that would automate this check in CI.

Deliverables: the two results.json, the check's output with its exit code, and the YAML.

Step 1 — Bring up the target with /quote_slow

Use the canonical Reservo server with this module's addition (the SLOW_DELAY_S constant, the import time, and the /quote_slow branch in do_POST; the complete code is in lesson 1). Start it in the background and read the port it chose (remember: port 0, the OS assigns a free one).

What to expect — the server prints its port and responds correctly on both /quote (fast) and /quote_slow (slow), both with the anchor number 7500:

$ python3.14 reservo_server.py &
Reservo listening on http://127.0.0.1:PORT

$ curl -s -X POST http://127.0.0.1:PORT/quote \
    -H 'Content-Type: application/json' -d '{"room":"Focus","tier":"basic","hours":3}'
{"price_cents": 7500}

$ curl -s -X POST http://127.0.0.1:PORT/quote_slow \
    -H 'Content-Type: application/json' -d '{"room":"Focus","tier":"basic","hours":3}'
{"price_cents": 7500}

Both endpoints return the same (7500): the logic is identical. The difference is only in the time —/quote_slow waits a fixed delay before responding—, which is exactly what makes it a pure performance regression (speed, not correctness).

Steps 2 and 3 — Measure and export the two loads

With lesson 3's exporter (load_and_export.py), launch the two runs with the same parameters (600 requests, concurrency 30) so the only variable is the endpoint. The first is the baseline (/quote), the second the current (/quote_slow). Real output:

What to expect — the baseline with a p95 of milliseconds; the degraded one with a much higher p95 due to the delay; each writes its JSON:

$ python3.14 load_and_export.py http://127.0.0.1:PORT /quote 600 30 results_baseline.json baseline
[baseline] /quote  600 req  concurrency 30
  rps=5449.6  error_rate=0.00%  checks=100.00%
  p50=4.7ms  p95=6.66ms  p99=19.91ms  max=22.63ms
  -> wrote results_baseline.json

$ python3.14 load_and_export.py http://127.0.0.1:PORT /quote_slow 600 30 results_actual.json actual
[actual] /quote_slow  600 req  concurrency 30
  rps=538.1  error_rate=0.00%  checks=100.00%
  p50=54.74ms  p95=61.27ms  p99=73.62ms  max=75.82ms
  -> wrote results_actual.json

And the artifacts on disk, ready to compare. Real output:

What to expect — two summary JSONs; notice the contrast in p95 (6.66 vs 61.27 ms) and in rps (5449.6 vs 538.1):

$ cat results_baseline.json results_actual.json
{
  "label": "baseline",
  "endpoint": "/quote",
  "requests": 600,
  "concurrency": 30,
  "duration_s": 0.11,
  "rps": 5449.6,
  "error_rate": 0.0,
  "checks_rate": 1.0,
  "latency_ms": {
    "min": 2.7,
    "p50": 4.7,
    "p95": 6.66,
    "p99": 19.91,
    "max": 22.63,
    "avg": 5.29
  }
}{
  "label": "actual",
  "endpoint": "/quote_slow",
  "requests": 600,
  "concurrency": 30,
  "duration_s": 1.115,
  "rps": 538.1,
  "error_rate": 0.0,
  "checks_rate": 1.0,
  "latency_ms": {
    "min": 46.29,
    "p50": 54.74,
    "p95": 61.27,
    "p99": 73.62,
    "max": 75.82,
    "avg": 54.73
  }
}

Both with error_rate: 0.0 and checks_rate: 1.0: the correctness held. It's a pure performance regression.

Step 4 — Detect the regression (with an exit code)

Now lesson 4's regression check (check_regression.py), comparing the baseline with the current one, with a +20% limit. Real output:

What to expect — the p95 rose from 6.66 to 61.27 ms (+820%), well above +20%: REGRESSION and exit code 1:

$ python3.14 check_regression.py results_baseline.json results_actual.json 20
PERFORMANCE REGRESSION CHECK (p95)
--------------------------------------------------------
baseline (baseline) : p95 = 6.66 ms
actual   (  actual) : p95 = 61.27 ms
change            : +54.61 ms  (+820.0%)
allowed limit     : +20.0%
--------------------------------------------------------
REGRESSION: the p95 rose from 6.66 ms to 61.27 ms (+820.0%, exceeds +20.0%)
RESULT: FAIL  (exit code 1)
$ echo $?
1

The check caught the regression and exited with code 1. Also verify that it does not fire a false alarm: comparing the baseline against another run of the same fast endpoint must give PASS (exit 0). Real output:

What to expect — two equivalent runs give almost identical p95s (+0.8%, within the margin): PASS and exit code 0:

$ python3.14 check_regression.py results_baseline.json results_baseline2.json 20
PERFORMANCE REGRESSION CHECK (p95)
--------------------------------------------------------
baseline (baseline) : p95 = 6.66 ms
actual   (baseline2) : p95 = 6.71 ms
change            : +0.05 ms  (+0.8%)
allowed limit     : +20.0%
--------------------------------------------------------
OK: the p95 didn't get worse than the limit (+0.8% <= +20.0%)
RESULT: PASS  (exit code 0)
$ echo $?
0

And the exit code deciding the deploy, as CI would:

What to expect — with the regression, the || fires and the deploy is blocked:

$ python3.14 check_regression.py results_baseline.json results_actual.json 20 > /dev/null \
    && echo "DEPLOY: authorized" \
    || echo "DEPLOY: BLOCKED (exit code $?)"
DEPLOY: BLOCKED (exit code 1)

That's the project's central result: you detected a performance regression by comparing two runs, and the verdict came out as a real exit code a pipeline would use to block the deploy.

Step 5 — The CI load.yml (content)

As a close, write the workflow that would automate this check. It's labeled content —git/gh are not run here—, and it combines the module's two tools: k6 run with thresholds (the absolute gate vs SLO) and the regression check (the relative gate vs baseline):

# CONTENT (not run here): .github/workflows/load.yml
# Reservo load test in CI: absolute gate (k6) + regression gate.
# See grafana.com/docs/k6 and docs.github.com/actions.
name: load-test

on:
  schedule:
    - cron: "0 3 * * *"      # nightly (lesson 7: not on every PR)
  release:
    types: [published]        # pre-release: launch gate
  workflow_dispatch:          # on-demand

jobs:
  load:
    runs-on: ubuntu-latest
    steps:
      - name: Check out the code
        uses: actions/checkout@v4

      - name: Start the Reservo API (the target)
        run: |
          python3 reservo_server.py &
          sleep 2

      - name: Install k6
        uses: grafana/setup-k6-action@v1

      - name: Run the load test (thresholds = absolute gate vs SLO)
        run: k6 run --out json=results.json load_test.js
        # A broken threshold -> k6 exits with 99 -> the step fails -> deploy blocked.

      - name: Download the last run's baseline
        uses: actions/download-artifact@v4
        with:
          name: k6-results-baseline
          path: baseline/

      - name: Regression check (relative gate vs baseline)
        run: |
          python3 check_regression.py baseline/results.json results.json 20
        # p95 got worse > 20% relative to the baseline -> exit 1 -> deploy blocked.

      - name: Save this run's JSON as an artifact
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: k6-results
          path: results.json

Notice the two gates chained: first k6 run (does it meet the absolute SLO?), then the regression check (did it get worse vs the baseline?). Either of the two failing puts the job red and blocks the deploy —lesson 4's two safety nets, in a single pipeline—. And the on: with lesson 7's correct cadence (nightly + pre-release + on-demand, never pull_request).

Rubric

Evaluate your deliverable against these criteria:

CriterionMeets if…
The target runsReservo comes up with /quote_slow declared; both endpoints return 7500 (same logic).
Two loads measuredBaseline (/quote) and current (/quote_slow) with the same parameters (same N, same concurrency); the only variable is the endpoint.
Real exportEach run wrote a results.json with p50/p95/p99, rps, error_rate, checks_rate. The files exist on disk.
Regression detectedThe check compared the p95s, reported "REGRESSION: the p95 rose from X to Y" and exited with exit code 1; echo $? showed 1.
No false alarmComparing the baseline against another equivalent run gave PASS (exit 0): the margin absorbs the noise.
Correctness heldBoth runs have error_rate: 0.0 and checks_rate: 1.0: it's a speed regression, not a result one.
CI as contentThe load.yml is labeled as content, with the two gates (absolute k6 + relative regression) and the correct cadence (schedule/release/workflow_dispatch, no pull_request).
Environment honestyWhat's executed (Python) is cited with its command and its exit code; k6 and the YAML go labeled as content; git/gh were never run.

Common mistakes

Measuring baseline and current with different parameters. What happens: the baseline is run with concurrency 30 and the current one with 50, and the p95 difference mixes the endpoint's effect with the load's. Why it happens: two variables are changed at once. How to detect it: if baseline and current don't share N and concurrency, the comparison isn't clean. How to fix it: same parameters in both runs; the only difference must be the endpoint (the "code change" you're testing).

Forgetting the PASS case. What happens: only the fact that the check fails with the regression is tested, without verifying it does not fire when there's no change. Why it happens: it's assumed a check that catches the bad is complete. How to detect it: if you never tested two equivalent runs, you don't know whether your limit is so strict it gives false reds. How to fix it: verify both cases —FAIL with the regression, PASS with equivalent runs—; a gate that always fails is as useless as one that never fails.

Presenting the load.yml as if it had run. What happens: the YAML is shown without a label, as if the pipeline had been executed. Why it happens: the environment honesty is lost. How to detect it: git/gh are not run here and k6 isn't installed; the YAML is content. How to fix it: label the YAML as content; what's executed is the Python, with its command and its echo $?.

Exercises

Exercise 1 — Run the complete journey. Bring up Reservo with /quote_slow, run the two loads (same parameters), export them, and run the regression check. (a) What exit code did the check give? (b) How much did the p95 get worse, in %? (c) Did both runs keep the correctness (error 0, checks 100%)?

See solution
  • (a) Exit code 1: the check detected the regression (the p95 got worse well above +20%). echo $? prints 1.
  • (b) In the reference run, from 6.66 to 61.27 ms: +820% (it multiplied by ~9.2). Your exact numbers will vary, but the order of magnitude is the same: the fixed delay of /quote_slow (~45 ms) dominates over the baseline's ~5 ms.
  • (c) Yes: both have error_rate: 0.0 and checks_rate: 1.0. The logic is identical (same 7500); only the time changed. Pure performance regression.

Exercise 2 — Adjust the limit. You ran the check with +20% and it failed. (a) With what limit would the baseline-vs-current comparison stop failing? (b) Would it be a good idea to raise the limit until it passes? (c) What limit makes sense for the PASS case (baseline vs baseline2, +0.8%)?

See solution
  • (a) Since the p95 rose +820%, you'd have to set a limit greater than 820% for the baseline-vs-current comparison to pass. A limit that absurd lets any regression through.
  • (b) No. Raising the limit until the gate passes is "fixing" the test instead of the app: it silences the alarm instead of addressing the regression. The limit is chosen by what the business tolerates (a typical 10–25%), not to make the build green.
  • (c) Any limit above +0.8% (the natural noise) lets the PASS case through. A +20% gives plenty of margin for the noise and still catches a real degradation (which is usually tens or hundreds of percent). That balance —loose against the noise, strict against the regression— is the right one.

Exercise 3 — Extend the pipeline. The load.yml has two gates (absolute k6 and relative regression). (a) What happens if k6 run passes the SLO but the regression check fails? (b) Why does the upload-artifact have if: always()? (c) Why does the on: not include pull_request?

See solution
  • (a) The job fails anyway and the deploy is blocked. The two gates are chained: any that exits with code ≠ 0 puts the job red. That the absolute SLO is met doesn't rescue the build if the performance degraded relative to the baseline. (It's exactly the degraded run's case: 61 ms passes the 200 ms SLO but is a 9x regression.)
  • (b) To upload the results.json even if a gate failed —which is when you most need it to investigate—. By default, a step doesn't run if the previous one failed; if: always() forces it to run always.
  • (c) Because the load test is slow, expensive, noisy, and needs a stable environment (lesson 7): it doesn't go on every PR. Its cadence is nightly + pre-release + on-demand.

Summary and next step

In this mini-project you did, with your own hands and actually executed, the module's whole analysis cycle: you brought up Reservo with /quote_slow, measured two loads with the same parameters (baseline against /quote, current against /quote_slow), exported each to a results.json, and ran a regression check that caught the p95 getting worse from 6.66 to 61.27 ms and failed with exit code 1 —while two equivalent runs gave PASS—. You confirmed it was a pure performance regression (correctness intact: error 0, checks 100%), and wrote the CI load.yml (content) with the two chained gates —k6's absolute and the regression's relative— and the correct cadence.

With this you close all of module 7: producing a test (M2–M6), analyzing it (read, export, detect the regression, locate the bottleneck), and automating it (the pipeline and its cadence). What comes next is module 8, the capstone of the whole guide: a complete Reservo load test —from smoke to load to stress, with stages, thresholds, and checks of the quote→book flow— that integrates everything you learned, including this module's analysis and CI.

Resources

← Previous7. When to run the load testNext →1. Module introduction: assembling the complete test