1. Module introduction: from measuring to analyzing and automating

Overview

Up to here you've learned to produce a load test from start to finish. You know how to write the script and launch virtual users (module 2), measure p95 latency, throughput, and error rate (module 3), shape the load with profiles and stages (module 4), turn a metric into a pass/fail verdict with thresholds (module 5), and verify the correctness of the responses under load with check() in realistic scenarios (module 6). Each run you launch leaves something behind: a result. This module is about the two things you haven't yet done with that result: analyze it with judgment and automate the test so it runs on its own.

The first half is analyzing. You'll read the summary and understand what it really tells you —did the p95 stay within the SLO? what was the error rate? what RPS did the system hold?—, meet the trend metric (k6's Trend, which summarizes an entire series of latencies into percentiles), export the result to a file (--out json / csv) to be able to analyze it elsewhere and compare runs, detect a performance regression by comparing a previous run (baseline) against the current one, and know how a bottleneck is investigated —without optimizing here—. The second half is automating: running k6 in continuous integration with a .github/workflows/load.yml where the thresholds act as a gate that blocks the deploy, and deciding when it's worth running the test.

Connection to the module: this lesson is the map. It explains the turn —from measuring to analyzing and automating—, declares the /quote_slow endpoint this module adds to the canonical API to be able to model a regression, fixes the environment rules (what's run and what goes as content), and draws the boundaries. It reuses everything before: the script and the VUs (M2), the metrics you'll analyze (M3), the profiles that generate the load (M4), the thresholds that become the pipeline's gate (M5), and the checks/scenarios (M6). What comes after is the capstone (M8), where you'll do a complete load test. Here we close the cycle of a test: produce → analyze → automate.

The chef who tastes the dish and the one who writes the recipe

Imagine a kitchen. You already know how to cook the dish: you measured the ingredients, controlled the heat, plated it. But cooking the dish once isn't what makes a restaurant good. Two things are missing.

The first is tasting the dish with judgment. It's not enough to see that "it looks good"; the chef tastes a spoonful and judges it against a standard: does it lack salt? is it thinner than yesterday's? And they note it in a notebook, to be able to compare today's dish with last week's. Without that notebook, each dish is an isolated event and no one notices the soup has been coming out thinner for three days. That's analyzing the result: reading the summary, exporting it to a notebook (the results file), and comparing it with the previous one to catch a regression.

The second is writing the recipe and hanging it in the kitchen so the dish comes out the same without the chef present. The recipe states the steps, the quantities, and —most important— the acceptance rule: "if the soup doesn't set, it doesn't go to the table." Any cook follows it, and the bad dish is stopped before reaching the customer. That's automating in CI: the load.yml is the hung recipe, and the threshold is the "doesn't go to the table" rule that stops the deploy on its own. This module teaches you both: tasting the dish with a notebook and hanging the recipe with its rule.

What you do when the run finishes

A very common mistake is treating the load test as an event that ends when the summary appears on the screen. You run k6, look at the p95, say "aha, 200 ms," and close the terminal. That result evaporated: it wasn't saved, it wasn't compared with anything, and the next time someone asks "did performance get worse with the last change?" there'll be nothing to answer with. The run was work thrown in the trash the moment the window closed.

What this module teaches you is that a run is the beginning of something, not the end. When it finishes, there are four moves that give it value:

• Read it with judgment. Not "does it look good?" but "is the p95 within the SLO we promised? did the error rate cross the limit? is the RPS it held enough for the expected traffic?". A number without a standard to compare it against says nothing.
• Export it. Save the result to a file (results.json) to be able to analyze it with other tools, attach it to a report, and —above all— compare it with future runs. The chef's notebook.
• Compare it with the past. Is today's p95 worse than last week's? That's a performance regression, and catching it requires having the previous run saved (the baseline).
• Automate it. Have all of the above run on its own, in a pipeline, with a gate that blocks the deploy if performance degraded —without anyone having to remember to run the test by hand—.

The four moves are this module. Lessons 2 to 5 cover analyzing (reading, exporting, comparing, investigating); lessons 6 and 7, automating (the CI and when to run it).

The endpoint this module declares: /quote_slow

To teach how to detect a regression you need to be able to provoke one: a state in which the system is measurable and correct, but slower than before. The canonical Reservo API is very fast on localhost (POST /quote responds in a few milliseconds), so on its own it doesn't serve to show "this got worse." We need a target that represents a code path that degraded.

As the guide's design establishes, a module that needs extra behavior adds and declares it. This module declares /quote_slow: it does exactly the same as /quote (receives {room, tier, hours}, returns {price_cents} with the same anchor numbers 7500 and 6000) but before responding it waits a fixed delay (time.sleep). It models a very real situation: someone made a change —added a call to another service, removed a database index, introduced an N+1 query— and the endpoint that used to respond in 5 ms now takes 50. The logic is still correct (the price is the same), but the performance fell. The run against /quote is the baseline (the previous performance); the run against /quote_slow is the current one (after the bad change). Comparing their p95s is detecting the regression.

This is the addition to the canonical server. The rest of the server (the one from module 1's lesson 6) doesn't change; only the SLOW_DELAY_S constant, the import time, and the /quote_slow branch in do_POST are added:

# --- ADDED by module 7 to Reservo's canonical server ---
import time

# Seconds of extra work per request in /quote_slow. Models a query that
# got slow after a code change: the REGRESSION the test must catch.
SLOW_DELAY_S = 0.045

# ...inside do_POST, after validating and BEFORE computing the price:
#     if self.path == "/quote_slow":
#         time.sleep(SLOW_DELAY_S)          # the path that degraded
#     cents = price_cents(room, tier, hours)
#
#     if self.path in ("/quote", "/quote_slow"):
#         self._send_json(200, {"price_cents": cents})

An honest note on why we use a fixed sleep here, and not the GIL-serialized CPU work module 5 declared. To teach thresholds (M5) we wanted a latency that depended on the load (cheap with low concurrency, expensive with high), and for that CPU work is ideal. To teach regressions (M7) we want the opposite: a stable and reproducible difference between two versions of the code, so the baseline-vs-current comparison is clean and doesn't depend on the noise of the moment. A fixed delay gives exactly that: /quote responds in milliseconds, /quote_slow in ~45 ms more, run after run. The regression jumps out.

The environment rule (again, because it matters)

This module has two protagonists that go in different categories, and confusing them would ruin the learning:

• What's actually run and cited is Python. The load generator that exports its metrics to a real results.json, the regression check that compares two runs and returns a real exit code (sys.exit), the analyzer that judges a run against an SLO, the CSV that accumulates runs: everything runs for real against localhost and its output is pasted as-is. When you see a block with a python3.14 ... command and an exit code, that happened.
• k6 and CI go as labeled content. k6 isn't installed in this environment (it's a Go binary with its own JavaScript runtime; node doesn't run it). The custom Trend, k6 run --out json, the summary, and —the star of the second half— the .github/workflows/load.yml, are content, faithful to k6's and GitHub Actions' official documentation, never a fabricated output presented as executed. When you see a k6 block or a CI YAML, it will be labeled as content. git and gh are never run here.

This separation is what makes the learning solid. You see the real mechanism —export to a file, compare two runs, fail with an exit code— with your real metrics in Python; and you see the industrial form of that same mechanism in k6 and in the CI YAML. They're the same idea at two scales.

This module's boundaries

• The k6 script and the VUs are module 2's. Here they're taken as known: we analyze and automate a script that already exists.
• The metrics (what the p95, the throughput, the error rate are) are module 3's. Here we don't re-explain them; we read, export, and compare them.
• The load profiles are module 4's. Here we generate the load with the usual generator; the analysis holds whatever the profile.
• The thresholds are module 5's. Here we reuse them as the gate of the CI pipeline. M5 built the verdict; M7 installs it in the whole pipeline and shows the YAML.
• The check()s and the scenarios are module 6's. Here the checks rate is just another metric that appears in the exported summary.
• Optimizing the app or the database (indexes, cache, fixing the slow query) is out of this guide: it's the "after." Here we mention how the bottleneck is located (lesson 5), but we don't fix it. We link to where it continues.
• The capstone —a complete Reservo load test, from smoke to stress with thresholds and checks— is module 8. This module contributes the analysis and the CI that capstone will use.

In one sentence: this module is about what you do with the result and how you automate the test. Reading it, exporting it, comparing runs to catch a regression, and putting it in a pipeline with a gate.

What the destination looks like (an executed preview)

So the map isn't only words, here's the end of the road, actually executed. First, the generator exports the metrics of two runs to JSON —a baseline against /quote (fast) and a current against /quote_slow (the path that degraded)—:

What to expect — two runs of 600 requests; the baseline with a p95 of milliseconds, the degraded one with a much higher p95 due to the fixed delay:

$ python3.14 load_and_export.py http://127.0.0.1:PORT /quote 600 30 results_baseline.json baseline
[baseline] /quote  600 req  concurrency 30
  rps=5449.6  error_rate=0.00%  checks=100.00%
  p50=4.7ms  p95=6.66ms  p99=19.91ms  max=22.63ms
  -> wrote results_baseline.json

$ python3.14 load_and_export.py http://127.0.0.1:PORT /quote_slow 600 30 results_actual.json actual
[actual] /quote_slow  600 req  concurrency 30
  rps=538.1  error_rate=0.00%  checks=100.00%
  p50=54.74ms  p95=61.27ms  p99=73.62ms  max=75.82ms
  -> wrote results_actual.json

And now the regression check compares the p95 of the two runs and fails with a real exit code because it got much worse than the allowed limit (+20%):

What to expect — the p95 went from 6.66 ms to 61.27 ms; the check declares it a REGRESSION and exits with code 1:

$ python3.14 check_regression.py results_baseline.json results_actual.json 20
PERFORMANCE REGRESSION CHECK (p95)
--------------------------------------------------------
baseline (baseline) : p95 = 6.66 ms
actual   (  actual) : p95 = 61.27 ms
change            : +54.61 ms  (+820.0%)
allowed limit     : +20.0%
--------------------------------------------------------
REGRESSION: the p95 rose from 6.66 ms to 61.27 ms (+820.0%, exceeds +20.0%)
RESULT: FAIL  (exit code 1)
$ echo $?
1

That's the whole module in three commands: you export two runs, compare them, and an automatic check catches that the performance got worse and fails —with the real exit code a CI pipeline would use to block the deploy—. (The exact numbers vary a bit in each run, because they depend on how the operating system distributes the time; what doesn't vary is the story: the degraded one is much slower than the baseline, and the check catches it.) The rest of the lessons take this apart piece by piece.

Common mistakes

Treating the run as the end. What happens: the test is run, the p95 is looked at on screen, and the terminal is closed without saving anything. Why it happens: "seeing the number" gets confused with "having the result." How to detect it: if you can't answer "did the p95 get worse relative to last week?", you didn't save the baseline. How to fix it: export each run to a file and save it; an unexported run is lost work (lesson 3).

Confusing an absolute threshold with a regression check. What happens: it's believed that if the p95 is below the SLO (p(95)<200), there's nothing to watch. Why it happens: an absolute threshold only looks at "is it fast?", not "is it slower than before?". How to detect it: a p95 that jumped from 6 ms to 61 ms still passes a 200 ms limit —the threshold doesn't see it, but it's a 10x regression—. How to fix it: use both tools —the absolute threshold (M5) for the SLO, and the regression check (lesson 4) for "did it get worse?"—. They catch different things.

Believing k6 or CI ran here. What happens: someone sees the load.yml or a k6 summary and cites it as "what this guide did." Why it happens: the k6 and GitHub Actions content looks very real. How to detect it: k6 isn't installed and git/gh are never run here; the executed stuff always comes with a python3.14 ... command. How to fix it: remember the rule —Python is run and cited; k6 and the CI YAML are labeled content, faithful to the docs—.

Exercises

Exercise 1 — Analyze vs automate. Classify each task as part of analyzing the result or of automating the test. (a) Reading the summary and checking the p95 is below the SLO. (b) Writing the load.yml that runs k6 every night. (c) Exporting the run to results.json. (d) Putting a threshold as a gate that blocks the deploy. (e) Comparing today's p95 with last week's.

See solution

• (a) Analyze (reading the summary with judgment, lesson 2).
• (b) Automate (the CI, lesson 6).
• (c) Analyze (exporting to be able to analyze and compare, lesson 3).
• (d) Automate (the gate in the pipeline, lesson 6).
• (e) Analyze (comparing runs, detecting a regression, lesson 4).

The two halves of the module: (a), (c), and (e) are analyze; (b) and (d) are automate.

Exercise 2 — Why /quote_slow with a fixed sleep? Module 5 used an endpoint with CPU work (latency that depends on the load) and this module uses one with a fixed sleep. (a) Why is a fixed delay better for teaching regressions? (b) What real situation does /quote_slow model?

See solution

• (a) To detect a regression you have to compare two runs and attribute the difference to the code change, not the noise of the moment. A fixed delay makes /quote_slow always ~45 ms slower than /quote, run after run, so the baseline-vs-current difference is stable and reproducible. Module 5's CPU work varies with the concurrency, which is perfect for seeing a threshold cross under load, but introduces noise into a comparison of two runs.
• (b) It models a code path that degraded after a change: someone added a call to another service, removed an index, or introduced a slow query. The logic is still correct (the price is the same), but the endpoint that responded in 5 ms now takes 50. It's exactly the kind of regression a load test in CI exists to catch before deploying.

Exercise 3 — The regression a threshold doesn't see. In the preview, the p95 went from 6.66 ms to 61.27 ms. An in-house SLO says p(95) < 200 ms. (a) Does the degraded run pass that absolute threshold? (b) Why, then, is it still a problem? (c) What tool catches it?

See solution

• (a) Yes. 61.27 ms < 200 ms, so a threshold p(95)<200 passes with the degraded run. By the absolute SLO, everything is "fine."
• (b) Because the p95 multiplied by more than 9 relative to the baseline (6.66 → 61.27 ms). Even if it's still below the limit today, it's a huge degradation: if the team ignores it, the next regression on top of this one will cross the SLO, and by then it'll be harder to know what change caused it. An early regression is a cheap alert.
• (c) The regression check (lesson 4), which compares the p95 against a previous run's and fails if it got worse by more than a relative limit (+20%, say). It doesn't look at "is it fast?" but "is it slower than before?" —and that's why it catches what the absolute threshold lets through—.

Summary and next step

This module closes the cycle of a load test: after producing it (M2–M6), it's time to analyze and automate it. Analyzing is reading the summary with judgment (p95 within the SLO? error rate? RPS?), exporting the result to a file to be able to study it elsewhere and compare runs, detecting a regression by comparing baseline vs current, and knowing how a bottleneck is investigated —without optimizing here—. Automating is running k6 in CI with a load.yml (content) where the threshold is the gate that blocks the deploy, and deciding when to run the test.

You saw the chef analogy (tasting the dish with a notebook = analyze; hanging the recipe with its rule = automate in CI), the endpoint this module declares —/quote_slow, with a fixed delay to model a reproducible regression—, the environment rule (Python is run; k6 and CI are content), and the executed destination: two runs exported to JSON and a regression check that catches the worsened p95 and fails with an exit code.

Before moving on you should be able to: name the two halves of the module and which lesson covers each; explain why /quote_slow uses a fixed sleep and what it models; and explain why an absolute threshold doesn't see a regression that a relative check does catch. What comes next, in lesson 2, is the first piece of analyzing: reading a run's summary and understanding the trend metric that summarizes a series of latencies into percentiles.

Resources

• k6 — Results output — the official reference for how k6 presents and exports a run's results: the end-of-test summary and the file outputs. The source of this module's k6 content.
• k6 — Running k6 in CI — how k6 integrates into a continuous integration pipeline; the foundation of lesson 6's load.yml.
• Google SRE Book — Service Level Objectives — why a result is judged against an SLO and not in the abstract; the criterion for reading the summary (lesson 2) and choosing limits.
• sys.exit — Python documentation — the mechanism the regression check returns its exit code with (0 = no regression, 1 = regression), the executable mirror of the exit code a CI uses to block the deploy.