4. Detecting a performance regression

Overview

A performance regression is when something that used to be fast becomes slower after a change. It's not a correctness bug —the response is still correct—, it's a speed bug: the same endpoint, with the same logic, now takes longer. And it has a characteristic that makes it slippery: it can't be detected by looking at a single run. A p95 of 61 ms can be perfectly within your SLO and still be a regression, if last week it was 6 ms. To catch it you need two runs —a baseline (the previous performance) and a current one (after the change)— and compare them. In this lesson you build exactly that: a regression check in Python that reads the p95 of two results.json, computes how much it got worse, and fails with an exit code if it exceeded a relative limit. You see it catch the /quote_slow regression (FAIL, exit code 1) and pass two equivalent runs (PASS, exit code 0). And you understand why this relative check catches things an absolute threshold (module 5) lets through.

Connection to the module: this lesson is the heart of analyzing. It consumes the files lesson 3 learned to export (the baseline and the current) and the ones lesson 2 learned to read, and it compares them. It reuses the module-5 exit code (sys.exit), but for a new rule: not "is it fast?" (absolute) but "is it slower than before?" (relative). What comes next (lesson 5) is the natural question after detecting a regression: "where did the time go?" —the bottleneck—. And lesson 6 puts this same check in the CI pipeline.

The speedometer compared to yesterday's, not the road's limit

Imagine you always drive the same route to work. There are two different questions you can ask about your speed. The first: "am I below the road's limit (120 km/h)?". It's an absolute rule: you compare your speed against a fixed number. The second: "am I slower than usual on this stretch?". It's a relative rule: you compare today's speed against your historical speed at the same place.

The two questions catch different problems. You can be going 40 km/h —well below the 120 limit, so the absolute rule says "all fine"— and yet be going half as fast as usual, because there's an accident ahead. The absolute rule doesn't see that problem; the relative one does. The threshold from module 5 is the road's limit: "the p95 must be below 200 ms." The regression check of this lesson is the comparison with your usual time: "today's p95 must not be much worse than yesterday's." You need both, because they catch different things —and a regression from 6 to 61 ms is exactly the one the absolute rule lets through and the relative one catches—.

What a regression is (and why it needs two runs)

A performance regression has three defining traits:

• It's relative, not absolute. It's not about crossing a fixed limit, but about getting worse relative to a reference point (the baseline). A p95 of 61 ms isn't "bad" in the abstract; it's a regression because it was 6 ms before.
• The correctness holds. The endpoint still returns the correct response (price_cents: 7500, 0% error). If the response were incorrect, it would be a functional bug, which a correctness test catches (E2E, unit). The regression is purely about speed.
• It comes from a change. Something between the baseline and the current run caused it: a commit that added a network call, removed an index, introduced an N+1 query. Detecting it early —in CI, before deploying— is what keeps it from reaching users.

Because of the first trait, a regression can't be seen in an isolated run. You need the baseline. Hence lesson 3 (exporting) is the prerequisite: without having saved the previous result, there's nothing to compare against. The regression lives in the difference between two runs, not in the numbers of a single one.

The regression check (executable)

Here's the check that does run. It reads two results.json —the baseline and the current one—, extracts each one's p95, computes the percentage change, and fails with sys.exit(1) if the p95 got worse by more than a relative limit you pass it (for example, +20%):

"""Performance regression check: compares two exported runs.

Reads the p95 of a BASELINE run and a CURRENT run (two results.json) and
fails with exit code 1 if the p95 got worse by more than a relative limit. It's a
performance regression test: it doesn't ask "is it fast?" but "is it SLOWER than before?".

Usage: python3.14 check_regression.py baseline.json actual.json MAX_INCREASE_PCT
"""
import json
import sys

baseline_path = sys.argv[1]
actual_path = sys.argv[2]
max_increase_pct = float(sys.argv[3]) if len(sys.argv) > 3 else 10.0

with open(baseline_path) as f:
    baseline = json.load(f)
with open(actual_path) as f:
    actual = json.load(f)

p95_base = baseline["latency_ms"]["p95"]
p95_now = actual["latency_ms"]["p95"]
delta = p95_now - p95_base
increase_pct = (delta / p95_base) * 100 if p95_base else 0.0

print("PERFORMANCE REGRESSION CHECK (p95)")
print("-" * 56)
print(f"baseline ({baseline['label']:>8}) : p95 = {p95_base:.2f} ms")
print(f"actual   ({actual['label']:>8}) : p95 = {p95_now:.2f} ms")
print(f"change            : {delta:+.2f} ms  ({increase_pct:+.1f}%)")
print(f"allowed limit     : +{max_increase_pct:.1f}%")
print("-" * 56)

if increase_pct > max_increase_pct:
    print(f"REGRESSION: the p95 rose from {p95_base:.2f} ms to {p95_now:.2f} ms "
          f"(+{increase_pct:.1f}%, exceeds +{max_increase_pct:.1f}%)")
    print("RESULT: FAIL  (exit code 1)")
    sys.exit(1)
else:
    print(f"OK: the p95 didn't get worse than the limit (+{increase_pct:.1f}% "
          f"<= +{max_increase_pct:.1f}%)")
    print("RESULT: PASS  (exit code 0)")
    sys.exit(0)

Three decisions that define the check:

• It compares p95, not the average. The p95 is the one that reflects the tail's experience (module 3); a regression usually hits the tail first. You could also compare p99 or RPS; the p95 is the most common indicator of a latency regression.
• The limit is relative (%), not absolute (ms). max_increase_pct is "how much slower I tolerate relative to the baseline." A relative limit adapts: if the baseline legitimately rises with the product's growth, the check keeps measuring the relative degradation, not a fixed number that becomes obsolete. A margin is left (not 0%) because there's natural noise between runs —the p95 varies by a few milliseconds even if the code doesn't change—.
• It fails with sys.exit(1). The same mechanism from module 5: a nonzero exit code is how a program tells a pipeline "I failed." Lesson 6 connects it to CI.

The FAIL case: the /quote_slow regression

We compare the baseline (fast endpoint, exported in lesson 3) against the current one (slow endpoint), with a +20% limit. Real output:

What to expect — the p95 went from 6.66 to 61.27 ms, a +820%: well above the allowed +20%, so REGRESSION and exit code 1:

$ python3.14 check_regression.py results_baseline.json results_actual.json 20
PERFORMANCE REGRESSION CHECK (p95)
--------------------------------------------------------
baseline (baseline) : p95 = 6.66 ms
actual   (  actual) : p95 = 61.27 ms
change            : +54.61 ms  (+820.0%)
allowed limit     : +20.0%
--------------------------------------------------------
REGRESSION: the p95 rose from 6.66 ms to 61.27 ms (+820.0%, exceeds +20.0%)
RESULT: FAIL  (exit code 1)
$ echo $?
1

The echo $? prints 1: the shell received the check's exit code. That 1 is what a CI pipeline uses to put the build red and block the deploy —you'll see it set up in lesson 6—.

The PASS case: two equivalent runs

Now we compare the baseline against another run of the same fast endpoint (/quote), to see that the check doesn't fire false alarms when performance didn't change. Real output:

What to expect — two runs of the fast endpoint give almost identical p95s (6.66 vs 6.71 ms), a +0.8% within the margin: OK and exit code 0:

$ python3.14 check_regression.py results_baseline.json results_baseline2.json 20
PERFORMANCE REGRESSION CHECK (p95)
--------------------------------------------------------
baseline (baseline) : p95 = 6.66 ms
actual   (baseline2) : p95 = 6.71 ms
change            : +0.05 ms  (+0.8%)
allowed limit     : +20.0%
--------------------------------------------------------
OK: the p95 didn't get worse than the limit (+0.8% <= +20.0%)
RESULT: PASS  (exit code 0)
$ echo $?
0

That +0.8% is the natural noise between two identical runs: the p95 never comes out exactly the same twice, because the operating system distributes the CPU slightly differently each time. The +20% margin absorbs that noise and only fires on a real degradation. That balance —strict enough to catch regressions, loose enough not to fire on noise— is the art part of a regression check.

Relative regression vs absolute threshold: why you need both

This is the point that makes the lesson valuable. In lesson 2, the analyzer judged the degraded run (p95 = 61.27 ms) against an absolute SLO of p95 < 200 ms and said PASS —61 is below 200—. In this lesson, the regression check judged the same run against the baseline and said FAIL —61 is 9x worse than 6.66—. The same run, two opposite verdicts, because they answer different questions:

	Absolute threshold (M5)	Regression check (M7)
Question	Is it below the SLO?	Is it slower than before?
Compares against	A fixed number (200 ms)	The baseline (the previous run)
Catches	Crossing the SLO limit	Degrading, even if still below the limit
Lets through	A regression that doesn't yet cross the SLO	A bad p95 that was always bad
Analogy	The road's limit	Your usual speed

Neither replaces the other. The absolute threshold protects the commitment to the user (the SLO); if you remove it, a p95 that was always slow would go unnoticed. The regression check protects against gradual degradation; if you remove it, performance erodes commit by commit and you only find out when it finally crosses the SLO —by which point there are many candidate changes and it's hard to find the culprit—. A mature pipeline runs both: the SLO as a floor, the regression as an early alarm.

Common mistakes

Trying to detect a regression with a single run. What happens: a p95 of 61 ms is looked at and someone asks "is this a regression?" without anything to compare against. Why it happens: it's forgotten that the regression is relative. How to detect it: if you don't have the previous p95 (the baseline), you can't answer. How to fix it: save (export) each run, and always compare against a baseline —lesson 3 is the prerequisite for this one—.

Setting the regression limit at 0%. What happens: the p95 is required not to rise at all, and the check fails constantly from the natural noise between runs. Why it happens: it's not considered that the p95 varies by a few milliseconds even if the code doesn't change. How to detect it: frequent FAILs with increases of +1% or +2% that correspond to no real change. How to fix it: leave a margin (10–25% is typical) that absorbs the noise and only fires on a real degradation. A check that always screams ends up ignored.

Comparing against a bad baseline. What happens: the "baseline" was taken from a run on a saturated machine, or with a different load profile, and the comparison makes no sense. Why it happens: baseline and current were measured under different conditions. How to detect it: erratic regression results that don't correspond to the code changes. How to fix it: measure baseline and current under the same conditions —same machine/environment, same number of requests and concurrency, same endpoint— so the only variable is the code change. Compare apples to apples.

Exercises

Exercise 1 — Regression or not? Relative limit: +20%. For each pair (baseline → current) say whether the check gives PASS or FAIL. (a) 100 ms → 105 ms. (b) 100 ms → 130 ms. (c) 8 ms → 60 ms. (d) 200 ms → 180 ms.

See solution

• (a) PASS. +5% ≤ +20%. Within the margin (normal noise).
• (b) FAIL. +30% > +20%. The p95 got worse than tolerated.
• (c) FAIL. +650% > +20%. It multiplied by 7.5; huge regression (even if 60 ms may be below the absolute SLO).
• (d) PASS. The p95 dropped (−10%): it improved, it didn't get worse. The check only fails on increases.

Exercise 2 — The case that triggers the lesson. The degraded run has p95 = 61.27 ms. The absolute SLO is p95 < 200 ms and the regression limit is +20% over a baseline of 6.66 ms. (a) What does the absolute threshold say? (b) What does the regression check say? (c) Why don't they contradict each other?

See solution

• (a) The absolute threshold says PASS: 61.27 ms < 200 ms, it meets the SLO.
• (b) The regression check says FAIL: 61.27 is +820% relative to 6.66, well above the allowed +20%.
• (c) Because they answer different questions. The absolute one asks "does it meet the commitment to the user (200 ms)?" —and yes, it does—. The relative one asks "did it get worse relative to before?" —and yes, it got much worse—. They don't contradict each other: they describe two true facts at once. The run meets today's SLO and is a regression relative to yesterday. That's why a mature pipeline runs both checks.

Exercise 3 — Extend the check. The current check only compares the p95. Describe (in prose, without writing all the code) how you'd extend it so it also fails if (a) the p99 got worse by more than the limit, and (b) the RPS dropped by more than a certain percentage. Why is the RPS compared the opposite way from the latency?

See solution

• (a) I'd read p99 from the two JSONs (baseline["latency_ms"]["p99"] and the current one's), compute its percentage increase the same as with the p95, and add a condition to the if: if either of the two (p95 or p99) exceeds the limit, it's a regression. Watching the p99 in addition to the p95 catches regressions that hit only the extreme tail.
• (b) For the RPS, the problem is a drop, not a rise: if the throughput falls a lot, the system got worse. I'd compute the RPS's percentage change and fail if it dropped by more than the limit (rps_now < rps_base * (1 - max_drop)). It's compared the opposite way from the latency because in latency "higher = worse," while in throughput "lower = worse": they're metrics of opposite sense. A complete regression check watches both directions.

Summary and next step

In this lesson you built a performance regression check: a program that compares the p95 of two runs —a baseline and the current— and fails with an exit code if the p95 got worse by more than a relative limit. You understood that a regression is relative (getting worse relative to a reference point), keeps correctness (it's a speed bug, not a result one), and comes from a change, and that's why it's not detected in an isolated run: it lives in the difference between two. You saw it catch the /quote_slow regression (FAIL, exit 1) and pass two equivalent runs (PASS, exit 0), with the natural noise absorbed by the margin.

The central point: the regression check (relative, "slower than before?") and the absolute threshold (M5, "below the SLO?") catch different things and both are needed. The same degraded run passes the 200 ms SLO and at the same time is a 9x regression —two true facts that are only seen with both tools—.

Before moving on you should be able to: define a regression with its three traits; explain why it needs a baseline; and contrast the relative check with the absolute threshold. What comes next, in lesson 5, is the question that arises right after detecting a regression: where did the time go? —how you investigate whether the bottleneck is in the app, the database, or the network— without optimizing here, because that's the "after."

Resources

• k6 — Thresholds — the contrast with this lesson: the threshold is the absolute check (SLO); the regression is the relative one (vs baseline). Both use an exit code to gate.
• Google SRE Book — Service Level Objectives — why the absolute commitment (the SLO) and the trend (the relative degradation) are watched at once, and how the margin is chosen.
• sys.exit — Python documentation — the mechanism the check returns 0 (no regression) or 1 (regression) with, the same exit-code language a CI uses to block the deploy.
• json — Python documentation — json.load, with which the check reads the p95 of the two results.json exported in lesson 3.