Module 7: Analyzing Results And Ci

5. Finding the bottleneck (app, DB, network)

Overview

You detected a regression: the p95 got worse. The immediate question is where did the time go?. An endpoint that takes 60 ms instead of 6 spent those 54 extra ms somewhere —processing in the app, waiting for the database, traveling over the network— and knowing which one tells you what to look at next. In this lesson you learn to locate the bottleneck: the signals that separate an app problem from a database or a network one, and how the load tool itself helps distinguish them (the timing breakdown of a request, a Trend per step). It's important to be clear about the scope: here we mention how it's investigated, not how it's optimized. Fixing the bottleneck —adding an index, adding a cache, rewriting the query— is the "after," outside this guide. What you do here, executed, is the first attribution: comparing the fast run with the slow one to see that the extra time is in the server and not the network.

Connection to the module: this lesson follows naturally from lesson 4. Detecting the regression (lesson 4) answers "did it get worse?"; locating the bottleneck answers "where?". It reuses the trend metric (lesson 2) —a Trend per step is the tool for isolating the slow step— and the exported runs (lesson 3). It's the guide's boundary: here the load engineer's work ends (measure, detect, point out where) and the optimizer's begins (fix it), which belongs to other guides.

The plumber who locates the leak before breaking the wall

When there's low water pressure in the shower, the plumber doesn't start breaking walls at random. First they locate the problem. They open the kitchen tap: if little water comes out there too, the leak is in the main pipe (shared); if the kitchen is fine, the problem is only the shower's branch. They close the stopcock and measure the pressure at different points. With a few tests they narrow down where the leak is —at the inlet, in a specific stretch, in the showerhead— without having broken a single wall yet. Only when they know where do they decide how to fix it.

Locating a performance bottleneck is that plumbing work. You don't optimize blindly ("let's cache everything, see if it improves"); first you narrow down where the time goes. In the app (the code that processes the request)? In the database (a slow query)? In the network (the bytes' journey)? Each one leaves a different signal, and the load test —with its timing breakdown— is your pressure gauge. This lesson teaches you to read the pressure gauge and point at the stretch. Breaking the wall (optimizing) is another trade.

The three suspects and their signals

When an endpoint is slow, the time went into one (or several) of three places. Each has a signature that helps distinguish it:

The app (CPU / code). The server spends time computing: a heavy loop, a costly serialization, work that grows with the load. Signal: the latency rises with the concurrency (more requests competing for CPU), and the time goes into the server's processing, not into connecting or transferring. Module 5's /quote_cpu (CPU work serialized by the GIL) is this case.
The database. The server is waiting for the DB: a query without an index, a lock, an N+1 query (one query per row in a loop). Signal: the time goes into the server's processing (like the app), but it doesn't scale with the CPU —the server is idle, waiting—; and the latency jumps when the data or the DB contention grows. This module's /quote_slow (a time.sleep that models an external wait) behaves like this: the server doesn't compute, it waits.
The network. The bytes take time to travel: high round-trip latency (distant clients), large payloads, bandwidth saturation. Signal: the time goes into connecting and transferring, not into the server's processing; it gets worse with geographic distance and response size, not with the server's load. On localhost this signal is almost nil —that's why our environment isolates the other two well—.

The key to distinguishing them is where within the request the time went. And there's where the breakdown the load tool gives comes in.

The timing breakdown of a request (k6, content)

k6 doesn't only measure the total latency (http_req_duration): it breaks it down into the phases of an HTTP request, and that breakdown is your pressure gauge for separating network from server. The metrics (labeled content, from k6's documentation):

http_req_blocked: time waiting for a free socket (before connecting).
http_req_connecting: time establishing the TCP connection. High = network/connection problem.
http_req_tls_handshaking: TLS negotiation (HTTPS).
http_req_sending: time sending the request. High = large payload / slow network.
http_req_waiting: time from finishing sending until the first byte of the response —the famous TTFB (time to first byte)—. It's the server's processing time (the app + the DB). High = app or database problem.
http_req_receiving: time downloading the response. High = large response / slow network.

The reading rule is simple and powerful: if the time concentrates in http_req_waiting (TTFB), the problem is in the server (app or DB); if it concentrates in connecting/sending/receiving, it's in the network. A k6 summary that separates it like this (content):

// CONTENT (not run here): k6's timing breakdown. See grafana.com/docs/k6
     http_req_duration..............: avg=61ms  p(95)=75ms
       { expected_response:true }...: avg=61ms  p(95)=75ms
     http_req_waiting...............: avg=60ms  p(95)=74ms   <- almost ALL the time: the SERVER
     http_req_connecting............: avg=0.3ms p(95)=0.5ms  <- network: negligible
     http_req_sending...............: avg=0.1ms p(95)=0.2ms
     http_req_receiving.............: avg=0.2ms p(95)=0.4ms

That portrait —60 of 61 ms in http_req_waiting— screams "the server is taking long to respond": the problem is app or DB, not the network. To go from "the server" to "app vs DB" you need to look inside the server (does it rise with the CPU? → app; is the server idle waiting? → DB), and there you use server-side tools: logs, a profiler, the database's metrics. The load test takes you to the server's door; crossing it is the optimization work.

The executable side: attributing the extra time

With our two exported runs we can make the first real attribution. Both hit the same localhost (same network, negligible), so any latency difference between /quote and /quote_slow is in the server, not the network. This small script makes it explicit:

"""Locates WHERE the latency grew by comparing two runs.

It optimizes nothing: it only attributes the p50/p95 increase. Since both runs
hit the SAME localhost (same network), an increase in the response time
points to the server's processing (the app or its database), not the network.

Usage: python3.14 where_time_went.py baseline.json actual.json
"""
import json
import sys

with open(sys.argv[1]) as f:
    base = json.load(f)
with open(sys.argv[2]) as f:
    now = json.load(f)

d_p50 = now["latency_ms"]["p50"] - base["latency_ms"]["p50"]
d_p95 = now["latency_ms"]["p95"] - base["latency_ms"]["p95"]
print(f"baseline {base['endpoint']:>12}: p50={base['latency_ms']['p50']}ms  "
      f"p95={base['latency_ms']['p95']}ms")
print(f"actual   {now['endpoint']:>12}: p50={now['latency_ms']['p50']}ms  "
      f"p95={now['latency_ms']['p95']}ms")
print(f"delta               : p50 {d_p50:+.2f}ms   p95 {d_p95:+.2f}ms")
print(f"same network (localhost) in both -> the extra time is in the SERVER")
print(f"next step (the 'after', outside this guide): profile that endpoint")

Real output:

What to expect — the slow endpoint added ~50 ms to the p50 and ~55 ms to the p95; since the network is the same, that time is in the server:

$ python3.14 where_time_went.py results_baseline.json results_actual.json
baseline       /quote: p50=4.7ms  p95=6.66ms
actual    /quote_slow: p50=54.74ms  p95=61.27ms
delta               : p50 +50.04ms   p95 +54.61ms
same network (localhost) in both -> the extra time is in the SERVER
next step (the 'after', outside this guide): profile that endpoint

The attribution is done: the ~50 extra ms are in the server, and since /quote_slow doesn't compute (it just does time.sleep), the signature is that of an external wait —the model of a slow database query—. A high http_req_waiting with the server idle points to the DB; if instead the latency had risen with the concurrency from CPU, it would point to the app. We get this far: "the time is in the server, with a wait signature." The next step —opening the endpoint, finding the slow query, and adding an index— is optimization, and it's the "after."

Where this guide ends and the "after" begins

It's worth being explicit about the boundary, because it's an easy temptation to cross. A load test detects and locates: it tells you the p95 got worse (lesson 4) and points out where the time went (this lesson, up to "the server / with this signature"). There its work ends. Optimizing —lowering that time— is a different discipline with its own tools: code profilers, query execution plans (EXPLAIN), indexes, caches, queues. This guide doesn't teach it, and mixing it in here would be going out of scope.

The reason for keeping the boundary clean is practical: the healthy cycle is measure → detect → locate → optimize → measure again. The first three are load testing (this guide); the fourth is performance engineering (another); and the fifth brings you back here, to verify with a new run that the optimization really lowered the p95 and didn't break anything. The load test is the judge that opens and closes the cycle; the optimization is what happens in the middle, elsewhere. When you locate the bottleneck, the natural link is to the app and database optimization guides of the ecosystem —and then back to a run to verify—.

Common mistakes

Optimizing without locating. What happens: a high p95 is seen and people start "improving" things at random —adding a cache, raising workers— without knowing where the time is. Why it happens: urgency pushes to act before diagnosing. How to detect it: performance changes that don't move the p95, because they attacked the wrong place. How to fix it: locate first (http_req_waiting? → server; connecting? → network), and only then optimize the right stretch. Like the plumber: don't break the wall until you know where the leak is.

Confusing server latency with network latency. What happens: a high p95 is attributed to "the network is slow" when actually the server takes long to respond. Why it happens: the timing breakdown isn't looked at. How to detect it: if http_req_waiting (TTFB) is almost all the latency, the problem is the server, not the network —no matter how much you "improve the network"—. How to fix it: read the breakdown; high waiting = server (app/DB), high connecting/receiving = network.

Believing this guide optimizes. What happens: someone expects the module to teach how to fix the slow query or add the index. Why it happens: detecting and fixing feel like a single job. How to detect it: if you're looking for "how do I make the endpoint faster," you're asking for the "after," which doesn't live here. How to fix it: understand the boundary —this guide measures, detects, and locates; optimization is another discipline— and follow the link to the optimization guides when it's time to fix.

Exercises

Exercise 1 — App, DB, or network? For each symptom, say which is the most likely suspect. (a) Almost all the latency is in http_req_waiting, and it rises when the concurrency grows (more CPU). (b) Almost all in http_req_waiting, but the server is idle (low CPU) and it got worse when the table grew. (c) Almost all in http_req_connecting and http_req_receiving, with large responses and distant clients.

See solution

(a) The app (CPU/code). The time is in the server (waiting) and scales with the concurrency from CPU: the code does heavy work competing for the processor. (Module 5's /quote_cpu.)
(b) The database. The time is in the server (waiting) but the CPU doesn't consume it —the server waits idle for the DB— and it gets worse with the data size: the signature of a slow query or one without an index. (This module's /quote_slow.)
(c) The network. The time goes into connecting and transferring (connecting/receiving), not the server's processing; the large payloads and geographic distance confirm it.

Exercise 2 — Read the breakdown. A summary says: http_req_duration: avg=120ms; http_req_waiting: avg=118ms; http_req_connecting: avg=0.5ms; http_req_receiving: avg=1ms. (a) Is the bottleneck in the network or the server? (b) What would you look at next to separate app from DB? (c) Does this guide tell you how to fix it?

See solution

(a) In the server. http_req_waiting (118 ms) is almost all the latency (120 ms); the network part (connecting + receiving ≈ 1.5 ms) is negligible. The network isn't the problem.
(b) I'd look, on the server side: if the latency rises with the concurrency from CPU (the server is at 100% CPU) → app; if the server is idle waiting and the latency grows with the data → database. Tools: a code profiler, the logs, the DB's metrics and EXPLAIN.
(c) No. This guide takes you to "the problem is in the server, with this signature." Fixing it (index, cache, rewriting the query) is the "after," in the optimization guides.

Exercise 3 — The complete cycle. Order these five steps in the healthy performance cycle and say which are this guide's and which are the "after": optimize, measure, measure again, locate, detect.

See solution

Order: measure → detect → locate → optimize → measure again.

measure (run the load test) — this guide.
detect (the p95 got worse: regression) — this guide (lesson 4).
locate (where did the time go? server/network) — this guide (this lesson).
optimize (fix the query, add the index) — the "after," another discipline.
measure again (confirm the p95 dropped and nothing broke) — this guide again.

The load test opens and closes the cycle (measure, detect, locate, and verify at the end); the optimization happens in the middle, outside this guide. That's why the boundary matters: the judge and the fixer are different trades.

Summary and next step

In this lesson you learned to locate a bottleneck after detecting a regression: the three suspects —app (CPU/code), database (wait), network (bytes' journey)— and their signatures. The key tool is the request's timing breakdown: if the time concentrates in http_req_waiting (TTFB), the problem is in the server (app or DB); if in connecting/sending/receiving, it's in the network. With the two exported runs you made the first real attribution: /quote_slow's ~50 extra ms are in the server (same network in both), with an external-wait signature —the model of a slow DB query—.

And you drew the boundary: this guide measures, detects, and locates; optimizing —lowering the time— is the "after," another discipline with its own tools. The healthy cycle is measure → detect → locate → optimize → measure again, and the load test opens and closes that cycle.

Before moving on you should be able to: name the three suspects and their signals; use the timing breakdown to separate server from network; and explain where this guide ends and optimization begins. What comes next, in lesson 6, is the module's second half —automating—: putting the load test in a CI pipeline, with the .github/workflows/load.yml and the threshold as a gate that blocks the deploy.

Resources

k6 — HTTP request timings — the timing breakdown of a request (http_req_waiting, connecting, sending, receiving) that separates a server bottleneck from a network one. The source of this lesson's content.
Google SRE Book — Monitoring Distributed Systems — how the client's metrics (the load test) are correlated with the server's to locate where the time goes; the foundation of "crossing the server's door."
k6 — Trend (custom metric) — a Trend per flow step, the tool for isolating which of an iteration's several requests is the slow one (complements the timing breakdown of a single request).
e2e-testing-with-playwright-guide — the sibling guide of the Testing ecosystem; along with the app and database optimization guides, it's where the "after" continues when it's time to fix the bottleneck you located here.

← Previous4. Detecting a performance regression Next →6. Running k6 in CI: the `load.yml` and the gate