Module 3: Metrics Latency Throughput Errors

8. Mini-project: measure Reservo's metrics

Next →

Overview

The moment has come to put the whole module in your hands. In this mini-project you bring up the Reservo API —with the declared slow endpoint—, run a load generator over it, and produce a complete metrics report: p50, p95, and p99 latency, RPS, and error rate, all actually measured. Then you write the equivalent k6 summary block as content, and interpret in writing the question that runs through the whole module: why the p95 matters to the user more than the average. It's not an exercise in repeating definitions; it's the real work of whoever interprets a load test, done by you from start to finish.

Connection to the module: this project exercises the seven lessons at once. You bring up the API (module 1's infrastructure), launch VUs (module 2), measure latency and compute percentiles (lessons 2-3), report throughput (lesson 4) and error rate (lesson 5), write and read a k6 summary (lesson 6), and interpret average vs p95 (lesson 7). It's the module's close and the prelude to module 4, where instead of a fixed load you'll start to vary the load over time (ramps, spikes) and observe how these same metrics move.

What you'll deliver

Your deliverable has four pieces:

  1. The Reservo API (server.py) running, with the slow endpoint /quote_slow declared.
  2. The load generator (loadgen.py) that hits an endpoint, measures each latency, and computes p50/p95/p99, RPS, and % error.
  3. The real output of a run over /quote_slow (measured metrics) and one over /quote_flaky (for the error rate).
  4. A written interpretation (a paragraph) that answers: for the Reservo user, why does the p95 say more than the average? With your own numbers.

Step 1: the Reservo API (with the slow endpoint declared)

This is the canonical Reservo server, identical to the whole guide's, plus the two lab endpoints this module declares (/quote_slow and /quote_flaky). Save it as server.py. Notice it's served on port 0: the operating system assigns a free one and the server prints it on its first line, to avoid colliding with anything.

# server.py — Reservo API (canonical) + module-3 declared endpoints
import json, random, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

# Hourly rates in integer CENTS (never float for money).
HOURLY_CENTS = {"Focus": 2500, "Studio": 4000, "Boardroom": 8000}

def price_cents(room, tier, hours):
    """Price in cents: rate * hours, with integer 20% discount if pro."""
    base = HOURLY_CENTS[room] * hours
    if tier == "pro":
        return base * 80 // 100   # integer discount, no decimals
    return base

class ReservoHandler(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass  # silence: don't pollute the test output

    def _send(self, status, payload):
        body = json.dumps(payload).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def _read_json(self):
        length = int(self.headers.get("Content-Length", 0))
        return json.loads(self.rfile.read(length) or b"{}")

    def do_GET(self):
        if self.path == "/rooms":
            rooms = [{"room": r, "hourly_cents": c} for r, c in HOURLY_CENTS.items()]
            self._send(200, {"rooms": rooms})
        else:
            self._send(404, {"error": "not_found"})

    def do_POST(self):
        body = self._read_json()
        room, tier, hours = body.get("room"), body.get("tier"), body.get("hours")

        if self.path == "/quote":
            self._send(200, {"price_cents": price_cents(room, tier, hours)})

        elif self.path == "/quote_slow":
            # DECLARED: simulates a slow dependency with a long TAIL.
            if random.random() < 0.10:
                time.sleep(random.uniform(0.12, 0.25))    # the tail
            else:
                time.sleep(random.uniform(0.004, 0.012))  # the common case
            self._send(200, {"price_cents": price_cents(room, tier, hours)})

        elif self.path == "/quote_flaky":
            # DECLARED: responds fast, but ~10% fails with 500.
            if random.random() < 0.10:
                self._send(500, {"error": "upstream_unavailable"})
            else:
                self._send(200, {"price_cents": price_cents(room, tier, hours)})

        elif self.path == "/book":
            self._send(200, {
                "booking_id": f"bk_{room}_{tier}_{hours}",
                "price_cents": price_cents(room, tier, hours),
                "confirmed": True,
            })
        else:
            self._send(404, {"error": "not_found"})

class ReservoServer(ThreadingHTTPServer):
    daemon_threads = True
    request_queue_size = 256   # large backlog: the queue is the endpoint's, not the socket's

if __name__ == "__main__":
    server = ReservoServer(("127.0.0.1", 0), ReservoHandler)  # port 0: the OS assigns
    print(server.server_address[1], flush=True)               # 1st line: the port
    server.serve_forever()

Before measuring, verify the anchors —that the API is still the usual one— by starting it and hitting it with curl. This is the real output:

Focus/basic/3h -> {"price_cents": 7500}
Focus/pro/3h   -> {"price_cents": 6000}

If you see 7500 and 6000, your Reservo is the canonical one and you can measure with confidence.

Step 2: the load generator

This is loadgen.py, the generator that brings together everything in the module. It receives the port, the endpoint, the total number of requests, and the concurrency; measures each latency client side, counts the errors, and at the end computes and prints the complete report with statistics.

# loadgen.py — generates load and reports REAL p50/p95/p99, RPS, and % error
import json, statistics, sys, time, urllib.request
from concurrent.futures import ThreadPoolExecutor

PORT = int(sys.argv[1])
PATH = sys.argv[2] if len(sys.argv) > 2 else "/quote"
TOTAL = int(sys.argv[3]) if len(sys.argv) > 3 else 2000
CONCURRENCY = int(sys.argv[4]) if len(sys.argv) > 4 else 50

URL = f"http://127.0.0.1:{PORT}{PATH}"
PAYLOAD = json.dumps({"room": "Focus", "tier": "basic", "hours": 3}).encode()

def one_request():
    """One request. Returns (latency_ms, ok). Timed CLIENT side."""
    start = time.perf_counter()
    try:
        req = urllib.request.Request(URL, data=PAYLOAD,
                                     headers={"Content-Type": "application/json"})
        with urllib.request.urlopen(req, timeout=10) as resp:
            resp.read()
            ok = (resp.status == 200)
    except Exception:
        ok = False           # timeout, dropped connection, 500... all is failure
    return (time.perf_counter() - start) * 1000, ok

def pct(data, p):
    """Percentile p (1-99) with statistics.quantiles, inclusive method."""
    return statistics.quantiles(data, n=100, method="inclusive")[p - 1]

def main():
    latencies, errors = [], 0
    wall_start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
        for latency_ms, ok in pool.map(lambda _: one_request(), range(TOTAL)):
            latencies.append(latency_ms)
            if not ok:
                errors += 1
    wall = time.perf_counter() - wall_start
    latencies.sort()

    print(f"endpoint         {PATH}")
    print(f"total requests   {TOTAL}")
    print(f"concurrency      {CONCURRENCY} clients")
    print(f"throughput RPS   {TOTAL / wall:8.1f} req/s")
    print(f"errors           {errors} ({errors / TOTAL * 100:.2f}%)")
    print("-- latency (ms), client side --")
    print(f"avg (average)    {statistics.fmean(latencies):8.2f}")
    print(f"p50              {pct(latencies, 50):8.2f}")
    print(f"p95              {pct(latencies, 95):8.2f}")
    print(f"p99              {pct(latencies, 99):8.2f}")

if __name__ == "__main__":
    main()

Step 3: measure (the real output)

Start the server, capture the port it prints, and run the generator. In one terminal:

# start the API; its first line is the port assigned by the OS
python3 server.py
# -> (prints something like 55870)

# in another terminal, with that port, run the generator over the slow endpoint
python3 loadgen.py 55870 /quote_slow 2000 50

What to expect on /quote_slow. Since the endpoint has a long tail, you'll see a moderate average but a p95 and p99 well above it —the tail's signature—. This is the real output:

endpoint         /quote_slow
total requests   2000
concurrency      50 clients
throughput RPS     1517.6 req/s
errors           0 (0.00%)
-- latency (ms), client side --
avg (average)       28.64
p50                 12.18
p95                182.81
p99                240.31

Now measure the error rate over the flaky endpoint:

python3 loadgen.py 55870 /quote_flaky 2000 30

What to expect on /quote_flaky. Excellent latency, but ~10% error. Real output:

endpoint         /quote_flaky
total requests   2000
concurrency      30 clients
throughput RPS     4761.1 req/s
errors           200 (10.00%)
-- latency (ms), client side --
avg (average)        6.25
p50                  5.45
p95                 10.06
p99                 27.71

With these two runs you have all the material for the report: from /quote_slow, the tailed-latency story (p95 = 182.81 ms, well above the 28.64 average); from /quote_flaky, the error-rate story (10% with a latency that, alone, would look healthy). Note that your numbers won't be identical to these down to the millisecond cent —a load test varies a bit between runs, depending on the machine and its state—, but the shape will be the same: on /quote_slow, p95 ≫ average; on /quote_flaky, ~10% error with low latency.

Step 4: the equivalent k6 summary (content)

As part of the deliverable, write what the k6 run summary would look like for an equivalent test over /quote_slow. Remember: k6 isn't installed; this is labeled content, faithful to the official format, not run. Map your measured metrics to the k6 lines:

  # CONTENT (this is how `k6 run` looks; k6 is not installed)

  █ TOTAL RESULTS

    HTTP
    http_req_duration..................: avg=28.6ms min=4.9ms med=12.2ms max=257ms p(90)=38.6ms p(95)=182.8ms
    http_req_failed....................: 0.00%   0 out of 2000
    http_reqs..........................: 2000    1517.6/s

    EXECUTION
    iterations.........................: 2000    1517.6/s
    vus................................: 50      min=50   max=50
    vus_max............................: 50      min=50   max=50

Notice the bridge: your Python p95 = 182.81 is k6's p(95)=182.8ms column; your errors 0 (0.00%) is http_req_failed 0.00%; your RPS 1517.6 is the rate of http_reqs 1517.6/s; your concurrency 50 is vus_max 50. It's not magic: it's the same thing you measured, in k6's format.

Step 5: the written interpretation

Close the deliverable with a paragraph that answers, with your numbers, the module's question: for the Reservo user, why does the p95 say more than the average? An example of a well-done answer:

On /quote_slow, the average latency was 28.64 ms, but the p95 was 182.81 ms —more than six times greater—. The average describes a user who almost doesn't exist: half the requests (p50 = 12.18 ms) were faster than half of that average, while 1 in every 20 users (the p95) waited 183 ms or more. If I promised the business "the quote takes 29 ms on average," I'd be hiding that a real fraction of the customers lives a wait six times worse. The p95 makes that tail user visible —the one who gets frustrated and maybe abandons—, and that's why it's the metric an SLO is written over (p(95) < 200 ms), not the average. Also, the /quote_flaky run reminds us that latency isn't enough: there the p95 was excellent (10 ms) but 10% of the requests failed, so "fast" didn't mean "good." The honest verdict uses the three instruments: latency p95, RPS, and error rate, read together.

Self-assessment rubric

Check each point. If you fail one, go back to the indicated lesson.

#CriterionDo you meet it?Lesson
1The API starts on port 0 and returns the anchors (7500, 6000).☐1
2The generator measures each latency client side and counts errors (ok = status == 200).☐2, 5
3You compute p50/p95/p99 with statistics.quantiles(..., n=100, method="inclusive").☐3
4You report the RPS as total / wall_time.☐4
5You report the error rate as errors / total and read it first.☐5
6On /quote_slow, your p95 is much larger than your average (the tail).☐7
7On /quote_flaky, you have ~10% error with low latency.☐5
8You write the k6 summary as labeled content, not as something executed.☐6
9Your interpretation explains, with your numbers, why the p95 > average matters.☐7
10Your verdict reads the three instruments together (latency, throughput, error).☐1, 5

Common mistakes

Reporting only /quote_slow and forgetting the error rate. What happens: the deliverable has a gorgeous p95 tail analysis, but no run with errors, so the third instrument goes unexercised. Why it happens: the tail lesson is the flashy one and steals the attention. How to detect it: if your report has no run with % error > 0, you're missing half the verdict. How to fix it: include the /quote_flaky run and read it in the correct order (error first).

Presenting the k6 summary as if you had run it. What happens: the deliverable pastes a k6 block with no label, implying k6 run was run. Why it happens: the environment rule is forgotten. How to detect it: if your k6 block doesn't say "content, not run," it breaks the guide's honesty. How to fix it: always label it. What you actually ran is the Python generator; the k6 summary is content faithful to the format.

Interpreting with adjectives instead of numbers. What happens: the interpretation says "the p95 is important because it shows the real experience," without a single number. Why it happens: it's easier to repeat the lesson than to apply it. How to detect it: if your paragraph doesn't cite your p95, your average, and your error rate, it's generic. How to fix it: anchor each statement in a measured number of yours ("my average was 28.64 but my p95 was 182.81, six times more").

Summary and next step

In this mini-project you measured Reservo's metrics with your own hands, from start to finish: you brought up the canonical API with its declared slow endpoint, ran a load generator that computes real p50/p95/p99, RPS, and % error, wrote the equivalent k6 summary as labeled content, and interpreted with your own numbers why the p95 tells the user more than the average. You saw, measured by you, the module's two truths: on /quote_slow, a p95 (182.81 ms) six times larger than the average (28.64) —the tail the average hides—; and on /quote_flaky, 10% error with excellent latency —the reminder that "fast" isn't "good"—.

With this you close the guide's interpretive heart. You now know what the three metric families are, how they're computed, how they're read in the k6 summary, and —most important— how to draw an honest verdict by reading them together and in the correct order. Before continuing you should be able to do the whole project without looking at the lessons: bring up, measure, report, and interpret.

What comes next, in module 4, is to stop measuring a fixed load and start to model how the load varies over time: the load profiles (stages, ramp-up and ramp-down, spikes) and k6's executors. The question changes from "what metrics does this load produce?" to "how do these metrics move when the load rises, holds, and falls?" —and to answer it you need exactly the instruments you mastered here—.

Resources

← Previous7. Average vs p95 on a tailed distributionNext →1. Module introduction: giving the load a shape