2. Functional vs load: two distinct questions

Overview

In the previous lesson we stated the distinction; in this one we open it to the bone, because understanding it well is what tells you when to write a load test and what to expect from it. The idea is this: a functional test and a load test are not "two versions of the same thing at different sizes." They're tests of a different nature, that fix different things and measure different things, and that can deliver opposite verdicts on the very same system. A functional test fixes an input and verifies an output: I pass /quote the Focus room, basic tier, 3 hours, and check that it responds exactly 7500. It passes or fails on correctness: either the number is right, or it isn't. A load test fixes a traffic level —how many users, for how long— and measures aggregate properties: latency (how long did it take?), throughput (how many per second?), error rate (how many failed?). It passes or fails on performance under pressure: it doesn't matter whether one number is correct, but whether the system held up and responded on time.

Connection to the module: this lesson is the central contrast everything else hangs from. You already know the Reservo API (lesson 1); here we use it as a bench to see the two tests side by side —one functional request with curl against a stream of concurrent requests with the Python generator—. Lesson 3 will take the load side and break it down into its three questions (users, p95, breaking point). Lesson 4 will show that "a traffic level" has several shapes (the types: smoke, load, stress...). Here we stay on the why: why they're two questions, why you need both, and why the load one can't be deduced from the functional one.

Two questions about the same bridge

An engineer receives a new bridge and runs two kinds of test on it, because they answer two questions that don't follow from each other.

The first: is the bridge well built? They walk across it, check that every joint is welded, that the railing has no gaps, that the asphalt is even. It's an inspection of correctness: is every piece where it should be, does each thing do what it should? A bridge can pass this inspection with flying colors —everything perfectly welded, not a single defect— and we still don't know the second thing.

The second: how much weight does it hold? The engineer puts trucks on it. One, and measures how much it flexes. Ten, and measures again. Fifty, a hundred, until finding the point where the structure starts to yield. It's a load test: it doesn't care whether a joint is well welded (that's already been checked), it cares how much demand it withstands before failing and how much it deforms under each weight level. And here's the crucial part: an impeccably built bridge can collapse with fifty trucks, and an ugly, poorly finished bridge can hold five hundred. The two properties —correctness and capacity— are independent. Knowing the bridge is well welded doesn't tell you how much weight it holds, and knowing it holds a lot of weight doesn't tell you whether the railing has gaps.

Your API is the bridge. The functional test checks the welds: does /quote of Focus/basic/3h return 7500? The load test puts the trucks on: how many concurrent requests does it hold, and how much does it "flex" (latency rise) under each level? You need both inspections, and neither saves you the other.

Worked example: the same API, two tests

We're going to run both tests on the same endpoint —POST /quote— so the difference stops being abstract. Suppose the Reservo API is already running on localhost (we build it in lesson 6; for now it's enough to see the two tests).

The functional test: one request, one correctness verdict

A functional test launches one request with a known input and verifies the output is the expected one. With curl:

What to expect — when requesting a quote of Focus/basic/3h, the response must be exactly {"price_cents": 7500}. This is real output, run against the Reservo server on localhost:

$ curl -s -X POST http://127.0.0.1:PORT/quote \
    -H 'Content-Type: application/json' \
    -d '{"room":"Focus","tier":"basic","hours":3}'
{"price_cents": 7500}

Verdict: correct. The API returned 7500, which is what we expected (2500 × 3). Notice what this test did not measure: we don't know how long it took (well, it took something, but we didn't measure it and didn't care), we don't know what would happen with a hundred of these at once, we don't know whether the response would still be 7500 under pressure. The functional test fixed the input (Focus/basic/3h) and verified the output (7500). Full stop. It's fast, it's cheap, and it answers a single question: is it correct? Yes.

The load test: many requests, one performance verdict

A load test launches many requests —ideally concurrent, to simulate real users hitting at once— and doesn't verify a number, but measures the aggregate behavior. Here we use the mini generator in Python (we build it in lesson 7; for now, look at what it reports). We ask it for 500 requests with 50 concurrent against the same /quote:

What to expect — the generator launches 500 Focus/basic/3h quotes with 50 clients at once, measures each one's latency and reports the aggregate. This is real output, run against the Reservo server:

$ python3.14 load_generator.py http://127.0.0.1:PORT 500 50
requests ............ 500 (concurrency 50)
all returned ........ price_cents=7500 (correct: True)
total duration ...... 0.103 s
throughput .......... 4856.2 req/s
latency min ......... 2.74 ms
latency avg ......... 9.60 ms
latency max ......... 46.19 ms
latency p95 ......... 29.13 ms

Verdict: it depends on your threshold. The load test doesn't say "correct" or "incorrect"; it delivers numbers that you compare against a target. With 50 concurrent users, the API sustained about 4856 requests per second, with a latency that in 95% of cases was under 29 ms, though some reached 46 ms. Is that good? If your target was "p95 under 500 ms," it passes with room to spare. If it was "p95 under 10 ms," it fails. That target —the threshold— is what turns the numbers into a verdict, and it's the subject of module 5.

Notice two things about this output. First: the generator also verified correctness (price_cents=7500 (correct: True)) —even under load, all 500 responses were correct—. That's a correctness check inside the load, something we'll dig into in module 6; a load test can (and usually does) verify that responses stay correct while it measures performance. Second, and more important: look at the gap between the average (9.60 ms) and the maximum (46.19 ms). The average tells one, kind story; the tail of the distribution tells another. That gap is the whole reason the p95 percentile exists, and it's the heart of lesson 3.

Putting them side by side

	Functional test	Load test
What it fixes	An input (Focus/basic/3h)	A traffic level (500 requests, 50 concurrent)
What it measures/verifies	That the output is correct (7500)	Latency, throughput, error rate (aggregates)
How it passes/fails	Correct vs incorrect	The numbers vs a threshold
How many requests	One (or a few)	Many, concurrent
Question it answers	Does it work?	Does it hold up, and how fast under pressure?
Tool in this guide	curl, a functional test	k6 (content), Python generator (executed)

Both hit the same endpoint, with the same correct input, and measured different things. That's the entire point of the lesson.

Why the load one can't be deduced from the functional one

Here's the trap that catches a lot of people: believing that if the system is correct and "feels fast" on one request, then it will hold up under load. It doesn't follow, and the reason is resource contention.

When a single request reaches your API, it has all the resources to itself: a free processor core, plenty of memory, the database connection available. It responds fast —in Reservo, fractions of a millisecond—. But when fifty requests arrive at once, they compete: for cores, for the server's threads, for database connections, for the lock protecting a shared resource. Some have to wait their turn, and that wait adds to their latency. That's why the same endpoint that responds in 0.3 ms with no contention can respond in 29 ms p95 with 50 clients on top. It's not that the code got slower; it's that the requests are queuing.

We can see it with the generator, contrasting two runs against the same API. First with no contention (concurrency 1: one request at a time, each with all the resources):

What to expect — with a single sequential client, there's no queue; the latency is tiny and the p95 almost equal to the average. Real output:

$ python3.14 load_generator.py http://127.0.0.1:PORT 100 1
requests ............ 100 (concurrency 1)
all returned ........ price_cents=7500 (correct: True)
total duration ...... 0.036 s
throughput .......... 2795.5 req/s
latency min ......... 0.22 ms
latency avg ......... 0.33 ms
latency max ......... 6.43 ms
latency p95 ......... 0.35 ms

With a single client, the p95 is 0.35 ms —practically equal to the average (0.33 ms)—: with no queue, almost all requests take the same. Now with contention (concurrency 50), which you already saw above: the p95 rises to 29.13 ms, almost a hundred times more, and pulls away from the average. Same API, same correct code, radically different latency —only how many people hit at once changed—. That's the property a functional test never reveals, because it always measures with one client and an empty kitchen. And it's the reason this guide exists.

Common mistakes

Running a load test with concurrency 1 and believing you measured load. What happens: someone launches 10,000 requests but one at a time (sequential), sees a very low p95, and declares the API "tested under load." Why it happens: many requests total gets confused with many at once. Load is created by concurrency, not by accumulated volume. How to detect it: if your test never has two requests in flight at the same time, there's no contention, and you didn't measure load —you measured 10,000 functional tests in a row—. How to fix it: load is defined by how many users/clients hit simultaneously (k6's VUs, the generator's max_workers). The contrast above (concurrency 1 vs 50) shows it: only the second measures any load.

Expecting a load test to say "correct" or "incorrect." What happens: someone runs k6, sees the summary full of numbers, and doesn't know whether it "passed." Why it happens: they come from the functional world, where a test gives a boolean (green/red). How to detect it: if you look at a p95 of 29 ms and don't know whether it's good or bad, you're missing the target. How to fix it: a load test delivers measurements; the verdict comes from a threshold —a limit like "p95 < 500 ms"— that you define from your requirements (module 5). Without a threshold, the numbers are just information, not approval.

Deducing behavior under load from a single fast request. What happens: a curl that takes 3 ms is measured and someone concludes "the API is fast, it'll hold up." Why it happens: intuition says if one is fast, many are too. But contention breaks that intuition. How to detect it: if your evidence for "holds up under load" is a measurement with no concurrency, you have no evidence. How to fix it: measure with increasing concurrency and observe how the p95 changes —which is exactly what the Python generator does and what k6 does with its VUs—.

Exercises

Exercise 1 — What does each test fix and measure? For each description, say whether it's a functional or a load test, and name what it fixes and what it measures/verifies. (a) "Send /book with Focus/basic/3h and check that confirmed is true." (b) "Launch 1000 requests to /quote with 100 concurrent and report the p95." (c) "Send /quote with a nonexistent room and check it responds 400." (d) "Sustain 200 users quoting for 10 minutes and measure the error rate."

See solution

• (a) Functional. It fixes an input (Focus/basic/3h) and verifies an output (confirmed == true). Correctness of one request.
• (b) Load. It fixes a traffic level (1000 requests, 100 concurrent) and measures a latency metric (p95). Performance under pressure.
• (c) Functional. It fixes an invalid input (nonexistent room) and verifies the correct response to that case (status 400). Correctness of error handling; a single request.
• (d) Load. It fixes a sustained traffic level (200 users, 10 minutes) and measures the error rate. Performance —and, because of the duration, a particular type you'll see in lesson 4 (soak)—.

Exercise 2 — Interpret the average/maximum gap. Look again at the concurrency-50 run: average 9.60 ms, p95 29.13 ms, maximum 46.19 ms. (a) Why is the maximum so far above the average? (b) If any user makes a request during that load, which latency is more honest to promise them: the average or the p95? (c) What would you have to change in the test for the p95 to get closer to the average?

See solution

• (a) Under concurrency, some requests wait their turn (contention): most are served quickly, but a few end up at the back of the queue and accumulate more latency. Those few "long tails" stretch the maximum well above the average, even though they're a minority.
• (b) The p95 is more honest. The average hides the unlucky ones: promising "9.60 ms" ignores that 1 in every 20 users saw 29 ms or more. The p95 says "95% saw this or less," which is a promise you can actually keep. (The why in depth is lesson 3.)
• (c) Lower the concurrency (fewer requests at once → less queue → shorter tail). With concurrency 1, you saw the p95 (0.35 ms) almost equal the average (0.33 ms). The gap grows with contention.

Exercise 3 — A misleading case. A colleague says: "I ran /quote 5000 times with curl in a loop and the average was 0.4 ms, so the API handles load perfectly." Explain in two or three sentences why their conclusion isn't justified, and what test they'd have to run to justify it.

See solution

Their curl loop runs the 5000 requests one after another (sequentially): there are never two in flight at once, so there was no contention and it didn't measure load —it measured 5000 fast functional tests in a row—. A low average with no concurrency says nothing about behavior with many simultaneous users. To justify "handles the load" they'd have to launch the requests concurrently (e.g. 100 clients at once) and look at the p95 and the error rate under that concurrency, not the average of a sequential run.

Summary and next step

In this lesson you saw, with the same API and the same endpoint, that functional and load testing are tests of a different nature. The functional one fixes an input and verifies an output —/quote of Focus/basic/3h → 7500—; it passes or fails on correctness. The load one fixes a traffic level and measures aggregate properties —latency, throughput, error rate—; it passes or fails on performance, according to a threshold you define. They're like the two bridge tests: checking the welds (correctness) and putting the trucks on (capacity) are independent questions, and neither follows from the other.

Above all, you saw why they don't follow: resource contention. A single request has all the resources and responds in 0.3 ms; fifty at once queue up and the p95 jumps to 29 ms —same correct code, latency a hundred times greater—. That's the property a functional test never reveals, because it always measures with an empty kitchen.

Before moving on you should be able to: say what each type of test fixes and what it measures; explain why concurrency (not total volume) is what creates load; and interpret the gap between the average and the maximum of a run as the effect of queuing.

What comes next is breaking down the load side into its concrete questions. In lesson 3 we'll see the three that every load test answers —how many concurrent users does it hold?, what's the p95 latency?, where's the breaking point?— and why, of all the ways to summarize latency, the average is the one that lies most.

Resources

• k6 — What is load testing? — k6's official guide on why and what testing load is, with the test types we'll see in lesson 4.
• Google SRE Book — Monitoring Distributed Systems — the source on why percentiles are measured and not averages; the conceptual frame for why the average lies.
• e2e-testing-with-playwright-guide — the sibling guide that covers the functional side through the browser. If your doubt is about UI correctness and not load, it lives over there.
• urllib.request — Python documentation — the standard-library HTTP client the load generator will make its requests with; we'll use it starting in lesson 7.