Module 6: Real Boundaries Db Files Http

6. The HTTP boundary with a real `http.server`

Next →

Description

Third and last boundary, the most external of the three: HTTP. The database lives on your machine; the file lives on your disk; but Reservo's PaymentGateway lives elsewhere —behind an HTTP API, on a server that isn't yours and that you reach over the network—. Charging a booking isn't calling an object in your memory: it's sending a POST with the amount, waiting for a server to respond, and reading its receipt. That wait is this boundary's novelty. Here, for the first time, the non-determinism we named in lesson 2 becomes the protagonist: the server can lag, can go down, can respond with an error, and your code has to deal with all of that. Testing this boundary is testing that your HTTP client —the one that builds the request, sends it, reads the response, and parses it— speaks the protocol correctly against a real server.

And here's the trick that makes all this possible without installing anything or depending on an external server: you spin up the server yourself, minimal, with the standard library's http.server. A fake PaymentGateway —that responds to POST /charge with a fixed receipt— served over real HTTP, running in a thread of your own process, on a port the operating system picks free. Your client talks to it over TCP exactly as it would talk to the production gateway: the request leaves your process, crosses the network stack, reaches the server, and the response comes back. It's real HTTP —the whole protocol is exercised— but under your control and without leaving your machine. The distinction with the sister guide is hard and worth fixing right away: here we spin up a minimal server to test a client; setting up a web app with a framework, routes, and a complete request-response cycle is testing-backend-applications-guide. Our server is a ten-line scaffold, not the subject of the test.

Connection to the module: this lesson closes the three boundaries. The database one (3, 4) and the file one (5) touched local resources; this one touches the network, the most external and least deterministic resource, and that's why it introduces the timeout —the tool so the wait doesn't hang your test—. With the three boundaries exercised, lesson 7 distills the common discipline (real but ephemeral resources, server in a thread with an ephemeral port, timeouts) and the rule of when to touch the real thing. The border with testing-backend-applications-guide is the hard limit of the whole module: minimal http.server to test the client, yes; a web framework to test the app, no —that's the other guide—.

Analogy: calling another office on the phone

Think of the difference between asking a colleague at your desk something and calling another office on the phone. The colleague next to you, you talk to and they answer instantly, always; you're in the same room, under the same rules. Calling another office is different in every way: you dial a number, the call leaves your building and travels over a network you don't control, on the other side someone has to answer —and they can lag, be busy, or not answer—. You speak in an agreed format ("good morning, I want to place an order, quantity X"), they respond in another ("your confirmation number is Y"), and if nobody picks up the receiver in, say, ten rings, you hang up and try another way —that ring limit is your timeout, the protection against waiting forever—.

The HTTP boundary is that phone call. The HttpPaymentGateway dials the number (POST /charge), the request leaves your process and crosses the network, and on the other side a server has to answer with a receipt. To test that call without depending on the real payments office —which would charge for real and be outside your control—, you set up a fake office that does answer the phone: an http.server that responds to the POST with a fixed receipt. Your client dials, the fake office answers, and you verify that the conversation —dial, speak in the correct format, understand the response— worked. And you put a timeout on it: if the office takes longer than acceptable, you hang up, so a slow server never freezes your suite. This lesson is setting up the office, making the call, and hanging up in time.

The fake gateway, served over real HTTP

Here's the fake office: an HTTP handler that responds to POST /charge with a receipt in JSON. It's the stdlib's http.server —zero dependencies—.

# tests/test_http_boundary.py — a fake PaymentGateway served over HTTP
import json
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer


class FakeGatewayHandler(BaseHTTPRequestHandler):
    """A fake PaymentGateway, served over real HTTP."""

    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(length).decode("utf-8"))
        reply = json.dumps({
            "id": "rcpt-http-1", "ok": True,
            "amount_cents": body["amount_cents"],   # echo of the received amount
        }).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(reply)))
        self.end_headers()
        self.wfile.write(reply)

    def log_message(self, *args):
        pass                                          # silences the server log

Read it for what it is: the minimal office that answers the phone. do_POST runs when a POST arrives. It reads the request body (Content-Length bytes), interprets it as JSON to extract the amount_cents, and responds with a 200 and a JSON receipt that echoes the amount —it returns the amount_cents it received—. That echo is a deliberate detail: if the receipt carries 6000, it's proof that the server received 6000 in the body, that is, that the client sent it correctly. The empty log_message is just so the server doesn't dirty the test output with its log lines. There are no routes, no framework, no validation: it's a scaffold so the client has someone to talk to.

Spinning up the server in a thread, on an ephemeral port

The office has to be listening while the client calls, so we spin it up in a separate thread —so the server attends at the same time the test makes the request— and on an ephemeral port —we pass it port 0 and the operating system picks a free one, avoiding clashes with other processes or with other tests—. We wrap it in a pytest fixture to start and shut it down cleanly.

import pytest
from reservo.http_gateway import HttpPaymentGateway


@pytest.fixture
def gateway_url():
    server = HTTPServer(("127.0.0.1", 0), FakeGatewayHandler)   # port 0 = ephemeral
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()
    try:
        yield f"http://127.0.0.1:{server.server_address[1]}"     # the real URL, with the chosen port
    finally:
        server.shutdown()
        thread.join()
        server.server_close()

Let's break it down. HTTPServer(("127.0.0.1", 0), FakeGatewayHandler) creates the server on localhost with port 0; after creating it, server.server_address[1] tells you which free port the system picked, and with it we build the real URL (http://127.0.0.1:<port>). We run it with serve_forever in a daemon thread (a background thread that doesn't prevent the program from ending). The yield hands the URL to the test; when the test finishes, the finally shuts the server down in order: shutdown() stops the serving loop, join() waits for the thread to end, server_close() frees the port. This pattern —resource spun up before the test, handed over by yield, torn down after— is exactly what lesson 7 formalizes as a resource fixture.

Worked example: a charge that crosses TCP

With the office set up, let's make the call. Three tests: the client charging directly, BookingService charging through the HTTP boundary in a complete flow, and the timeout cutting off a slow wait.

def test_charge_makes_a_real_http_call(gateway_url):
    gateway = HttpPaymentGateway(gateway_url)
    receipt = gateway.charge(6000)                    # real POST, over TCP
    assert receipt.ok is True
    assert receipt.amount_cents == 6000               # the server received 6000
    assert receipt.id == "rcpt-http-1"


def test_book_charges_across_the_http_boundary(gateway_url):
    repo = SqliteBookingRepository(sqlite3.connect(":memory:"))
    service = BookingService(Calendar(), FixedClock(CLOCK),
                             HttpPaymentGateway(gateway_url), SpyEmailSender(), repo)
    booking = service.book(FOCUS, ANA, START, END)    # the charge crosses HTTP for real
    assert repo.get(booking.id).price_cents == 6000

The first test is the pure call: gateway.charge(6000) makes the POST, and the assertions prove that the receipt came back with ok, with the correct amount_cents (6000, the echo that confirms the server received the amount), and with the id the server set. The second is more ambitious: it builds BookingService with the real HttpPaymentGateway as its payment collaborator, and does book. There the booking's charge crosses HTTP for real —the service talks to the gateway, the gateway makes the POST, the server responds— and the booking is persisted in SQLite :memory:. It's an integration that crosses two boundaries at once: the payment one over HTTP and the database one.

What to expect. On my machine (Python 3.14.0, pytest 9.1.1):

python3 -m pytest tests/test_http_boundary.py -v
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 3 items

tests/test_http_boundary.py::test_charge_makes_a_real_http_call PASSED [ 33%]
tests/test_http_boundary.py::test_book_charges_across_the_http_boundary PASSED [ 66%]
tests/test_http_boundary.py::test_timeout_is_enforced PASSED [100%]

============================== 3 passed in 2.03s ===============================

Three greens, and there's no double at the payment seam: it's a real HTTP call, over TCP, to a server running in another thread. The charge left your process, traveled over the network, and came back with its receipt. Notice the 2.03s total: almost all of that time is from the third test, the timeout one, which deliberately talks to a slow server. The first two —the real call— run in milliseconds. That contrast is exactly the topic of the last part.

The timeout: hanging up when nobody answers in time

The network can hang, and a client without a timeout would wait forever —freezing your test, your suite, your CI—. That's why HttpPaymentGateway passes a timeout to urlopen, and we have to prove that it really cuts off. We set up a second office that answers late —it sleeps a second before responding— and talk to it with a client whose timeout is a tenth of a second: it must give up before the server answers.

import time


class SlowGatewayHandler(FakeGatewayHandler):
    def do_POST(self):
        time.sleep(1.0)                               # slower than the timeout
        super().do_POST()


# ...a slow_gateway_url fixture that serves SlowGatewayHandler, like gateway_url...


def test_timeout_is_enforced(slow_gateway_url):
    gateway = HttpPaymentGateway(slow_gateway_url, timeout=0.1)
    with pytest.raises(TimeoutError):
        gateway.charge(6000)

The slow server sleeps a whole second before responding; the client has a timeout=0.1, so at a tenth of a second it gives up and raises TimeoutError. The pytest.raises(TimeoutError) asserts that this is exactly what happens: the client hung up instead of waiting the full second. This tests a critical part of your client's behavior against the network —that it doesn't hang forever— that no stub could exercise, because a stub responds instantly and never lags. And it explains the 2.03s total: the slow server keeps sleeping its second in the background even though the client already hung up, plus the second of the sleep itself that the thread completes before shutting down. The timeout is your ring limit: you protect the suite from a server that doesn't answer.

Why this is real HTTP and not a web framework

It's worth fixing the limit, because it's the module's hard border. What you did is real HTTP: there's a TCP connection, a request with its method and its body, a response with its status code and its JSON, a real timeout. The HttpPaymentGateway client exercises its whole protocol. But the server is deliberately minimal: a do_POST that responds just enough, without routes (/charge is the only thing it attends, and it doesn't even discriminate the path), without validation, without a data model, without error handling, without the thousand things a web framework brings. And it's that way on purpose: here the subject of the test is the client, and the server is only the scaffold so it has someone to talk to.

Testing a real web app is the opposite: there the subject is the server —your routes, your validation, your request-response logic, your dependencies—, and it's tested with a framework's tools (FastAPI's TestClient, for example). That's testing-backend-applications-guide, and it's a whole discipline. The rule for not crossing the line: if you find yourself writing routes, validation, or business logic in your test server, you left this guide's scope —or you're testing the wrong server—. Here, the server is three lines that answer the phone; nothing more.

Common mistakes

Using a fixed port instead of an ephemeral one. What happens: someone spins up the test server on a fixed port (8080), and the test fails intermittently with Address already in use when that port is occupied —by another test, by a previous run that didn't close, by another app—. Why it happens: a fixed number is the first thing you write. How to detect it: if your HTTP tests fail sometimes with address-in-use errors, or you can't run two at once, it's the fixed port. How to fix it: pass port 0 to HTTPServer and read the one the system picked with server.server_address[1], as in the fixture. An ephemeral port never clashes, and lets you run tests in parallel.

Forgetting to shut down the server (or the thread). What happens: someone starts the server in a thread and doesn't shut it down; the threads accumulate between tests, the port stays occupied, or the suite doesn't finish cleanly. Why it happens: the thread runs in the background and "seems" not to bother. How to detect it: ports that stay occupied, runs that hang at the end, or warnings about live threads. How to fix it: wrap the startup in a fixture with try/finally (or yield), and in the teardown call shutdown(), join(), and server_close(), as in gateway_url. A daemon thread prevents the process from hanging if something fails, but the explicit shutdown is the correct thing.

Not putting a timeout on the client. What happens: someone calls urlopen(request) without a timeout; against a server that hangs, the test waits indefinitely and blocks the whole suite. Why it happens: in local tests the server always responds fast, so the missing timeout goes unnoticed. How to detect it: if a network test can get stuck when the other end doesn't respond, the timeout is missing. How to fix it: always pass a timeout to urlopen (as HttpPaymentGateway does), and test that it cuts off —with a slow server and pytest.raises(TimeoutError)—. An HTTP client without a timeout is a time bomb in any suite; the network boundary can always lag.

Exercises

Exercise 1 — What the echo tests. The FakeGatewayHandler responds with "amount_cents": body["amount_cents"] —it echoes the amount it received—, and the test asserts receipt.amount_cents == 6000. Explain what that echo tests exactly that a receipt with a fixed amount ("amount_cents": 6000 hardcoded in the server) wouldn't test.

See solution

The echo tests that the amount actually traveled from the client to the server: that HttpPaymentGateway.charge(6000) serialized 6000 in the JSON body, sent it in the POST, the server read it from the body (body["amount_cents"]), and returned it. If receipt.amount_cents == 6000, it's because that 6000 made the complete circuit —out in the request, back in the response—. It tests, specifically, that the client builds the request body correctly.

A receipt with a fixed amount ("amount_cents": 6000 hardcoded in the server) wouldn't test that: the server would return 6000 regardless of what the client sent. If the client had a bug and sent {"amount": 6000} (with the wrong key) or {"amount_cents": 60} (the wrong amount), the test would still pass, because the receipt's 6000 was put there by the server out of its own pocket, it didn't come from the client. The echo closes that blind spot: by returning what it received, the server lets you verify that the client sent the correct thing. It's a small but important technique so that testing an HTTP client verifies the sending, not just the receiving.

Exercise 2 — Why a thread? The fixture runs server.serve_forever() in a separate thread, not in the test's thread. Explain what would happen if the test called server.serve_forever() directly in its own thread, before making the request.

See solution

If the test called server.serve_forever() in its own thread, it would hang there forever and never get to make the request. serve_forever() is a blocking loop: it keeps attending requests indefinitely and doesn't return control until someone calls shutdown() from another thread. Since the test would be trapped inside serve_forever(), it would never run the next line (gateway.charge(6000)), so there'd be no client making the request —the server would be listening, but nobody would call—. The test would be frozen.

That's why the server goes in a separate thread: that way it runs "in the background", attending, while the test's thread continues its course and makes the request. The two threads work at once —one serves, the other calls— which is exactly what a client-server conversation needs: someone listening and someone speaking, simultaneously. The daemon thread also guarantees that, if something goes wrong and it doesn't shut down cleanly, the process can end anyway without being hung by the server. Running the server in the same thread as the client is asking a single person to answer the phone and speak on it at the same time: impossible.

Exercise 3 — Where the border with the other guide is. For each task, say whether it fits in this guide (test the HttpPaymentGateway client with a minimal http.server) or in testing-backend-applications-guide (test a web app with a framework): (a) verify that the client retries a charge if the server responds 503; (b) verify that your API's POST /bookings route validates the body and returns 422 if a field is missing; (c) verify that the client raises a clear error if the server returns malformed JSON; (d) verify that your API, with its database and its authentication, creates a booking end to end.

See solution
  • (a) This guide. The subject is the HttpPaymentGateway client and its behavior against server responses. You set up a minimal http.server that responds 503 and verify that the client retries. No framework needed: the server is a scaffold that returns whatever code you want.
  • (b) The other guide. The subject is a route of your API —its validation, its 422 code, its body model—. That's server logic tested with a web framework's tools (FastAPI's TestClient, for example). You crossed the line: here we don't set up routes or validation.
  • (c) This guide. Again the subject is the client: how it reacts to a malformed body. You set up an http.server that responds with non-JSON text and verify that the client raises a clear error. Minimal scaffold, client under test.
  • (d) The other guide. "Your API end to end, with its database and its authentication" is exactly a complete web app —the subject is the whole server, with its framework, its routes, its request-response cycle—. It's the heart of testing-backend-applications-guide.

The rule you're sharpening: if the subject of the test is your HTTP client (how it builds requests and reacts to responses), it's this guide, with a minimal http.server as scaffold. If the subject is your server/app (routes, validation, request-response logic), it's the other guide, with a framework. The minimal server here exists to give the client someone to talk to; never to be tested itself.

Summary and next step

In this lesson you crossed the third boundary, the HTTP one, the most external of the three. With the phone call to another office you understood that charging over HTTP is sending a request that leaves your process, crosses the network, and depends on someone answering in time. You spun up a fake PaymentGateway served over real HTTP with http.server, in a thread and on an ephemeral port, and tested the HttpPaymentGateway client against it with real output: a POST that crossed TCP and came back with its receipt (amount_cents == 6000, the echo that confirms the send), a complete book charging through the boundary, and a timeout that hung up on a slow server instead of waiting forever. And you fixed the hard border: minimal http.server to test the client is this guide; a web framework to test the app is testing-backend-applications-guide.

Before moving on you should be able to: spin up a minimal http.server in a thread with an ephemeral port and shut it down cleanly; test an HTTP client against it, including the timeout; and decide whether a task falls in this guide (test the client) or in the backend apps one (test the server).

With the three boundaries exercised —database, files, HTTP—, what comes next is distilling the common discipline. In lesson 7 we bring together the three keys that made each test fast and deterministic (:memory: or tmp_path instead of shared resources, server in a thread with an ephemeral port, timeouts that bound the wait) and formulate the decision rule that governs the whole module: double what you don't control or is slow on the path you don't test; touch the real thing at the boundary you do test.

Resources

← Previous5. The file boundary with `tmp_path`Next →7. Fast and deterministic: when to touch the real thing and when to double