5. Reporting hooks: header and summary

Overview

A framework that runs well but says nothing about itself leaves the team guessing: against what environment did this run?, what framework version?, how many really passed? This lesson gives the Reservo framework a voice with the two reporting hooks: pytest_report_header, which prints information at the start of the run —the version, the environment, the seed, under the session header—, and pytest_terminal_summary, which prints a summary at the end —a banner with the result, after pytest's count of passed and failed—. Between the two they frame each run: a header that announces with what configuration it starts, and a footer that summarizes how it ended.

You already wrote a minimal version of pytest_report_header in lesson 2 (it returned one line with the version). Here you complete it: you add the environment and the seed, reading them from the config with the --env of lesson 3. And you meet the new hook, pytest_terminal_summary, which brings with it an object you had not used yet: the terminalreporter, pytest's interface for writing to the terminal. You are going to learn its three tools —write_sep (a separator line with a centered title), write_line (a line of text), and terminalreporter.stats (the dictionary with the results, from which the passed and the failed are counted)—. All executed: the header reservo framework: v1.0.0 at the top, the banner reservo framework summary at the bottom, and the summary reacting to a failure (7 passed, 1 failed).

Connection to the module: these two hooks consume what you built before. pytest_report_header and pytest_terminal_summary read the --env you declared with pytest_addoption in lesson 3 —it is the first real use of that lever within the framework itself—. And they are the last piece of the kit of four hooks before lesson 6 looks at the conftest from the outside (what separates it from a packaged plugin?) and lesson 7 gives the rule of when to use each tool. The boundary with module 7 stays firm: the header shows the environment, but what the framework does differently in ci than in local is there; here we only announce it.

Analogy: the cover page and the meeting minutes

Think of a formal meeting that leaves a record. Before starting, the secretary writes a cover page: date, place, who attends, the session number. It is not the meeting's content —it is the frame that places it—, and it goes at the very top so whoever reads the record knows, at a glance, what meeting it is about. pytest_report_header is that cover page: it runs at the start, when nothing has happened yet, and its job is to announce the context —"framework v1.0.0, environment ci, seed 1234"— so the rest of the output is read with that frame in place.

When the meeting ends, the secretary writes the closing minutes: how many topics were resolved, how many were left pending, the result. It goes at the end, and can only be written at the end, because until the meeting ends the count is not known. pytest_terminal_summary is those minutes: it runs at the close of the session, when all the results are already known, and that is why it can say "7 passed, 1 failed" —a datum that did not exist at the start—. The cover page announces the intent; the minutes report the outcome. A good record has both, and each at its moment: the one written before knowing anything, and the one that can only be written when everything has happened.

pytest_report_header: the cover page, complete

In lesson 2 the header had one line. Now we enrich it so it announces the environment and the seed, reading them from the config:

# conftest.py  (project root)
FRAMEWORK_VERSION = "1.0.0"


def pytest_report_header(config):
    # Start anchor: framework information under the header.
    env = config.getoption("--env")
    seed = config.getoption("--seed")
    return [f"reservo framework: v{FRAMEWORK_VERSION}",
            f"reservo env: {env}  |  seed: {seed}"]

The novelties with respect to the minimal version of lesson 2:

• It reads the config. config.getoption("--env") and config.getoption("--seed") obtain the values of the options you declared in lesson 3. This is the correct moment to read them: pytest_report_header runs after pytest parsed the command line, so the options already have a value. (Compare with pytest_addoption, which runs before and where the options do not yet have any value.)
• It returns a list of strings. Before it returned a single string; now a list of two. pytest_report_header accepts both forms: a string is one line, a list of strings is several lines, each under the header. Returning a list is how you add several lines of context without concatenating them by hand.
• It is still return, not print. As in lesson 2: you hand the lines to pytest so it places them in the header, in its order and with its format. You do not print them yourself.

The result is a two-line cover page —what framework, and with what environment and seed— that will appear right below pytest's platform ... line.

pytest_terminal_summary: the minutes, and the terminalreporter

The end hook is different in one key point: it does not return text for pytest to print; it receives the object that writes to the terminal and writes itself. Here it is:

# conftest.py  (project root)
def pytest_terminal_summary(terminalreporter, exitstatus, config):
    # End anchor: framework summary, after pytest's summary.
    tr = terminalreporter
    env = config.getoption("--env")
    n_passed = len(tr.stats.get("passed", []))
    n_failed = len(tr.stats.get("failed", []))
    tr.write_sep("=", "reservo framework summary")
    tr.write_line(f"framework:  v{FRAMEWORK_VERSION}")
    tr.write_line(f"env:        {env}")
    tr.write_line(f"result:     {n_passed} passed, {n_failed} failed")

The three new pieces —the three parts of the terminalreporter— each deserve their paragraph, because they are the vocabulary for writing to the terminal from a hook:

• terminalreporter (here abbreviated tr). It is the object pytest uses to print everything you see on screen —the header, the progress dots, the final count—. In lesson 2 you saw it registered as a plugin in --trace-config (TerminalReporter object). pytest passes it to this hook so you write through it, with its format and in the correct place of the output. It is the difference with report_header: there you return text, here you use the reporter.
• tr.write_sep("=", "title"). Writes a separator line that fills the width of the terminal with the character you give it ("=") and centers a title in the middle: ======= title =======. It is the same kind of banner pytest uses for its own sections (==== test session starts ====). It gives your summary pytest's native look, not that of a loose print.
• tr.write_line("text"). Writes a normal line of text, without adornments. It is what you use for the body of the summary, line by line.
• tr.stats. The dictionary with the run's results, grouped by outcome: tr.stats["passed"] is the list of tests that passed, tr.stats["failed"] the one of the ones that failed, and there are keys for skipped, error, etc. Counting is len(...) of the list. We use .get("passed", []) instead of ["passed"] for safety: if no test passed, the key "passed" may not exist, and .get(..., []) returns an empty list instead of a KeyError. It is the source of the "how many passed and how many failed", and is only populated at the end —that is why this datum lives in the closing hook, not in the start one—.

The exitstatus parameter (the run's exit code) is in the signature because pytest passes it, even though we do not use it here; you could query it to say "all green" versus "there were failures". And config is there to read --env, just like in the header.

Worked example: the run framed, header at the top and summary at the bottom

Run the Reservo suite with --env=ci, with the two hooks in place:

python3 -m pytest --env=ci

What to expect. On my machine (Python 3.14.0, pytest 9.1.1):

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
reservo framework: v1.0.0
reservo env: ci  |  seed: 1234
rootdir: /private/tmp/reservo_mini
collected 7 items

tests/unit/test_pricing.py ..                                            [ 28%]
tests/unit/test_refunds.py ...                                           [ 71%]
tests/integration/test_book_flow.py ..                                   [100%]

======================= reservo framework summary =======================
framework:  v1.0.0
env:        ci
result:     7 passed, 0 failed
============================== 7 passed in 0.01s ===============================

Read the output as a frame. At the top, under platform ... and above rootdir ..., the two header lines: reservo framework: v1.0.0 and reservo env: ci | seed: 1234. That is the cover page, with the ci it read from --env. At the bottom, after the test block but before pytest's final 7 passed, the banner reservo framework summary (the work of write_sep) with three lines (the work of write_line), and the result: 7 passed, 0 failed that came from counting tr.stats. The run ended up framed: who the framework is and with what environment at the top, how it went at the bottom.

The summary reacts to a failure

The value of reading tr.stats is that the summary really counts, does not repeat a fixed number. To see it, temporarily add a test that fails on purpose and run again:

# tests/unit/test_broken_demo.py  (temporary, to see the summary count a failure)
from reservo.pricing import price_cents


def test_pro_discount_is_wrong_on_purpose(focus_room, pro_member):
    # Deliberate failure: the pro pays 6000, not 6500.
    assert price_cents(focus_room, pro_member, 3) == 6500

What to expect (the end of the run with --env=ci):

======================= reservo framework summary =======================
framework:  v1.0.0
env:        ci
result:     7 passed, 1 failed
=========================== short test summary info ============================
FAILED tests/unit/test_broken_demo.py::test_pro_discount_is_wrong_on_purpose - AssertionError: assert 6000 == 6500
========================= 1 failed, 7 passed in 0.03s ==========================

result: 7 passed, 1 failed: the summary read len(tr.stats["passed"]) (seven) and len(tr.stats["failed"]) (one) of this concrete run and reported them. It is not a hardcoded text; it is the real state of the tests, counted at the close. Notice also the order: your framework banner comes out before pytest's short test summary info and the final 1 failed, 7 passed —pytest_terminal_summary inserts itself in the reporting phase, next to pytest's own sections—. Delete the broken test and the suite goes back to 7 passed, 0 failed.

Why the header cannot count and the summary can

It is worth underlining the asymmetry, because it is the underlying lesson about the lifecycle. pytest_report_header could not say "7 passed": it runs at the start, before a single result exists, so tr.stats would be empty. It can only announce what is already known at startup —the configuration: version, environment, seed—. pytest_terminal_summary can count, because it runs at the end, when tr.stats is populated with the outcome of each test. Each hook has available exactly the information of its phase: the start one knows the intent (what it is run with), the end one knows the result (how it went). Choosing the correct hook is, again, choosing the correct moment —wanting to count failures in the header is asking the cover page for a datum only the minutes have—.

Common mistakes

Putting the result count in pytest_report_header. What happens: someone tries to make the header say "0 failures so far" by reading tr.stats from pytest_report_header, and always gets zero —or an error, because the header does not receive terminalreporter—. Why it happens: it is assumed that "the header and the summary are the same in different places", when they run in opposite phases. How to detect it: if your header always reports zero passed/failed no matter how the suite ends, you are reading results that do not yet exist. How to fix it: the result count goes in pytest_terminal_summary (end, with tr.stats populated); the header (start) only announces configuration. They are two phases, two available data.

Using print inside pytest_terminal_summary instead of the terminalreporter. What happens: someone writes print("summary...") inside the summary hook, and the line comes out misplaced —captured, or before/after where it should, without pytest's format—. Why it happens: print is the natural reflex for "write something". How to detect it: if your summary appears with a format that clashes (without the terminal-width banner, or in an odd place with respect to the short test summary), it is a print. How to fix it: use the object pytest passes you —terminalreporter.write_sep for the banner, terminalreporter.write_line for the lines—. Writing through the reporter gives you the native format and the correct place in the reporting phase; print competes with pytest's output capture and loses.

Indexing tr.stats["passed"] when there are no passed (or ["failed"] when there are no failures). What happens: someone writes len(tr.stats["failed"]) and in a perfect run (zero failures) gets KeyError: 'failed', which blows up the summary itself. Why it happens: tr.stats only has the keys of the outcomes that occurred; if no test failed, there is no "failed" key. How to detect it: a KeyError inside pytest_terminal_summary, right when the suite came out all green (or all red). How to fix it: use tr.stats.get("failed", []) instead of tr.stats["failed"] —.get with an empty-list default returns [] when the key is not there, so the len gives zero without breaking—. It is exactly why the lesson's hook uses .get(..., []) and not the direct indexing.

Exercises

Exercise 1 — Add the skipped to the summary. The team wants the summary to also report how many tests were skipped (skipped). Write the line you would add to pytest_terminal_summary to include skipped: N, and explain why you use .get("skipped", []) instead of ["skipped"].

n_skipped = len(tr.stats.get("skipped", []))
tr.write_line(f"skipped:    {n_skipped}")

Exercise 2 — Header or summary? For each datum the team wants to show, say whether it goes in pytest_report_header (start) or in pytest_terminal_summary (end), and why. (a) The framework version. (b) How many tests failed. (c) The environment (--env) against which it ran. (d) The total time of the run. (e) The seed used to generate data.

Exercise 3 — Diagnose the summary that blows up. A coworker wrote this pytest_terminal_summary, and in the runs where all the tests pass, the summary fails with KeyError: 'failed': n_failed = len(terminalreporter.stats["failed"]); terminalreporter.write_line(f"failed: {n_failed}"). Why does it blow up right when everything goes well, and what is the fix?

Summary and next step

In this lesson you gave the framework a voice with the two reporting hooks. pytest_report_header is the cover page: it runs at the start and announces the configuration —version, environment and seed, reading the --env and --seed of lesson 3 with config.getoption— returning a list of lines pytest places under the header. pytest_terminal_summary is the closing minutes: it runs at the end and writes a summary through the terminalreporter object —write_sep for the banner with a centered title, write_line for the body— counting the real results with tr.stats.get("passed", []) and .get("failed", []). You saw it frame the Reservo run (header at the top, banner at the bottom) and the summary really count a failure (7 passed, 1 failed). And you engraved the asymmetry of the lifecycle: the header only knows the configuration (start), the summary knows the results (end) —each hook with the information of its phase—.

Before moving on you should be able to: complete a pytest_report_header that reads options from the config and returns several lines; write a pytest_terminal_summary that uses write_sep, write_line and tr.stats.get(...) to report results; explain why the count goes in the summary and not in the header; and protect yourself from the KeyError of stats with .get(..., []).

With this you close the kit of four framework hooks —option, collection, header, summary—. What comes next looks at the conftest from the outside. In lesson 6 you are going to answer the question that was left open in lesson 2: if your conftest.py is already a plugin, what separates it from a packaged plugin, one installable with pip that works in any project? The answer is a single thing —how pytest discovers it— and you are going to see it with the real listing of the pytest11 entry points of installed plugins and by loading a loose plugin by name with -p reservo_plugin, without installing anything.

Resources

• pytest_terminal_summary — hooks reference — the official entry of the hook: what it receives (terminalreporter, exitstatus, config) and how to write the summary. The source of the exact signature.
• How to add information to the terminal report — pytest — the section of the plugins guide that shows pytest_report_header and the use of terminalreporter to write to the terminal.
• TerminalReporter reference — pytest — the detail of the reporter's methods (write_sep, write_line, write) and of the stats dictionary, to calibrate what to print and how to count the results.