Module 7: Analyzing Results And Ci

3. Exporting results (JSON/CSV) and why

Next →

Overview

The summary that appears on the screen at the end of a run is convenient for a quick glance, but it's ephemeral: you close the terminal and it's gone. To do something serious with a result —analyze it with another tool, attach it to a report, and above all compare it with future runs— you have to export it to a file. In this lesson you learn to export: in k6 with k6 run --out json=results.json (and --out csv), presented as labeled content; and on the executable side, with the Python generator that writes its real metrics to an actual results.json, plus a historical CSV that accumulates runs to see the trend between releases. The central idea is that exporting isn't a luxury: it's what turns a run from an event that evaporates into an artifact that's saved, studied, and compared.

Connection to the module: this lesson produces the file lesson 2 learned to read and that lesson 4 is going to compare. Without exporting there's no baseline, and without a baseline there's no regression detection —so this lesson is the foundation of the analyzing half—. It reuses the module-3 metrics (p50/p95/p99, RPS, error), now serialized to disk. What comes next (lesson 4) takes two of these files and compares them to catch a regression.

The photo of the scale, not the number you shouted

Imagine that every morning you weigh yourself and shout the number out loud: "78!". The information existed for a second and it's gone. You can't know whether you went up relative to last month, because there's no record; you only have the echo of today's number. Now imagine that every morning you write down the weight in a notebook with the date. Suddenly you have something much more valuable than a number: you have a series. You can see the trend, compare today with a month ago, detect that you've been going up little by little for three weeks.

The on-screen summary is the shouted number. The export to a file is the notebook. A results.json saved with the run's date is an entry in that notebook, and a folder full of them —or a CSV that accumulates them— is the series that lets you see the performance trend over time. The question "did the p95 get worse with the last release?" can only be answered if you wrote down the previous one. Exporting is writing down.

Exporting in k6 (content)

k6 exports a run's detail with the --out flag. It's labeled content —k6 isn't installed here—, faithful to the official documentation:

# CONTENT (not run here): export k6's results. See grafana.com/docs/k6.

# To JSON: one JSON line per metric point (each request, each check...).
k6 run --out json=results.json load_test.js

# To compressed JSON (files grow fast; gzip helps):
k6 run --out json=results.gz load_test.js

# To CSV: one row per metric point, handy for spreadsheets.
k6 run --out csv=results.csv load_test.js

An important nuance about k6's format. k6's --out json doesn't write a single object with the summary: it writes one metric point per line (JSON Lines format), one entry for each latency, each check, each interval. A fragment looks like this (content):

// CONTENT (not run here): shape of k6's --out json (one line per point).
{"type":"Point","metric":"http_req_duration","data":{"time":"2026-07-25T03:00:01Z","value":4.7,"tags":{"status":"200","name":"quote"}}}
{"type":"Point","metric":"http_req_duration","data":{"time":"2026-07-25T03:00:01Z","value":6.6,"tags":{"status":"200","name":"quote"}}}
{"type":"Point","metric":"http_reqs","data":{"time":"2026-07-25T03:00:01Z","value":1,"tags":{"status":"200"}}}

That raw, detailed format is powerful —you can load it into a time-series database, into Grafana, into a pandas notebook— but it's not the summary: it's the raw material the summary comes from. To compare runs simply, you almost always want a summarized file: a single photo with p50/p95/p99, RPS, and error per run. That's exactly what the Python generator produces, and it's what we'll use to detect regressions.

The executable side: exporting a summary to JSON

Here's the Python generator that does run, with this module's central addition: besides measuring, it writes its metrics to a JSON file. It's the load_generator you already know (from the previous modules) with two changes: it computes the percentiles with statistics.quantiles and, at the end, serializes a summary dictionary to disk with json.dump.

"""Load generator that EXPORTS its metrics to JSON.

Hits a Reservo endpoint with N concurrent requests, measures the real latency
of each, computes p50/p95/p99, RPS, and error rate, and writes it all to a
JSON file. That JSON is the artifact that's later analyzed elsewhere and compared
between runs (the executable equivalent of `k6 run --out json=results.json`).

Usage: python3.14 load_and_export.py BASE_URL PATH TOTAL CONCURRENCY OUT.json LABEL
"""
import json
import statistics
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE_URL = sys.argv[1]
PATH = sys.argv[2]
TOTAL = int(sys.argv[3])
CONCURRENCY = int(sys.argv[4])
OUT = sys.argv[5]
LABEL = sys.argv[6] if len(sys.argv) > 6 else "run"


def one_request():
    """One POST request. Returns (latency_ms, ok) where ok=correct response."""
    payload = json.dumps({"room": "Focus", "tier": "basic", "hours": 3}).encode()
    req = urllib.request.Request(
        f"{BASE_URL}{PATH}", data=payload,
        headers={"Content-Type": "application/json"}, method="POST",
    )
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req) as resp:
            body = json.loads(resp.read())
        ok = resp.status == 200 and body.get("price_cents") == 7500
    except (urllib.error.URLError, OSError):
        ok = False
    latency_ms = (time.perf_counter() - start) * 1000
    return latency_ms, ok


def pct(sorted_ms, p):
    """Percentile p (0..100) with statistics.quantiles (inclusive method)."""
    if len(sorted_ms) < 2:
        return sorted_ms[0] if sorted_ms else 0.0
    cuts = statistics.quantiles(sorted_ms, n=100, method="inclusive")
    return cuts[p - 1]


def main():
    latencies = []
    ok_count = 0
    wall_start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
        futures = [pool.submit(one_request) for _ in range(TOTAL)]
        for fut in futures:
            latency_ms, ok = fut.result()
            latencies.append(latency_ms)
            ok_count += 1 if ok else 0
    wall_seconds = time.perf_counter() - wall_start

    latencies.sort()
    error_rate = (TOTAL - ok_count) / TOTAL
    results = {
        "label": LABEL,
        "endpoint": PATH,
        "requests": TOTAL,
        "concurrency": CONCURRENCY,
        "duration_s": round(wall_seconds, 3),
        "rps": round(TOTAL / wall_seconds, 1),
        "error_rate": round(error_rate, 4),
        "checks_rate": round(ok_count / TOTAL, 4),
        "latency_ms": {
            "min": round(min(latencies), 2),
            "p50": round(pct(latencies, 50), 2),
            "p95": round(pct(latencies, 95), 2),
            "p99": round(pct(latencies, 99), 2),
            "max": round(max(latencies), 2),
            "avg": round(statistics.mean(latencies), 2),
        },
    }
    with open(OUT, "w") as f:
        json.dump(results, f, indent=2)
    m = results["latency_ms"]
    print(f"[{LABEL}] {PATH}  {TOTAL} req  concurrency {CONCURRENCY}")
    print(f"  rps={results['rps']}  error_rate={results['error_rate']:.2%}  "
          f"checks={results['checks_rate']:.2%}")
    print(f"  p50={m['p50']}ms  p95={m['p95']}ms  p99={m['p99']}ms  max={m['max']}ms")
    print(f"  -> wrote {OUT}")


if __name__ == "__main__":
    main()

Two decisions worth understanding:

  • statistics.quantiles(data, n=100, method="inclusive") divides the sorted series into 100 groups and returns the 99 cut points; cut number p is the pth percentile. So cuts[94] (with p=95) is the p95. It's the rigorous way module 3 introduced, now at the service of a saved file.
  • The JSON is a summary, not the raw detail. Unlike k6's --out json (one line per point), here we write a single compact photo per run: label, endpoint, rps, error_rate, and the latency_ms block with the percentiles. It's exactly what's needed to compare two runs without loading millions of points.

We run it against the two endpoints —/quote (baseline) and /quote_slow (degraded)—. Real output:

What to expect — each run prints its summary and confirms the file written:

$ python3.14 load_and_export.py http://127.0.0.1:PORT /quote 600 30 results_baseline.json baseline
[baseline] /quote  600 req  concurrency 30
  rps=5449.6  error_rate=0.00%  checks=100.00%
  p50=4.7ms  p95=6.66ms  p99=19.91ms  max=22.63ms
  -> wrote results_baseline.json

$ python3.14 load_and_export.py http://127.0.0.1:PORT /quote_slow 600 30 results_actual.json actual
[actual] /quote_slow  600 req  concurrency 30
  rps=538.1  error_rate=0.00%  checks=100.00%
  p50=54.74ms  p95=61.27ms  p99=73.62ms  max=75.82ms
  -> wrote results_actual.json

And this is what ended up on disk —the exported artifact, which you can open, version, attach, or compare—. Real output of the files' content:

What to expect — a summary JSON per run, with the percentiles and the throughput:

$ cat results_baseline.json
{
  "label": "baseline",
  "endpoint": "/quote",
  "requests": 600,
  "concurrency": 30,
  "duration_s": 0.11,
  "rps": 5449.6,
  "error_rate": 0.0,
  "checks_rate": 1.0,
  "latency_ms": {
    "min": 2.7,
    "p50": 4.7,
    "p95": 6.66,
    "p99": 19.91,
    "max": 22.63,
    "avg": 5.29
  }
}

That file is the notebook with today's entry. The run no longer evaporated: it was saved, ready for lesson 4 to compare it with the next.

Accumulating runs in a CSV (comparing the trend)

A single file is a photo; the value of exporting appears when you accumulate runs and see the trend. A CSV is ideal for that: each row is a run, the file grows over time, and it opens in any spreadsheet. This small appender reads a results.json and adds a row to the CSV:

"""Appends an exported run to a historical CSV.

Exporting isn't just for one run: accumulating runs in a CSV lets you compare
the trend between releases. Each row is a run; the CSV grows over time.

Usage: python3.14 append_run_csv.py results.json runs.csv
"""
import csv
import json
import os
import sys

with open(sys.argv[1]) as f:
    r = json.load(f)
csv_path = sys.argv[2]
row = {
    "label": r["label"], "endpoint": r["endpoint"], "rps": r["rps"],
    "p50": r["latency_ms"]["p50"], "p95": r["latency_ms"]["p95"],
    "p99": r["latency_ms"]["p99"], "error_rate": r["error_rate"],
}
write_header = not os.path.exists(csv_path)
with open(csv_path, "a", newline="") as f:
    w = csv.DictWriter(f, fieldnames=list(row.keys()))
    if write_header:
        w.writeheader()
    w.writerow(row)
print(f"appended run '{r['label']}' to {csv_path}")

We run it with the two exported runs. Real output:

What to expect — two rows in runs.csv, one per run, with the fast baseline and the slow degraded one side by side:

$ python3.14 append_run_csv.py results_baseline.json runs.csv
appended run 'baseline' to runs.csv
$ python3.14 append_run_csv.py results_actual.json runs.csv
appended run 'actual' to runs.csv
$ cat runs.csv
label,endpoint,rps,p50,p95,p99,error_rate
baseline,/quote,5449.6,4.7,6.66,19.91,0.0
actual,/quote_slow,538.1,54.74,61.27,73.62,0.0

There's the notebook with two entries, and the trend jumps out: the p95 went from 6.66 to 61.27 ms and the RPS collapsed from 5449 to 538. With two rows you already see it; with twenty, a slow regression accumulating release after release would also show. That's the whole reason for exporting: without the notebook, each run is a shouted number no one can compare.

Common mistakes

Confusing k6's --out json with a summary. What happens: someone runs k6 run --out json=out.json, opens the file expecting to see p95: ..., and finds millions of lines of raw points. Why it happens: k6's --out json writes the detail (one point per metric), not the summary. How to detect it: the file has a {"type":"Point",...} line per request, not a photo. How to fix it: if you want the summary, use k6's handleSummary to write a compact JSON (k6 content), or —as here— export a summary yourself. The raw --out json is for analysis tools, not for reading by eye.

Not putting a date or label on the exported run. What happens: several results.json are saved without distinguishing which is from when, and when comparing you don't know which is the baseline. Why it happens: exporting without metadata. How to detect it: a folder of identically named files, impossible to sort. How to fix it: include a label (or timestamp / commit hash) in the JSON and in the file name, to know which run is which. Here the label ("baseline"/"actual") plays that role.

Exporting and never comparing. What happens: hundreds of results.json accumulate that no one looks at again. Why it happens: exporting becomes a purposeless ritual. How to detect it: if you have exported files but never compare two, exporting isn't giving you anything. How to fix it: the value of exporting is comparing —analyzing the trend (this CSV) or detecting a regression (lesson 4)—. Export with the intention of comparing.

Exercises

Exercise 1 — Summary JSON vs raw JSON. k6's --out json and the Python generator's results.json are both "JSON," but very different. (a) What does each contain? (b) Which would you use to quickly compare two runs and why? (c) What would the raw one be good for?

See solution
  • (a) k6's --out json contains the detail: one line per metric point (each latency, each check), in JSON Lines format. The generator's results.json contains the summary: a single photo with p50/p95/p99, RPS, and error of the run.
  • (b) The summary. To compare two runs it's enough to read each one's p95; the summary has it directly, whereas from the raw one you'd have to recompute the percentiles from millions of points.
  • (c) The raw one is good for deep analysis: loading it into Grafana or pandas, seeing the latency over time within the run (did it rise at the end?), filtering by tags (by endpoint, by status). It's the raw material; the summary is the conclusion.

Exercise 2 — Read the CSV. The runs.csv has these two rows: baseline,/quote,5449.6,4.7,6.66,19.91,0.0 and actual,/quote_slow,538.1,54.74,61.27,73.62,0.0. (a) How much did the p95 get worse? (b) And the RPS? (c) What story does the actual row tell?

See solution
  • (a) The p95 went from 6.66 ms to 61.27 ms: it multiplied by ~9.2 (got worse by +820%).
  • (b) The RPS fell from 5449.6 to 538.1: down to ~1/10. With each request taking ~45 ms more, the system dispatches far fewer per second.
  • (c) That the endpoint got slow (the fixed delay of /quote_slow): the logic is still correct (same price, 0% error), but the performance collapsed —it takes 9x longer and holds 1/10 of the throughput—. It's the portrait of a performance regression.

Exercise 3 — Design the notebook. You want to save the summary of each nightly run to be able to see the p95 trend over the last month. (a) What would you add to the exported JSON so the runs can be sorted in time? (b) Would you name the files the same or differently, and how?

See solution
  • (a) A timestamp (the run's date and time, in ISO format like 2026-07-25T03:00:00Z) and, if it runs in CI, the commit hash that was tested. That way each run is anchored to a moment and a code version, and they can be sorted chronologically to chart the trend.
  • (b) Differently, including the date in the name: results-2026-07-25.json, results-2026-07-26.json... A unique name per run avoids overwriting the previous one (losing the baseline) and lets you sort the folder by date. Saving them all in an accumulated CSV, as here, is the other half: the CSV is the series, the JSONs are the individual photos.

Summary and next step

In this lesson you learned to export a run so it stops being an ephemeral number and becomes an artifact. In k6 it's done with --out json / --out csv (content), remembering that --out json writes the raw detail (one point per line), not the summary. On the executable side, the Python generator writes a summary results.json —p50/p95/p99, RPS, error— with json.dump, and an appender accumulates runs in a runs.csv to see the trend. You saw the real files on disk: the fast baseline (p95 6.66 ms) and the slow degraded one (p95 61.27 ms), side by side in the CSV.

The idea to take from the lesson: exporting is writing down in the notebook. Without yesterday's entry you can't know whether you got worse today; exporting saves the baseline that makes comparison possible. Before moving on you should be able to: explain the difference between k6's raw JSON and a summary JSON; name why you export (analyze elsewhere, save, compare); and read a trend in a runs CSV. What comes next, in lesson 4, is the payoff of having exported: taking two of these files —a baseline and a current run— and comparing them to detect a performance regression, with a check that fails with an exit code.

Resources

← Previous2. Reading the summary and the trend metricNext →4. Detecting a performance regression