Precision@k

Description

The previous lesson closed with a warning: recall@k can reach 1.0 without saying anything about how much noise comes mixed in with the correct result. This lesson builds the metric that does answer that question — precision@k: of the k chunks search actually returned, what fraction belongs to the expected doc_id?

You're going to implement precision_at_k, run it alongside recall_at_k on the same EVAL_SET, and see something recall@k alone doesn't show: while recall rises steadily with k, precision does not — it rises a bit at first and then starts falling, because every additional result has less and less chance of being from the correct document.

Connection to the module

With recall@k (Lesson 04) and precision@k (this lesson) complete, the module has both halves of the shape-of-the-problem evaluation. Lesson 06 uses both together — not just one — to diagnose exactly what's happening with lexical retrieval on this corpus.

Analogy: of what you brought back, how many were correct?

Back to the exam: if recall@k asks "is the correct answer among the k you marked?", precision@k asks the opposite: "of the k you marked, how many were correct?". It's the same pair of questions you'd ask a research assistant you told "bring me the 5 most relevant documents on this topic": recall tells you whether the document you genuinely needed is in the pile they brought back; precision tells you what fraction of that pile wasn't a waste of time.

An assistant can have perfect recall by bringing back 50 documents — the one you needed is surely in there — but with terrible precision, because you had to go through 49 irrelevant documents to find it. That's exactly the trade-off this lesson is about to measure with real numbers.

The formula

precision@k = |relevant ∩ retrieved in the top-k| / k

Unlike recall@k, precision@k's denominator isn't "the total relevant documents that exist" — it's k, the size of the results budget that was requested. Applied to this guide's EVAL_SET, where the ground truth is one doc_id per query:

def precision_at_k(results: list[Chunk], expected_doc_id: str, k: int) -> float:
    """Fraction of the k returned chunks that belong to expected_doc_id.
    If search() returns fewer than k chunks (because the corpus has no more
    candidates with a positive score), the missing slots count as
    non-relevant -- divide by k, not by len(results)."""
    if k == 0:
        return 0.0
    hits = sum(1 for c in results if c.doc_id == expected_doc_id)
    return hits / k

The decision to divide by k (the requested budget) and not by len(results) (what actually came back) is deliberate: if you asked for k=10 and the index only had 3 chunks sharing any term with the query, the 7 "empty slots" count against precision, they aren't ignored. With Reservo's corpus this almost never matters in practice — there are at least 15 chunks with a positive score for each of the EVAL_SET's six queries, so search always fills all k slots for the k values this module uses — but the rule is stated explicitly for when it does matter.

Worked example: precision@k for the six queries, k from 1 to 7

Using the same index and EVAL_SET from Lessons 03-04:

def evaluate_precision(eval_set, index, k):
    scores = []
    for query, expected_doc_id in eval_set:
        results = search(query, k, index)
        scores.append(precision_at_k(results, expected_doc_id, k))
    return scores


for k in [1, 2, 3, 4, 5, 6, 7]:
    scores = evaluate_precision(EVAL_SET, index, k)
    mean_precision = sum(scores) / len(scores)
    print(f"precision@{k} = {mean_precision:.4f}")

What to expect:

precision@1 = 0.1667
precision@2 = 0.3333
precision@3 = 0.3333
precision@4 = 0.3333
precision@5 = 0.2667
precision@6 = 0.2778
precision@7 = 0.2619

Compare it, side by side, with the previous lesson's recall@k:

k       recall@k    precision@k
1       0.1667      0.1667
2       0.5000      0.3333
3       0.6667      0.3333
4       0.8333      0.3333
5       0.8333      0.2667
6       1.0000      0.2778
7       1.0000      0.2619

Two very different patterns in the same table. Recall@k rises monotonically — it never goes down as k increases, as you confirmed in Lesson 04. Precision@k does not: it rises from k=1 to k=2 (0.1667 → 0.3333), stays flat through k=4, and then drops steadily (0.3333 → 0.2667 → 0.2778 → 0.2619). Each result added past fourth place, on average, has less chance of belonging to the correct doc_id than the previous ones — exactly what you'd expect from a ranking that already put what it considered most relevant first: the "tail" of the ranking is, on average, noisier than the head.

The special case of k=1: precision and recall are the same number

Look at the table's first row: recall@1 = 0.1667 and precision@1 = 0.1667 — exactly the same value. It's not a coincidence of this particular corpus; it's a mathematical property of k=1. With a single result returned, "is the correct one in the top-1?" (recall) and "is the one result returned correct?" (precision) are literally the same question — there's only one slot, and either the correct chunk fills it or it doesn't. The difference between the two metrics only shows up once k grows past 1.

r1 = sum(recall_at_k(search(q, 1, index), e) for q, e in EVAL_SET) / len(EVAL_SET)
p1 = sum(precision_at_k(search(q, 1, index), e, 1) for q, e in EVAL_SET) / len(EVAL_SET)
print(f"recall@1={r1:.4f}  precision@1={p1:.4f}  equal={r1 == p1}")

What to expect:

recall@1=0.1667  precision@1=0.1667  equal=True

Breakdown by query, k=3

The 0.3333 average in precision@3 hides quite a bit of variation between queries — it's worth looking at them one by one:

for query, expected_doc_id in EVAL_SET:
    results = search(query, 3, index)
    p = precision_at_k(results, expected_doc_id, 3)
    hits = sum(1 for c in results if c.doc_id == expected_doc_id)
    print(f"  precision@3={p:.4f}  ({hits}/3 correct)  {query!r}")

What to expect:

  precision@3=0.6667  (2/3 correct)  'What is the cancellation policy for Boardroom bookings?'
  precision@3=0.6667  (2/3 correct)  'How much discount does the pro tier get?'
  precision@3=0.3333  (1/3 correct)  'What equipment is in the Focus room?'
  precision@3=0.0000  (0/3 correct)  "Can I get a refund if I didn't show up?"
  precision@3=0.0000  (0/3 correct)  'What payment methods does Reservo accept?'
  precision@3=0.3333  (1/3 correct)  'Is there wifi in the Lounge?'

Here's the nuance the recall average (0.6667 = 4/6) doesn't show: yes, four of six queries "have recall" at k=3 — the correct doc_id shows up somewhere — but only two of those four have more than one correct chunk among the three returned (the discount one, with 2/3 correct among its three results, comes from two different membership-tiers-faq chunks; same for Boardroom). The other two that "have recall" (Focus, wifi/Lounge) achieve it with just one correct chunk out of three — the rest of the top-3, in both cases, is noise from cancellation-policy.

Common mistakes

1. Dividing by len(results) instead of by k. As explained in the formula, this guide divides by the requested k, not by how many results actually came back — it matters when search returns fewer than k (small corpus, query with few shared terms). With Reservo's EVAL_SET the number almost never changes, but the convention needs to be fixed before "almost never" turns into "sometimes yes, and nobody knows why the number doesn't add up".

2. Expecting precision@k to rise with k, the same as recall@k. This is the most common mistake when seeing this table for the first time — the "bigger is better" intuition doesn't apply to precision. As you saw, precision@k tends to fall (or stay flat) as k rises, because the additional results are rarely better than the first ones in an already-ordered ranking.

3. Reporting only the precision average, without the per-query breakdown. The 0.3333 average in precision@3 mixes queries with 2/3 correct (good) and queries with 0/3 correct (no hits at all) — two very different situations a single number doesn't distinguish. This lesson's breakdown section exists precisely so that detail isn't lost.

4. Using precision@k as the only metric to decide whether a system "works". A system with perfect precision@1 (1.0) that only answers one query out of a hundred correctly isn't a good system — you need both metrics together, and that's exactly the next lesson's topic.

Exercises

Exercise 1: Confirm precision@1 == recall@1 for each individual query (Easy)

Without averaging, compute recall@1 and precision@1 for each of the six queries separately, and confirm they're identical query by query, not just in the average.

for query, expected_doc_id in EVAL_SET:
    results = search(query, 1, index)
    r = recall_at_k(results, expected_doc_id)
    p = precision_at_k(results, expected_doc_id, 1)
    print(f"  recall={r:.1f}  precision={p:.1f}  equal={r == p}  {query!r}")

  recall=1.0  precision=1.0  equal=True  'What is the cancellation policy for Boardroom bookings?'
  recall=0.0  precision=0.0  equal=True  'How much discount does the pro tier get?'
  recall=0.0  precision=0.0  equal=True  'What equipment is in the Focus room?'
  recall=0.0  precision=0.0  equal=True  "Can I get a refund if I didn't show up?"
  recall=0.0  precision=0.0  equal=True  'What payment methods does Reservo accept?'
  recall=0.0  precision=0.0  equal=True  'Is there wifi in the Lounge?'

Exercise 2: Find the k with the best precision (Medium)

Over the range k=1 to k=10, find the value of k that maximizes precision@k for the full EVAL_SET. Report it alongside the recall@k corresponding to that same k.

best_k, best_p = None, -1.0
for k in range(1, 11):
    p = sum(evaluate_precision(EVAL_SET, index, k)) / len(EVAL_SET)
    r = sum(recall_at_k(search(q, k, index), e) for q, e in EVAL_SET) / len(EVAL_SET)
    print(f"  k={k:2d}  precision={p:.4f}  recall={r:.4f}")
    if p > best_p:
        best_k, best_p = k, p
print(f"\nbest precision: k={best_k}, precision={best_p:.4f}")

  k= 1  precision=0.1667  recall=0.1667
  k= 2  precision=0.3333  recall=0.5000
  k= 3  precision=0.3333  recall=0.6667
  k= 4  precision=0.3333  recall=0.8333
  k= 5  precision=0.2667  recall=0.8333
  k= 6  precision=0.2778  recall=1.0000
  k= 7  precision=0.2619  recall=1.0000
  k= 8  precision=0.2292  recall=1.0000
  k= 9  precision=0.2037  recall=1.0000
  k=10  precision=0.2000  recall=1.0000

best precision: k=2, precision=0.3333

Exercise 3: Precision only over the queries that DO have recall (Hard)

For k=3, compute the average precision considering only the four queries with recall@3 = 1.0 (excluding the two that don't find the correct document at all). Compare that number against the full EVAL_SET's precision@3 (0.3333) and explain the difference.

con_recall = [(q, e) for q, e in EVAL_SET
              if recall_at_k(search(q, 3, index), e) == 1.0]
print(f"queries with recall@3=1.0: {len(con_recall)}")

precision_subset = sum(precision_at_k(search(q, 3, index), e, 3)
                        for q, e in con_recall) / len(con_recall)
print(f"precision@3 only over those: {precision_subset:.4f}")
print(f"precision@3 over the full EVAL_SET: "
      f"{sum(evaluate_precision(EVAL_SET, index, 3)) / len(EVAL_SET):.4f}")

queries with recall@3=1.0: 4
precision@3 only over those: 0.5000
precision@3 over the full EVAL_SET: 0.3333

Summary and next step

• Precision@k answers: of the k chunks search returned, what fraction belongs to the expected doc_id? It's divided by k (the requested budget), not by how many results actually came back.
• Run on the six anchor queries: precision@k rises from k=1 to k=2-4 (maximum 0.3333) and then drops steadily through k=7 (0.2619) — a pattern opposite to recall@k, which only rises.
• At k=1, recall and precision are mathematically the same number — confirmed query by query in Exercise 1, not just in the average.
• The per-query breakdown showed that even among the queries that "have recall", the amount of mixed-in noise varies a lot (2/3 correct versus 1/3 correct) — the precision average without a breakdown hides that difference.

Next lesson: 06 — When lexical retrieval fails. With recall@k and precision@k now measured, this lesson dissects, word by word and score by score, why cancellation-policy-002/-003 win questions they shouldn't — the heart of the module.

Additional resources

1. Manning, Raghavan & Schütze — Introduction to Information Retrieval, ch. 8.3: "Evaluation of ranked retrieval results" — The formal definition of precision in a ranking context, alongside recall, in the same reference as the previous lesson.
2. production-rag-and-document-ingestion-guide — Module 6, Lesson 04 (Recall@k): the sibling metric this lesson complements, using the same index and EVAL_SET.
3. Wikipedia — Precision and recall — A visual overview of the trade-off between the two metrics, with the classic Venn diagram that summarizes this lesson's formula.
4. Python 3.14 — What's New — The version this module's entire codebase runs on.