Recall@k

Description

The previous lesson left a loose end, counted by hand in Exercise 2: of the six anchor queries, four have the correct doc_id somewhere in the top-3, though only one has it in first place. This lesson gives that question a name and a formula — recall@k — and computes it for several values of k, on the six EVAL_SET queries, with the index built in the previous lesson.

You're going to implement recall_at_k in under five lines, run it for k from 1 to 7, and see a very clear pattern in the real numbers: recall goes up with k, but not evenly across queries — one of the six holds out until the very end.

Connection to the module

This lesson doesn't touch the index or the EVAL_SET — it works entirely on top of what Lesson 03 already built. It's the first of the module's two shape-of-the-problem metrics; Lesson 05 adds the second (precision@k) on the same material, and together they feed Lesson 06's diagnosis.

Analogy: did the correct answer show up among the first k you checked?

Back to the exam with the answer key: imagine that, instead of a standard multiple-choice exam, every question comes with an ordered list of possible answers, and the student can mark up to k of them per question — not just the first one. Recall@k asks, for each question on the exam: is the correct answer somewhere among the first k the student marked? It doesn't matter whether they marked it first or last among those k — all that matters is whether it's inside that window or not.

Applied to search: for a given query, search(query, k, index) gives you an ordered list of up to k chunks. Recall@k asks whether, somewhere in that list, at least one chunk with the expected doc_id shows up. It's an all-or-nothing question per query — 1.0 if it's there, 0.0 if it isn't — and the full set's "recall@k" is the average of those ones and zeros over the six queries.

The formula, and why it's binary in this case

In information retrieval, the general definition of recall is:

recall@k = |relevant ∩ retrieved in the top-k| / |total relevant|

In a corpus where "relevant" can mean multiple documents per query, the denominator (|total relevant|) can be greater than one. In this guide's EVAL_SET, the ground truth is a single doc_id per query — the anchor table never marks two documents as correct for the same question — so the denominator is always 1, and the formula simplifies to a binary question: did at least one chunk from the expected doc_id show up in the top-k?

def recall_at_k(results: list[Chunk], expected_doc_id: str) -> float:
    """1.0 if any returned chunk belongs to expected_doc_id, else 0.0.
    EVAL_SET's ground truth is a single doc_id per query, so
    recall@k here answers: 'did the correct document show up in the top-k?'"""
    return 1.0 if any(c.doc_id == expected_doc_id for c in results) else 0.0

results already comes truncated to k because that's exactly what search(query, k, index) returns — recall_at_k doesn't need to know which k was used, it just looks at the list it received. The "@k" lives in how search was called, not in recall_at_k's own formula.

Worked example: recall@k for the six queries, k from 1 to 7

Using Lesson 03's index and EVAL_SET:

def evaluate_recall(eval_set, index, k):
    scores = []
    for query, expected_doc_id in eval_set:
        results = search(query, k, index)
        scores.append(recall_at_k(results, expected_doc_id))
    return scores


for k in [1, 2, 3, 4, 5, 6, 7]:
    scores = evaluate_recall(EVAL_SET, index, k)
    mean_recall = sum(scores) / len(scores)
    hits = int(sum(scores))
    print(f"recall@{k} = {mean_recall:.4f}  ({hits}/{len(EVAL_SET)})")

What to expect:

recall@1 = 0.1667  (1/6)
recall@2 = 0.5000  (3/6)
recall@3 = 0.6667  (4/6)
recall@4 = 0.8333  (5/6)
recall@5 = 0.8333  (5/6)
recall@6 = 1.0000  (6/6)
recall@7 = 1.0000  (6/6)

Recall rises monotonically with k — it never goes down, because adding one more result can never make a doc_id that was already inside stop being inside — and it reaches 1.0000 only at k=6. Between k=4 and k=5 the number doesn't change (0.8333 both times): adding a fifth result didn't bring in any new query. It's worth looking, query by query, at exactly which k each one gets in at:

for query, expected_doc_id in EVAL_SET:
    for k in range(1, 8):
        results = search(query, k, index)
        if recall_at_k(results, expected_doc_id) == 1.0:
            print(f"  enters at k={k}  {query!r}")
            break
    else:
        print(f"  NEVER enters (k<=7)  {query!r}")

What to expect:

  enters at k=1  'What is the cancellation policy for Boardroom bookings?'
  enters at k=2  'How much discount does the pro tier get?'
  enters at k=2  'What equipment is in the Focus room?'
  enters at k=6  "Can I get a refund if I didn't show up?"
  enters at k=4  'What payment methods does Reservo accept?'
  enters at k=3  'Is there wifi in the Lounge?'

Five of the six queries get in at a reasonable k (between 1 and 4). The trap query — refund/no-show — is the only one that needs k=6 to show up at all: out of the corpus's 57 chunks, no-show-policy's first chunk only shows up in sixth place in the BM25 ranking for that query. Lesson 06 is about to dissect exactly why, word by word.

What recall@k does NOT tell you

A high recall number is good news, but it's incomplete news. Compare k=4 (recall=0.8333, 5/6) with k=7 (recall=1.0000, 6/6): to get from the first to the second, you had to raise the results budget from 4 to 7 — almost double — just to rescue one more query (the trap). Recall@k doesn't penalize at all how much noise comes mixed in with the correct result — a k=7 with perfect recall could be returning, alongside the correct chunk, six completely irrelevant ones, and recall@7 would still read 1.0000 without flinching. That's exactly the question precision@k, in the next lesson, does answer.

Common mistakes

1. Thinking recall@k can go down as k goes up. It's mathematically impossible with this definition: search(query, k+1, index) always includes everything search(query, k, index) already had, plus one more element (or the same, if there are no more candidates with a positive score). If recall@k drops as k rises in your code, there's a bug in how you're calling search, not in the data.

2. Computing recall over a single chunk's doc_id, instead of "any chunk from the document". recall_at_k uses any(...) deliberately — if search returns two different chunks from no-show-policy in the top-k, that's still recall=1.0, not 2.0. The metric is binary per query, not a count of how many chunks from the correct document showed up.

3. Confusing "recall reaches 1.0 at some k" with "the system works well". As you saw above, the refund/no-show trap needs k=6 out of 57 possible chunks just to show up once — that's a technically perfect recall at k=6, but it's a signal of a system that nearly fails completely on that specific query. The number alone, without looking at which k it takes to achieve it, can hide exactly the problem it's supposed to expose.

4. Using recall@k with a k different from what search_docs would actually use in production. If Module 3 capped k at a reasonable maximum (say, k=5) to avoid sending the model too much extra text, measuring recall@20 says nothing about what the real agent is going to see — it measures a scenario that never happens in production.

Exercises

Exercise 1: Compute recall@1 by hand (Easy)

Without running any code, using this lesson's "which k each query enters at" table, compute recall@1 by hand: how many of the six queries enter at exactly k=1? Confirm with evaluate_recall.

scores = evaluate_recall(EVAL_SET, index, k=1)
print(scores, sum(scores) / len(scores))

[1.0, 0.0, 0.0, 0.0, 0.0, 0.0] 0.16666666666666666

Exercise 2: Find the minimum k for perfect recall (Medium)

Without looking at this lesson's table, write code that finds, for the full EVAL_SET, the minimum value of k such that recall@k = 1.0 (all queries find their expected doc_id). Run it and report that k.

k = 1
while True:
    scores = evaluate_recall(EVAL_SET, index, k)
    if sum(scores) == len(EVAL_SET):
        break
    k += 1
print(f"recall@k = 1.0 is first reached at k={k}")

recall@k = 1.0 is first reached at k=6

Exercise 3: Compare recall@k with and without the trap query (Hard)

Build a reduced EVAL_SET, without the fourth query (the refund/no-show one), and compute recall@1 and recall@3 on that five-query subset. Compare against the same k values on the full six-query EVAL_SET, and explain how much the number changes when a single hard query is removed.

eval_set_sin_trampa = [pair for pair in EVAL_SET if pair[1] != "no-show-policy"]
print(f"queries in the subset: {len(eval_set_sin_trampa)}")

for k in [1, 3]:
    full = sum(evaluate_recall(EVAL_SET, index, k)) / len(EVAL_SET)
    reduced = sum(evaluate_recall(eval_set_sin_trampa, index, k)) / len(eval_set_sin_trampa)
    print(f"recall@{k}  full (6 queries)={full:.4f}   without trap (5 queries)={reduced:.4f}")

queries in the subset: 5
recall@1  full (6 queries)=0.1667   without trap (5 queries)=0.2000
recall@3  full (6 queries)=0.6667   without trap (5 queries)=0.8000

Summary and next step

• Recall@k answers: of the EVAL_SET's queries, in what fraction did the expected doc_id show up somewhere in search's top-k? It's binary per query (1.0 or 0.0) because this guide's ground truth is a single correct document per question.
• Run on the six anchor queries: recall@1 = 0.1667, recall@3 = 0.6667, recall@6 = 1.0000 — it rises monotonically with k, but the trap query (refund/no-show) only gets in at k=6, out of 57 possible chunks.
• Recall@k does not penalize noise mixed into the result — a perfect recall at k=7 says nothing about how many of those 7 chunks are actually from the correct document.
• A small EVAL_SET (six queries) is sensitive to every individual query — Exercise 3 confirmed it with numbers: removing a single query shifted recall by several percentage points.

Next lesson: 05 — Precision@k. The metric that does measure how much noise comes mixed in with the correct result — the half of the picture recall@k leaves unanswered.

Additional resources

1. Manning, Raghavan & Schütze — Introduction to Information Retrieval, ch. 8.3: "Evaluation of ranked retrieval results" — The formal definition of recall in a ranking context, which this lesson simplifies for a single relevant document per query.
2. production-rag-and-document-ingestion-guide — Module 6, Lesson 03 (The fixed evaluation set): the source of the index and EVAL_SET this lesson uses unchanged.
3. Python — the built-in any() and sum() functions — The two functions all of recall_at_k and evaluate_recall's logic is built on.
4. Python 3.14 — What's New — The version this module's entire codebase runs on.