Measuring what gets retrieved

Description

Every search run in this guide returned text chunks, and until now the question was always "are these the correct chunks?" — Module 6 answers it with recall and precision, this module's Lessons 02-05 operate it with citations, thresholds, and currency filters. This lesson asks a different question, one that doesn't depend on whether the content is correct: how much does what gets retrieved weigh? Every chunk search_docs returns, regardless of relevance, is going to take up space in the final prompt the model receives — characters that turn into tokens, tokens that cost money and count against the context window's limit. A system that never measures this has no way to answer basic operational questions: how much context does each search_docs call add, on average? Does it change much between k=3 and k=5? Does Lessons 04 and 05's filtering reduce that bill, or is it irrelevant to the size? This lesson builds the measurement.

Connection to the module

This lesson picks up Lessons 04 and 05's already-filtered results — not search_docs's raw ones — and adds a new dimension to them: size. It's deliberately the last piece before talking about cost (Lesson 07): measuring how much gets retrieved is the step before measuring how much retrieving it costs.

Analogy: how many books fit on the desk

Back to the desk: finding the correct books and discarding the ones that don't apply (Lessons 03-05) solves which books to hand over. A different, practical question remains: how many books, physically, fit on the desk of the person who's going to read them? Bringing ten relevant books at once isn't useful if there's only room for three — you simply can't hold everything at once, and at some point you have to decide how many volumes, and how thick, get carried. This lesson doesn't decide the order they get arranged on the desk, or which ones get summarized to take less space — that's context-engineering-guide's job — it measures, with a tape measure, how much what's already been decided to carry weighs.

retrieval_footprint: the minimal piece

from search_docs_tool import search_docs

CHARS_PER_TOKEN_APPROX = 4  # general rule-of-thumb heuristic for English, NOT a
                             # real tokenizer -- see the note below


def retrieval_footprint(hits):
    """Measures a batch of search_docs results' size: how many chunks,
    how many total characters of text, and a token ESTIMATE
    (not an exact count -- see this lesson's note)."""
    total_chars = sum(len(h["text"]) for h in hits)
    approx_tokens = round(total_chars / CHARS_PER_TOKEN_APPROX)
    return {"chunks": len(hits), "chars": total_chars, "approx_tokens": approx_tokens}

Before using it, an honest clarification worth reading carefully: CHARS_PER_TOKEN_APPROX = 4 is a general rule-of-thumb heuristic for English text — the same kind of approximation that circulates in any documentation about budgeting context — not the result of a real tokenizer. This environment has no network access to install Claude's official tokenizer, so this lesson doesn't pretend to have an exact count: approx_tokens is, literally, a rough estimate, labeled as such in every result. A production system with access to the official token-counting library would get an exact number in its place — the swap is direct (same signature, a different approx_tokens implementation), and it doesn't change anything else in this lesson.

Worked example: the six anchor queries' footprint, k=3 and k=5

anchor_queries = [
    "What is the cancellation policy for Boardroom bookings?",
    "How much discount does the pro tier get?",
    "What equipment is in the Focus room?",
    "Can I get a refund if I didn't show up?",
    "What payment methods does Reservo accept?",
    "Is there wifi in the Lounge?",
]

for k in (3, 5):
    print(f"--- k={k} ---")
    for q in anchor_queries:
        hits = search_docs(q, k=k)
        footprint = retrieval_footprint(hits)
        print(f"  {footprint}  {q}")

What to expect:

--- k=3 ---
  {'chunks': 3, 'chars': 560, 'approx_tokens': 140}  What is the cancellation policy for Boardroom bookings?
  {'chunks': 3, 'chars': 593, 'approx_tokens': 148}  How much discount does the pro tier get?
  {'chunks': 3, 'chars': 533, 'approx_tokens': 133}  What equipment is in the Focus room?
  {'chunks': 3, 'chars': 556, 'approx_tokens': 139}  Can I get a refund if I didn't show up?
  {'chunks': 3, 'chars': 510, 'approx_tokens': 128}  What payment methods does Reservo accept?
  {'chunks': 3, 'chars': 534, 'approx_tokens': 134}  Is there wifi in the Lounge?
--- k=5 ---
  {'chunks': 5, 'chars': 986, 'approx_tokens': 246}  What is the cancellation policy for Boardroom bookings?
  {'chunks': 5, 'chars': 857, 'approx_tokens': 214}  How much discount does the pro tier get?
  {'chunks': 5, 'chars': 890, 'approx_tokens': 222}  What equipment is in the Focus room?
  {'chunks': 5, 'chars': 869, 'approx_tokens': 217}  Can I get a refund if I didn't show up?
  {'chunks': 5, 'chars': 821, 'approx_tokens': 205}  What payment methods does Reservo accept?
  {'chunks': 5, 'chars': 847, 'approx_tokens': 212}  Is there wifi in the Lounge?

Two direct observations from these numbers. First, with k=3 each search adds between 128 and 148 approximate tokens to the prompt — a narrow range, because MAX_CHARS=280 (Module 3) caps each chunk's individual size, so the total size at a fixed k varies little between questions. Second, going from k=3 to k=5 almost doubles the footprint (from ~140 to ~220 average tokens) without that meaning double the useful information — you already know, from Module 2 (Lesson 07), that k doesn't reorder results, it only adds more low-relevance tail to the end of an already-sorted list. Every extra unit of k has a guaranteed size cost and a diminishing relevance benefit — the same trade-off, now with concrete numbers.

The footprint after filtering: Lessons 04-05's side benefit

It's worth confirming something Lessons 04 and 05 didn't measure explicitly: filtering by threshold or by currency doesn't just improve the quality of what gets cited — it also reduces the footprint, as a direct side effect of having fewer chunks.

THRESHOLD = 4.0


def apply_threshold(hits, min_score=THRESHOLD):
    return [h for h in hits if h["score"] >= min_score]


for q in anchor_queries:
    hits = search_docs(q, k=3)
    before = retrieval_footprint(hits)
    after = retrieval_footprint(apply_threshold(hits))
    print(f"{q}")
    print(f"  before:  {before}")
    print(f"  after:   {after}")

What to expect:

What is the cancellation policy for Boardroom bookings?
  before:  {'chunks': 3, 'chars': 560, 'approx_tokens': 140}
  after:   {'chunks': 3, 'chars': 560, 'approx_tokens': 140}
How much discount does the pro tier get?
  before:  {'chunks': 3, 'chars': 593, 'approx_tokens': 148}
  after:   {'chunks': 3, 'chars': 593, 'approx_tokens': 148}
What equipment is in the Focus room?
  before:  {'chunks': 3, 'chars': 533, 'approx_tokens': 133}
  after:   {'chunks': 3, 'chars': 533, 'approx_tokens': 133}
Can I get a refund if I didn't show up?
  before:  {'chunks': 3, 'chars': 556, 'approx_tokens': 139}
  after:   {'chunks': 3, 'chars': 556, 'approx_tokens': 139}
What payment methods does Reservo accept?
  before:  {'chunks': 3, 'chars': 510, 'approx_tokens': 128}
  after:   {'chunks': 2, 'chars': 336, 'approx_tokens': 84}
Is there wifi in the Lounge?
  before:  {'chunks': 3, 'chars': 534, 'approx_tokens': 134}
  after:   {'chunks': 2, 'chars': 373, 'approx_tokens': 93}

Only two of the six queries lose a chunk with THRESHOLD=4.0 (you already saw this in Lesson 04), and in those two the footprint drops proportionally — from 128 to 84 approximate tokens, from 134 to 93. The point isn't that the savings are dramatic on this 57-chunk corpus — they aren't — but that the effect is real and free: the same filter already being applied for quality reduces, with no extra work at all, the search's context bill. On a production corpus with thousands of chunks and queries that bring in more low-relevance noise, that same side effect becomes far more significant.

What this lesson does NOT decide

It's worth being explicit about the boundary, because it's easy to confuse "measuring the size" with "deciding what to do with that size". retrieval_footprint measures how many chunks, characters, and approximate tokens come out of search_docs (already filtered by Lessons 04-05) — it doesn't decide what order they enter the final prompt in, whether it's worth summarizing a long chunk instead of including it whole, or how the context window's space gets split between these chunks and the rest of the conversation (the turn history, the system prompt, other tool_results). Those three decisions — order, compression, shared budget — are exactly context-engineering-guide's content. This lesson hands over the number that guide needs as input; it makes none of its decisions.

Common mistakes

1. Confusing approx_tokens with an exact count. The 4-characters-per-token heuristic is a rule-of-thumb approximation, not the output of a real tokenizer — it can drift noticeably on text with lots of numbers, symbols, or terms outside common English vocabulary (like this corpus's hyphenated doc_id values, cancellation-policy, which a real tokenizer might split differently than this heuristic estimates). Useful for an order-of-magnitude estimate; not for a production budget without checking against the real tokenizer.

2. Measuring the footprint on search_docs's raw results, without first applying Lessons 04-05's filters. The footprint that actually reaches the final prompt is that of the chunks that survived drop_stale_sources and apply_threshold — measuring before filtering systematically overestimates how much context is genuinely being spent.

3. Assuming more k always means a better answer, ignoring the size cost. The "Worked example" section showed that going from k=3 to k=5 almost doubles the footprint without necessarily bringing more relevance — Module 2 already established that a high k only exposes more low-relevance tail. Every unit of k is a cost decision, not a free improvement.

4. Thinking this lesson decides the final context budget. As the previous section clarified, retrieval_footprint measures, it doesn't decide. Confusing this leads to trying to solve problems here — chunk order in the prompt, when to compress — that belong to context-engineering-guide.

Exercises

Exercise 1: Measure a single query's footprint (Easy)

Run search_docs("Is there wifi in the Lounge?", k=1) and compute its retrieval_footprint. How many approximate tokens does a single chunk add to the prompt?

hits = search_docs("Is there wifi in the Lounge?", k=1)
print(retrieval_footprint(hits))

{'chunks': 1, 'chars': 174, 'approx_tokens': 44}

Exercise 2: Compare a batch's total footprint before and after the full filter (Medium)

Add up the six anchor queries' retrieval_footprint with k=3, before and after applying apply_threshold. Report how many approximate tokens get saved in total over the full batch.

total_before_chars = 0
total_after_chars = 0

for q in anchor_queries:
    hits = search_docs(q, k=3)
    total_before_chars += retrieval_footprint(hits)["chars"]
    total_after_chars += retrieval_footprint(apply_threshold(hits))["chars"]

tokens_before = round(total_before_chars / CHARS_PER_TOKEN_APPROX)
tokens_after = round(total_after_chars / CHARS_PER_TOKEN_APPROX)

print(f"total before:  {total_before_chars} chars (~{tokens_before} tokens)")
print(f"total after:   {total_after_chars} chars (~{tokens_after} tokens)")
print(f"savings: ~{tokens_before - tokens_after} tokens over the six-query batch")

total before:  3286 chars (~822 tokens)
total after:   2951 chars (~738 tokens)
savings: ~84 tokens over the six-query batch

Exercise 3: Design a maximum-budget check (Hard)

Write a function within_budget(hits, max_tokens) that returns True if hits's retrieval_footprint doesn't exceed max_tokens, False otherwise. Then write trim_to_budget(hits, max_tokens) that, if within_budget returns False, discards the lowest-scoring chunk one at a time until the footprint fits the budget (or until there are no chunks left). Test it with the Boardroom query at k=5 and a budget of max_tokens=150.

def within_budget(hits, max_tokens):
    return retrieval_footprint(hits)["approx_tokens"] <= max_tokens


def trim_to_budget(hits, max_tokens):
    """Discards, one at a time, the lowest-scoring chunk until the
    approximate token budget is met -- or until there are no chunks left."""
    remaining = sorted(hits, key=lambda h: h["score"], reverse=True)
    while remaining and not within_budget(remaining, max_tokens):
        remaining.pop()  # the last one in the score-sorted list = the lowest
    return remaining


hits = search_docs("What is the cancellation policy for Boardroom bookings?", k=5)
print("original footprint:", retrieval_footprint(hits))

trimmed = trim_to_budget(hits, max_tokens=150)
print("trimmed footprint:", retrieval_footprint(trimmed))
print("surviving chunks:", [h["chunk_id"] for h in trimmed])

original footprint: {'chunks': 5, 'chars': 986, 'approx_tokens': 246}
trimmed footprint: {'chunks': 3, 'chars': 560, 'approx_tokens': 140}
surviving chunks: ['cancellation-policy-003', 'cancellation-policy-002', 'membership-tiers-faq-000']

Summary and next step

• retrieval_footprint(hits) measures chunks, characters, and a token estimate (a 4-characters/token heuristic, honestly labeled as an approximation, not an exact count) for a batch of search_docs results.
• With k=3, each of the six anchor queries' searches adds between 128 and 148 approximate tokens; going to k=5 almost doubles that figure without necessarily bringing more relevance — the same k limit Module 2 already measured from a different angle.
• Filtering by threshold (Lesson 04) reduces the footprint as a direct side effect: fewer irrelevant chunks cited is also less context spent, a small savings on this 57-chunk corpus but cumulative in a real production system.
• This lesson measures the input; it doesn't decide the order, compression, or shared budget within the final prompt — that's explicitly and entirely context-engineering-guide's job.

Next lesson: 07 — The cost of the pipeline. From how much each search weighs to how much it costs to run it — real time to index and search with BM25, and the honest contrast with a paid embedding.

Additional resources

1. context-engineering-guide — where order, compression, and the context window's shared budget get decided, using the footprint this lesson measures as input.
2. production-rag-and-document-ingestion-guide, Module 2, Lesson 07 (search/query/k, run) — the original observation that a high k only exposes more low-relevance tail, the basis for this lesson's cost argument.
3. production-rag-and-document-ingestion-guide, Module 3, Lesson 06 (Scoping the tool: k and truncation) — MAX_CHARS=280, the already-established limit that keeps the per-chunk footprint range narrow.
4. Python — len() on strings — the foundation of retrieval_footprint's character measurement, and why it measures text length, not real tokens.