Module 2: Indexing Chunks for Retrieval

Module 2: Indexing Chunks for Retrieval

Next →

Description

Module 1 left you with a pile of Chunk: fragments of Reservo's documents, already parsed, cleaned, and carrying their metadata (doc_id, title, section, position). That solves half the problem. The other half is this: when an agent needs to answer "what is Boardroom's cancellation policy?", it can't read all 57 chunks in the corpus one by one hoping to land on the right one. It needs an index — a structure that, given a handful of words, hands back the relevant chunks directly, without scanning everything.

This module builds that index end to end, with BM25: the same lexical retrieval algorithm that Elasticsearch and OpenSearch run in production, hand-rolled with collections.Counter and math from the standard library (plus numpy for the term-document matrix). You'll see, executed and with real output, how an inverted index gets built, how term frequency and IDF get computed, how those two pieces combine into the BM25 score, and how search(query, k) uses that score to return the right Chunk objects — confirmed against the same six anchor questions you'll use for the rest of the guide.

Hard rule of this guide (still in force in this module)

Indexing and retrieval engineering runs for real — every line of code in this module runs on Python 3.14 over the full Reservo corpus (13 documents, 57 chunks — the same canonical corpus that Module 1 ingested, with zero-padded 3-digit chunk_id, e.g. cancellation-policy-003), and every "What to expect" block is real output, not invented. The only thing that's still conceptual in this guide is a model's (claude-sonnet-5) decision about when to call a search tool — and that doesn't show up in this module yet: there's no model involved here, only your index and your queries.

🛑 Honest labeling (read this before continuing — it repeats in every lesson that touches the index)

BM25 is real LEXICAL retrieval. It counts exact word matches, weighted by how frequent they are within a chunk and how rare they are across the whole corpus. It's the same algorithm running underneath an Elasticsearch match or the "lexical" half of a production hybrid search — it's not a teaching toy, it's a real piece of the retrieval toolbox.

BM25 is NOT a semantic embedding. It doesn't know that "refund" and "reimbursement" mean the same thing. It doesn't understand paraphrase, it doesn't understand synonyms, it doesn't understand meaning — only term matching. If a query uses a word the corpus never uses, BM25 can fail completely, even when the question and the correct answer are talking about exactly the same thing. Lesson 06 of this module demonstrates that limit by running it, not by describing it. Whenever you want the version that does capture meaning, the answer lives in embeddings-deep-dive-guide (embedding theory) and vector-databases-fundamentals-guide (how a real vector index gets built and queried, under the hood) — this module names that boundary, it doesn't cross it.

Where we are in the guide

Production RAG and Document Ingestion Guide — Reservo's agent
├── Module 1: Parsing and chunking documents
│   → From dirty HTML/markdown/text to Chunk with metadata
├── Module 2: Indexing chunks for retrieval  ← YOU ARE HERE
│   → Inverted index, TF, IDF, hand-rolled BM25, search(query, k)
├── Module 3: search_docs as an agent tool
├── Module 4: Agentic retrieval in the loop
├── Module 5: Incremental and idempotent ingestion
├── Module 6: Evaluating retrieval quality
├── Module 7: Running RAG in production
└── Module 8: Project — a production-ready search_docs tool, for Reservo

This is Module 2 of 8. If you already took Module 1, you know where the Chunk objects you'll index here come from; if you're starting from this module, that's fine — this module rebuilds the same fixed 13-document Reservo corpus with the same Chunk shape, so you can follow it start to finish without having run a single line of Module 1. By the end, you'll have a complete, executed BM25 index able to answer search(query, k) -> list[Chunk] over the real Reservo corpus, and you'll know exactly what kind of question that index can answer well and which it can't.

Analogy: a book's back-of-the-book index

At the back of many nonfiction books there's an index: an alphabetical list of keywords, each followed by the page numbers where it appears. You look up "cancellation" in that index, and instead of flipping through all 300 pages of the book, you go straight to pages 45, 112, and 203. The index doesn't understand the book — it doesn't know that "cancellation" and "voiding a booking" talk about the same thing — but it knows, with mechanical precision, exactly which pages that word appears on.

An inverted index is exactly that back-of-the-book index, applied to Reservo's chunks instead of pages: for every word in the vocabulary, it stores the list of chunks where it appears (and how many times). Searching for "boardroom" doesn't force you to re-read all 57 chunks — the index already knows, ahead of time, which six of them contain that word.

But a real book index doesn't stop at "appears or doesn't appear": pages where a word appears many times probably talk more about that topic than a page where it appears once, in passing. And a rare word — "Phonebooth" — is more informative than a common word — "room", which shows up on almost every page. BM25 is the version of that index that weighs both things: how often a term repeats within a chunk (frequency) and how rare that term is across the whole corpus (rarity). It's the same back-of-the-book index, but with a relevance criterion instead of just presence/absence — while still never stopping being, at its core, exact word matching. A back-of-the-book index doesn't understand that "car" and "automobile" are the same thing; neither does BM25.

The case that stays the same: Reservo

This module indexes Reservo's complete, fixed corpus: 13 documents, 57 chunks — exactly the same ones Module 1's chunker produced (5 chunks per room manual, one for each of its 5 fixed sections; 4 chunks for each policy/FAQ document and for operations-manual-raw), covering policies (cancellation, refund, no-show), FAQs (bookings, memberships, payments, wifi/equipment), and the five room manuals — including the anchors from agent-fundamentals-and-tool-calling (Focus, Studio, Boardroom) plus the two rooms unique to this guide (Lounge, Phonebooth). Document 13, operations-manual-raw, is the "dirty" text Module 1 practiced cleaning up (repeated headers, broken hyphenation, page breaks simulating a PDF extraction); from this module onward, its already-cleaned version gets indexed like any other internal FAQ.

doc_idTopic
cancellation-policyCancellation window by tier (basic/pro), how to cancel
refund-policyWhen a refund applies
no-show-policyConsequences of not showing up
booking-faqHow to book, deposit, booking window
membership-tiers-faqbasic vs pro, 20% discount
payment-methods-faqAccepted/unaccepted payment methods
focus-room-manual · studio-room-manual · boardroom-room-manual · lounge-room-manual · phonebooth-room-manualCapacity and equipment for each room
wifi-and-equipment-faqWifi and equipment shared across all rooms
operations-manual-rawInternal opening/closing checklist (clean starting from this module)

The fixed evaluation set you'll use in lesson 07 and in the mini-project — the same table that reappears, unchanged, in Module 6 — has a deliberate trap in the fourth row: a refund question whose correct answer is NOT the refund document.

QueryExpected doc_id
"What is the cancellation policy for Boardroom bookings?"cancellation-policy
"How much discount does the pro tier get?"membership-tiers-faq
"What equipment is in the Focus room?"focus-room-manual
"Can I get a refund if I didn't show up?"no-show-policy (trap: NOT refund-policy)
"What payment methods does Reservo accept?"payment-methods-faq
"Is there wifi in the Lounge?"wifi-and-equipment-faq

The Chunk you'll index

from dataclasses import dataclass


@dataclass(frozen=True)
class Chunk:
    chunk_id: str    # unique identifier, e.g. "cancellation-policy-003"
    doc_id: str       # source document, e.g. "cancellation-policy"
    title: str        # document title
    section: str      # section within the document
    position: int      # chunk's position within the document (0, 1, 2...)
    text: str         # the chunk's text, in English (sample data)

This is exactly the Chunk that an ingestion pipeline like Module 1's produces: content plus the metadata needed to cite the source later (doc_id, title, section, position). This module doesn't change that shape — it takes it as given and builds the index on top of it.

Prerequisites

Required knowledge:

  • ✅ Basic Python: dataclass, dictionaries, collections.Counter, list comprehensions.
  • ✅ A notion of logarithms (math.log) — explained from scratch when they show up, but it helps to have seen them before.

Recommended:

  • ✅ Having finished Module 1 (or at least knowing what a Chunk is and where it comes from) — not required, this module rebuilds the fixed corpus from scratch.

NOT required:

  • ❌ You don't need sentence-transformers, chromadb, or scikit-learn — these libraries aren't preinstalled and require network access; this guide never installs them. If at some point in your career you need real semantic embeddings, that's the door of embeddings-deep-dive-guide and vector-databases-fundamentals-guide, not this one.
  • ❌ You don't need to know BM25 ahead of time — it's built from scratch, term by term.
  • ❌ You don't need an internet connection or an API key: this whole module runs locally.

Environment:

  • ✅ Python 3.14.0 with its standard library (re, collections.Counter, math, dataclasses).
  • ✅ numpy 2.5.1, already installed in this environment — used to build the term-document matrix and to verify the hand-rolled IDF, not for any kind of embedding.
  • ✅ A text editor and a terminal.

Module roadmap

Lesson 01 — Module introduction (this one)

The back-of-the-book index analogy, the fixed 13-document/57-chunk corpus, and the map of the 8 lessons.

Lesson 02 — What a retrieval index is

Why a linear scan (reading every chunk, one by one) doesn't scale, executed and measured on this same corpus. The family of retrieval indexes and where BM25 falls within it.

Lesson 03 — The inverted index

The term → chunk-list structure, built and run over the full Reservo corpus. Postings, vocabulary, and why it solves the problem from lesson 02.

Lesson 04 — Term frequency and IDF

How often a term repeats within a chunk (TF) and how rare it is across the whole corpus (IDF), hand-rolled with Counter/math and verified with a numpy matrix. A first ranking attempt (TF-IDF + cosine) that works, but isn't perfect — the reason BM25 exists.

Lesson 05 — BM25 scoring

The full Robertson-Zaragoza formula: k1 and b, done by hand, with a term-by-term calculation verified against the real function. A direct comparison against lesson 04's TF-IDF on the same query.

Lesson 06 — The lexical limit: a synonym that misses

🛑 The module's honesty lesson: a query with a word the corpus never uses (reimbursement instead of refund) run against the index, showing the real failure — zero results or irrelevant ones. Points to embeddings-deep-dive-guide.

Lesson 07 — search(query, k), executed

You assemble build_index and search(query, k) -> list[Chunk], and run it against the module's six anchor queries, with real scores.

Lesson 08 — Mini-project: a BM25 index over Reservo

You build the full index across three files of your own and confirm, in your terminal, that all six anchor queries return the correct doc_id in first place — plus lesson 06's lexical-limit check.

Progression map

Lesson 01 (this one) → The fixed corpus + the back-of-the-book index analogy
Lesson 02            → Why you need an index (linear scan doesn't scale)
Lesson 03            → The inverted index: term → chunks
Lesson 04            → TF and IDF, by hand and verified with numpy
Lesson 05            → BM25: TF + IDF combined with saturation and normalization
Lesson 06            → The lexical limit, DEMONSTRATED
Lesson 07            → search(query, k), run over the 6 anchor queries
Lesson 08            → Project: the complete index, in three files, executed

Difficulty: ⭐⭐ ──────────────────▶ ⭐⭐⭐

What you'll achieve in this module

By completing all 8 lessons, you'll be able to:

  1. Explain why a linear scan doesn't scale and what an inverted index solves.
  2. Build an inverted index (term → list of chunks) from scratch, with pure stdlib.
  3. Compute TF and IDF by hand, and verify those calculations with a numpy matrix.
  4. Implement full BM25 (k1, b, frequency saturation, length normalization) and compute a score by hand, term by term.
  5. Demonstrate BM25's lexical limit by running code — not just describing it.
  6. Assemble search(query, k) -> list[Chunk] and run it against a fixed set of queries, with reproducible results.

Before and after

BEFORE the module:
→ "Indexing is just storing the chunks in a list"
→ "An inverted index is just an abstract data structure"
→ "TF-IDF and BM25 are basically the same thing"
→ "If the index doesn't find a relevant chunk, you need to fix the index"

AFTER the module:
→ Indexing is building a structure that answers queries WITHOUT scanning everything
→ You build an inverted index yourself, with Counter and a dict, and you run it
→ BM25 improves on TF-IDF with frequency saturation (k1) and length normalization (b)
→ If the query's word never appears in the corpus, NO lexical index
  will find it — that's a limit of the algorithm FAMILY, not a bug

Traps to avoid while taking this module

1. "BM25 will understand that 'refund' and 'reimbursement' mean the same thing"

No. BM25 compares exact terms (after simple tokenization). If the query's word doesn't appear in any chunk, that term contributes nothing to the score — no matter how semantically related it is to what does appear. Lesson 06 demonstrates this by running exactly that case.

2. "More results in the top-k is always better"

Not necessarily. k is a budget: how many chunks the caller of search is willing to receive. A bigger k doesn't improve ranking quality — it only changes how many of the already-ranked results get returned. Lesson 07 shows this with k=3 over the six anchor queries.

3. "I can already expose this as an agent tool with what I learned in M2"

Not yet. This module gives you search(query, k) -> list[Chunk] as an ordinary Python function. Wrapping it in the name/description/input_schema contract of a tool — so an agent can invoke it — is Module 3. Here you build the engine; there you connect it.

4. "If I build an inverted index, I don't need TF or IDF anymore"

The opposite: the inverted index is the structure that makes it fast to look up which chunks a term appears in — but on its own, it doesn't say which of those chunks is more relevant. TF and IDF (lesson 04) are the relevance criterion computed on top of that structure. One without the other isn't enough.

5. "numpy is here to do embeddings"

Not in this module. numpy is used exclusively to build the term-document matrix and to verify, in vectorized form, the same IDF calculations you did by hand — not for any kind of semantic vector. That distinction is repeated explicitly in lesson 04.

How to work through this module

  1. Run every script, don't just read it. The numbers in this module (57 chunks, 509 vocabulary terms, BM25 scores with several decimal places) only make sense once you see them come out of your own terminal.
  2. Compare the "before" (TF-IDF) against the "after" (BM25) in lesson 05. The contrast, on the same query, is the best way to understand what frequency saturation and length normalization add.
  3. Don't skip lesson 06. It's the shortest lesson in the module and the most important: without watching the lexical limit fail in your own terminal, it's easy to overestimate what BM25 can do.

Estimated time:

Lesson 01 (this one) →  15 min reading
Lesson 02            →  20 min + running the linear scan
Lesson 03            →  25 min + building the inverted index
Lesson 04            →  30 min + running TF/IDF and the numpy matrix
Lesson 05            →  30 min + running BM25 and the hand calculation
Lesson 06            →  20 min + running the trap query
Lesson 07            →  25 min + running the 6 anchor queries
Lesson 08            →  35 min + assembling the full index

Total: ~3.3 hours

Evidence of success

Before moving on to Module 3 (search_docs as an agent tool), you should be able to:

  • ✅ Build an inverted index from a list of Chunk, with pure stdlib.
  • ✅ Compute TF, IDF, and the BM25 score of a chunk against a query, by hand and in code.
  • ✅ Explain and demonstrate why BM25 is lexical, not semantic — with a real query that fails.
  • ✅ Run search(query, k) against the full Reservo corpus and judge, with real criteria, which of the six anchor queries land the expected doc_id in the top-1 spot and which don't — and why.

Summary

  • This module builds Reservo's RAG search engine: a hand-rolled BM25 index, with no external libraries, over the corpus's fixed 57 chunks.
  • BM25 is real lexical retrieval — the same algorithm behind Elasticsearch/OpenSearch — never a semantic embedding. That distinction repeats in every lesson that touches the index, and lesson 06 demonstrates it by running a real failure.
  • numpy only shows up for the term-document matrix and to verify the hand-computed IDF — not for any kind of embedding.
  • The module closes with search(query, k) -> list[Chunk], run against the six anchor queries that will reappear, unchanged, in Module 6 — at k=3, only one of the six lands the expected doc_id in the top-1 spot. The other five are, once executed, the module's most honest evidence of what BM25 can and can't do: a couple of short, generic chunks (cancellation-policy-002/-003, which mention "cancellation" and list other doc_id values between backticks) share enough common vocabulary with almost any question to sneak in — and often win — ahead of the chunk that actually answers it (lesson 07).
  • Boundary: turning search into a tool an agent can invoke is Module 3; real embeddings and vector indexes belong to embeddings-deep-dive-guide and vector-databases-fundamentals-guide.

Next lesson: 02 — What a retrieval index is. We start by seeing, executed, why a linear scan over 57 chunks already starts to show its seams.

Additional resources

  1. Python — dataclasses — The exact shape of the Chunk this module indexes.
  2. Python — collections.Counter — The structure that counts terms in each chunk throughout the module.
  3. Robertson & Zaragoza — "The Probabilistic Relevance Framework: BM25 and Beyond" — The academic reference for the BM25 formula this module implements.
  4. NumPy — Official documentation — The library used for lesson 04's term-document matrix.
  5. Python 3.14 — What's New — The version all the engineering in this module runs on.
← PreviousMini-project: Ingest the Reservo CorpusNext →What Is a Retrieval Index