Module 8: Project — A Production-Ready `search_docs`

What your RAG still needs

Next →

Description

Lesson 06 left an honest number: recall@1 = 0.1667, and BM25 confusing a cross-reference chunk with the real answer in four of six queries. This lesson doesn't try to fix that number — genuinely fixing it would go past the boundary this guide fixed since its design. Instead, it answers the question an honest system always leaves open at the end: where do you go after this guide? Four ecosystem guides each solve a specific, real limit of what search_docs built across these eight modules — and this lesson tells you, precisely, which one solves which, so you don't arrive at any of them without knowing why you need it.

Connection to the module

This is the guide's last conceptual lesson — it runs no new code. It picks up Lesson 06's executed diagnosis (and Module 6, Lesson 06's) and turns it into a map of next steps, with the same boundary discipline that held throughout every previous module: naming the exact neighboring guide, not vaguely "there's more to learn".

Analogy: the building's full blueprint, with the doors leading to other buildings

This guide's eight modules built a complete, working building: document intake, catalog, desk, customer service, catalog maintenance, quality audit, protocol for handling hard cases. The building works, end to end, with real evidence on every floor. But no real production building is an island: it has doors leading to other specialized buildings — one that knows how to understand a question's meaning beyond its exact words, one that decides how much of the agent's workspace each retrieved document takes up, one that remembers previous conversations with the same customer. This lesson is the blueprint with those doors marked, each one with the exact sign of where it leads, so the next person using this building knows exactly where to go next when the work done here isn't enough.

The map: four guides, four different limits

                    search_docs (this guide, M1-M8)
                    Lexical BM25, k1=1.5/b=0.75
                    recall@1=0.1667 measured and honest
                         │
    ┌────────────────────┼────────────────────┬─────────────────────┐
    ▼                    ▼                    ▼                     ▼
LIMIT 1:            LIMIT 2:             LIMIT 3:              LIMIT 4:
BM25 doesn't        what gets            no conversation       the ranking doesn't
understand          retrieved isn't      memory between        improve with
synonyms or         curated or           questions              re-ranking or
meaning             budgeted in the                              hybrid search
                     window
    │                    │                    │                     │
    ▼                    ▼                    ▼                     ▼
embeddings-deep-    context-             agent-memory-         advanced-rag-
dive-guide /        engineering-         and-state-guide       techniques-guide
vector-databases-   guide
fundamentals-guide

Every arrow on this map corresponds to a limit this guide explicitly measured in some module, not a generic "basic RAG" shortcoming. Let's go through the four, one by one.

Limit 1: BM25 is lexical, not semantic — the limit running through the entire guide

What this guide measures about it: every lesson that touched the index, from Module 2 onward, labeled BM25 as real lexical retrieval — exact term matching and frequency, the same algorithm Elasticsearch/OpenSearch use in production — never as a semantic embedding. Module 2's Lesson 06 check proved it with evidence: search("reimbursement", k=3) returns [], not because the system is broken, but because "reimbursement" — a real synonym of "refund" in everyday English — simply doesn't exist in the indexed vocabulary. No k1/b tuning can fix this: the problem isn't one of ranking, it's that the candidate term doesn't even come into play.

Where it genuinely gets solved:

  • embeddings-deep-dive-guide (AI Engineering) — the full theory: transformer architecture, tokenization, pooling, how a model learns that "reimbursement" and "refund" are neighbors in vector space even though they share not a single letter. This guide used an index; that guide teaches how an embedding works under the hood, from scratch.
  • vector-databases-fundamentals-guide (AI Engineering) — how a real vector index gets built and queried (HNSW, IVF, PQ) and how a production vector database gets operated (ChromaDB, Pinecone, Weaviate, Qdrant). This guide never installed any of these — they require network access and, in production, budget — but every lesson that touched the index named this real option at the exact point where it would apply.

Limit 2: what gets retrieved isn't curated or budgeted in the context window

What this guide measures about it: Module 7 (Lesson 06) measured how many chunks, characters, and approximate tokens each search_docs search carries — but deciding what order they enter the final prompt in, when to summarize instead of including full text, and how much relative space each one takes up against the rest of the conversation was never part of this guide. search_docs produces a list of candidate chunks, already filtered by apply_threshold; what happens to that list after it leaves here is, on purpose, another guide's territory.

Where it genuinely gets solved:

  • context-engineering-guide (sibling guide) — the context window's fine-grained budget: what goes in, in what order, when to compress or summarize, how to prioritize between what search_docs brought and everything else competing for space in the prompt (conversation history, other tools' results, system instructions). This guide RETRIEVES; context-engineering-guide decides how that enters the prompt — the boundary was named explicitly in this guide's DESIGN.md from the very first module, and held across all eight.

Limit 3: there's no conversation memory between questions

What this guide measures about it: every run of this guide's agent — including this module's Lesson 04 — resolves one question, in one conversation, with no memory at all of previous interactions with the same user. search_docs's index is memory of documents — Reservo's corpus, fixed and shared by any user asking — not memory of conversation: what this specific user asked last week, what preferences they mentioned, what bookings they made before.

Where it genuinely gets solved:

  • agent-memory-and-state-guide — persisting, across sessions, what a user was shown, their preferences, their history of interactions with the agent. This is precisely the guide you're in right now if you're reading this module inside it — the distinction it draws with itself is the most important of the four: search_docs's index is document memory shared by everyone; conversation memory is memory of this user, in this relationship with the agent, and lives in a completely different layer of the system.

Limit 4: the ranking doesn't improve with re-ranking or hybrid search

What this guide measures about it: Module 6 (Lesson 07) already mapped out, in detail, the three techniques that would improve BM25's ranking without changing the retrieval stage itself: re-ranking with cross-encoders (rereads each candidate with the full query, tells "mentions the topic" apart from "answers the question" — would fix the lexical magnet), hybrid search (combines BM25's lexical score with semantic similarity — the only one of the three that would fix the structural "reimbursement" limit), and query expansion/HyDE (transforms the query before searching). None of the three got implemented in this guide — DESIGN.md fixed that boundary from the first module.

Where it genuinely gets solved:

  • advanced-rag-techniques-guide (AI Engineering) — the full implementation of all three techniques, with real models, on a real corpus. This guide built the foundation — the ingestion pipeline, the index, the tool, the evaluation — that those techniques get applied on top of; that guide builds the upgrade, it doesn't re-explain it from scratch.

Summary table: which limit, which guide, what this guide measures about it

LimitMeasured in this guideSolved in
BM25 doesn't find synonyms ("reimbursement" → [])Module 2, Lesson 06embeddings-deep-dive-guide
No real vector index or vector databaseHard rule across the whole guide (named in every index lesson)vector-databases-fundamentals-guide
What gets retrieved isn't curated/budgeted in the promptModule 7, Lesson 06 (measures the size, doesn't decide placement)context-engineering-guide
No conversation memory between questionsAbsent by design in every agent example (M4, M8-L04)agent-memory-and-state-guide
The ranking doesn't improve with re-ranking/hybrid/expansionModule 6, Lesson 07 (full map of all three techniques)advanced-rag-techniques-guide
Semantic evaluation with an LLM judge (RAGAS, faithfulness)Boundary named since DESIGN.md; never run hereevaluation-frameworks-guide

Why none of these six limits is a flaw in this guide

Each of the table's six limits is a design boundary, fixed before this guide's first line of code was written, not an oversight discovered at the end. This guide's DESIGN.md says so explicitly from its first paragraph: this guide teaches the production engineering for an agent to retrieve information from documents — the ingestion pipeline, the index, the tool, incremental ingestion, pure retrieval evaluation, and the operational layer — not embedding theory from scratch, not context window budgeting, not conversation memory, not advanced retrieval techniques. That boundary exists because each of those five neighboring guides already covers its topic in genuine depth, and repeating it here — superficially, without the same executed rigor — would be worse than naming it and moving on.

What you can genuinely claim, with the evidence from this guide's eight modules, is this: the search_docs you built is an honest, real production foundation, not a toy prototype. It genuinely retrieves (57 chunks, full BM25), genuinely connects to an agent (real tool_use/tool_result protocol), genuinely survives changing documents (idempotency by content hash), genuinely gets measured (recall/precision with fixed ground truth), and genuinely operates with judgment (citations, threshold, zero results). Each of this map's five neighboring guides connects to this foundation — none replaces it.

Common mistakes

  1. Thinking "this guide doesn't cover X" means search_docs is incomplete or badly built. Every limit on this map is a boundary fixed by design, with the exact guide that solves it named — not an accidental gap. search_docs delivers exactly what it promises: real lexical retrieval, with its real lexical limits, measured and labeled in every lesson.

  2. Jumping straight to advanced-rag-techniques-guide without having measured the problem first. This guide's Module 6 (and this module's Lesson 06) exist precisely so you know what specifically fails — the lexical magnet, the structural vocabulary limit — before applying a technique to fix it. Applying re-ranking without knowing the real problem is missing vocabulary (not ordering) wastes effort on the wrong technique.

  3. Confusing document memory (this guide) with conversation memory (agent-memory-and-state-guide). search_docs's index is the same for any user asking — it doesn't change based on who's asking or what they asked before. Building conversation memory on top of search_docs, without distinguishing the two layers, mixes two systems with completely different lifecycles and goals.

  4. Skipping context-engineering-guide "because I already filtered with apply_threshold". apply_threshold (Module 7) decides which chunks pass the minimum-relevance filter — it doesn't decide what order they enter the prompt in, or how much space they take up against the rest of the conversation's context. These are two different decisions, at two different pipeline stages.

Exercises

Exercise 1: Classify a symptom into its correct limit (Easy)

For each of these three symptoms, identify which of this lesson's four limits it belongs to: (a) the agent confidently answers about a policy Reservo never documented; (b) the agent forgets, in the second question of a conversation, the room the user mentioned in the first; (c) search_docs("reimbursement policy") returns [] even though refund-policy does exist.

See solution

(a) Not any of this lesson's four limits — it's exactly the problem Module 7 (Lesson 03, handle_no_results) already solved within this guide: an agent without the instruction to answer only with what the tools return can fall back on its general knowledge instead of admitting "we don't have that". (b) Limit 3 — conversation memory, solved in agent-memory-and-state-guide; no agent example in this guide retains context between separate questions. (c) Limit 1 — BM25's structural lexical limit, solved with semantic embeddings in embeddings-deep-dive-guide; it's literally the same case, with the same word, Module 2 (Lesson 06) already demonstrated.

Exercise 2: Rank the five neighboring guides by how urgently Reservo would need them (Medium)

Imagine Reservo is going to real production tomorrow with exactly this project's system. Based on Lesson 06's evaluation score (recall@1=0.1667) and what you know about each neighboring guide, argue which of the five would solve the most urgent problem first, and which could wait.

See solution

There's no single correct answer — it depends on Reservo's real use case — but a reasonable argument: advanced-rag-techniques-guide (re-ranking/hybrid) would be the highest priority, because it directly attacks the lowest, most visible number (recall@1=0.1667) without requiring a change to the whole retrieval architecture — a re-ranking cross-encoder gets added on top of the existing BM25 index, without discarding it. context-engineering-guide would come next, because the system already works end to end (this module's Lesson 04 proves it) and the context budget only becomes a real problem as conversation and document volume grows. agent-memory-and-state-guide and embeddings-deep-dive-guide/vector-databases-fundamentals-guide are bigger investments — they require new infrastructure (a memory database, a vector database) — and would reasonably wait for the base system to prove real value with real users first. The important argument, more than the exact order, is that each choice is based on measured evidence (Lesson 06's real score), not a vague intuition of "improve everything at once".

Exercise 3: Design the evaluation question for agent-memory-and-state-guide (Hard)

This guide's EVAL_SET measures document retrieval with a fixed expected doc_id. If you had to design an analogous EVAL_SET for conversation memory — not for search_docs, but for a future memory layer that remembers user preferences across sessions — what field would replace doc_id, and what would an analogous recall_at_k do in that context? Write the signature in prose, without implementing it.

See solution

Instead of (query, expected_doc_id), a conversation-memory EVAL_SET would need something like (session_context, expected_fact) — where session_context is a previous conversation's simulated history (for example, "the user mentioned preferring to book Focus in the mornings") and expected_fact is the specific fact memory should retrieve in a later session ("which room does this user prefer?" → "Focus"). An analogous recall_at_k would answer "does the expected fact show up among the k memories the memory system brings back for this session?" — the same binary structure as this guide's recall_at_k, but against a per-user store of facts instead of a shared document corpus. The underlying difference, which justifies why this lives in a completely different guide: search_docs's ground truth is fixed and objective (the correct document exists, unambiguously, in the corpus); a conversation memory's ground truth depends on what happened in a specific earlier session, which varies by user and by history — a structurally different evaluation surface, which agent-memory-and-state-guide covers with its own rigor.

Summary and next step

  • search_docs is a real production foundation — 57 chunks, full BM25, a tool with a contract, incremental ingestion, evaluation with fixed ground truth, an operational layer — with six limits named by design, not discovered by accident.
  • Four ecosystem guides each solve a specific limit, measured in this guide: embeddings-deep-dive-guide/vector-databases-fundamentals-guide (lexical → semantic), context-engineering-guide (retrieve → budget in the prompt), agent-memory-and-state-guide (document memory → conversation memory), advanced-rag-techniques-guide (pure BM25 → re-ranking/hybrid/query expansion).
  • None of the four replaces what got built here — all of them connect on top of this foundation, extending it in a specific direction.

Next lesson: 08 — Project: shipping search_docs. The complete pipeline, end to end, in a single script — the entire guide's final deliverable.

Additional resources

  1. production-rag-and-document-ingestion-guide — DESIGN.md, "What this guide teaches (and what it does NOT)" section: the full boundary, fixed since the design, that this lesson picks up with executed evidence from all eight modules.
  2. production-rag-and-document-ingestion-guide — Module 6, Lesson 07 (the-path-to-better-recall): the detailed map of re-ranking/hybrid search/query expansion this lesson summarizes in the final table.
  3. embeddings-deep-dive-guide, vector-databases-fundamentals-guide, advanced-rag-techniques-guide — the three AI Engineering guides that solve, respectively, semantic theory, vector database infrastructure, and advanced retrieval techniques.
  4. context-engineering-guide and agent-memory-and-state-guide — the two sibling guides that decide what happens to what gets retrieved (window budget) and what the agent remembers between conversations (user memory), respectively.
← PreviousThe evaluation scoreNext →Project: shipping `search_docs`