Module 8: Project — A Production-Ready `search_docs`

Description

Seven modules built, one at a time, a complete retrieval system for Reservo's documents: 57 chunks with metadata parsed and cleaned from three raw formats (Module 1), a hand-rolled BM25 index that makes them searchable (Module 2), the search_docs tool an agent can call with a declared contract (Module 3), an agent that decides when to search, reformulates, and combines tools in the same turn (Module 4), an ingestion pipeline that survives changing documents without duplicating or losing anything (Module 5), an evaluation harness that measures — with real numbers, not a promise — how well it retrieves (Module 6), and an operational layer that cites sources, filters noise, and honestly admits when there's nothing to answer with (Module 7). Each piece was tested separately, in its own module, with its own executed evidence.

This module adds no new piece. It's the capstone: you're going to assemble the seven pieces into a single pipeline, in your own working directory, and run it end to end over Reservo's full canonical corpus — the same RAW_DOCS + chunker block fixed since Module 1, not a character changed. By the end you'll have, run with your own terminal: full ingestion, the index built, search_docs answering real questions, an agent combining search_docs with get_quote in the same turn, an incremental reingestion that duplicates nothing, an honest evaluation report (real recall@k and precision@k, no number tuned to look better), and the operational layer filtering out what shouldn't reach a user. The last lesson closes out the entire guide and tells you, precisely, which ecosystem guide to follow for each limit this system deliberately leaves open.

Connection to the module

This lesson doesn't run new code — it places the project on the map of the eight modules, fixes exactly what gets reused from M1-M7 (with each piece's real signature, not a paraphrase), and describes the single pipeline Lessons 02-08 assemble step by step through to the final deliverable.

Where we are in the guide

Production RAG and document ingestion — the Reservo agent
├── Module 1: Parsing and chunking documents
│   → 13 raw documents (7 .md, 5 .html, 1 dirty .txt) → 57 chunks with metadata
├── Module 2: Indexing chunks for retrieval
│   → Hand-rolled BM25, k1=1.5/b=0.75, search(query, k) -> list[Chunk]
├── Module 3: search_docs as an agent tool
│   → SEARCH_DOCS_SCHEMA, search_docs(query, k=3) -> list[dict], MAX_K=5, MAX_CHARS=280
├── Module 4: Agentic retrieval in the loop
│   → run_agent/dispatch_parallel, reformulate, multi-hop, combine with get_quote, check_grounding
├── Module 5: Incremental and idempotent ingestion
│   → content_hash/doc_hash, detect_changes, reingest(), purge_doc — disposable sqlite3
├── Module 6: Evaluating retrieval quality
│   → fixed EVAL_SET, recall_at_k/precision_at_k, evaluate() — real numbers, no LLM judge
├── Module 7: Operating RAG in production
│   → cite_source, handle_no_results, apply_threshold — the layer that decides what reaches the user
└── Module 8: Project — search_docs, production-ready  ← YOU ARE HERE
    → The seven pieces, one pipeline, run end to end over the full corpus

No previous module was left "half-built" waiting for this close — each one delivered its piece finished and tested. What was missing wasn't new engineering: it was the evidence that the seven pieces, put together in the same directory, with no last-minute adjustment, still produce exactly the same numbers each module measured separately. That's this module's job.

Analogy: the day Reservo opens the full front desk

Each previous module built, trained, and tested a different member of Reservo's support team: the one who parses and organizes incoming new documents (Module 1), the one who builds the searchable catalog (Module 2), the one who staffs the desk with a clear sign of what they can answer (Module 3), the one who decides on the spot whether the catalog needs consulting or an already-computed rate is enough (Module 4), the one who makes sure the catalog never goes stale when something changes (Module 5), the one who periodically audits whether the catalog genuinely finds what it should (Module 6), and the one who filters out, at the final handoff, whatever isn't good enough to leave the desk (Module 7). Each of those seven people was trained and evaluated separately, in their own trial shift, with their own performance report.

This module is opening day: the seven people working together, on the same shift, in the same real building, answering the same person walking in the door with a question that mixes several things at once. It's not a new rehearsal — it's the confirmation, with the real public of that first day, that the previous seven months of work genuinely works together.

What gets reused from M1-M7 (read this before continuing, with the real signatures)

No piece in this module gets rewritten from scratch. The full list, with the exact signature each module fixed:

From	Piece	Signature
M1	`RAW_DOCS`, the full chunker	`Chunk(chunk_id, doc_id, title, section, position, text)` frozen; `ingest_document(doc_id, fmt, raw_text) -> list[Chunk]`; `build_corpus() -> list[Chunk]` — 57 chunks, 0-indexed `chunk_id` (`"cancellation-policy-000"`)
M2	The BM25 index	`K1 = 1.5`, `B = 0.75`; `build_index(chunks) -> Index`; `search(query, k, index) -> list[Chunk]`
M3	The agent tool	`SEARCH_DOCS_SCHEMA` (`name`/`description`/`input_schema`); `search_docs(query, k=3) -> list[dict]`; `MAX_K = 5`; `MAX_CHARS = 280`
M4	The agentic runner	`run_agent(question, model_script, tools, max_iterations=10)`; `dispatch_parallel(tool_use_blocks, tools)`; `check_grounding(final_text, history) -> list[str]` (reused, in turn, from `agent-fundamentals-and-tool-calling`)
M5	The incremental chunk store	`content_hash(chunk) -> str`; `doc_hash(raw_text) -> str`; `ChangeSet(new, modified, deleted, unchanged)`; `detect_changes(conn, current_docs) -> ChangeSet`; `reingest(conn, current_docs) -> dict`; `purge_doc(conn, doc_id) -> int`
M6	The evaluation harness	`EVAL_SET: list[tuple[str, str]]` (6 queries, fixed expected `doc_id`); `recall_at_k(results, expected_doc_id) -> float`; `precision_at_k(results, expected_doc_id, k) -> float`; `evaluate(eval_set, index, k, search_fn)`
M7	The operational layer	`cite_source(hit) -> str`; `claim_supported(hit, required_terms) -> bool`; `handle_no_results(hits) -> str \| None`; `apply_threshold(hits, min_score=4.0) -> list[dict]`
`agent-fundamentals-and-tool-calling`	Reservo's canonical tools	`get_quote(room, tier, hours) -> dict`; `book_room(...)`; `cancel_booking(id)`; anchors `Focus=2500`/`Studio=4000`/`Boardroom=8000` cents/hour, pro `*80//100`

If any of these signatures feels unfamiliar, that's a sign the corresponding module needs a second pass before continuing — this project doesn't re-explain any of them from scratch.

The single pipeline: how the seven pieces connect

RAW_DOCS (13 raw docs)
    │  ingest_document()  [M1]
    ▼
build_corpus() -> 57 Chunk
    │  build_index()  [M2, k1=1.5/b=0.75]
    ▼
Index (BM25 over 57 chunks, 509 terms)
    │  search_scored()  [M2, exposes the score]
    ▼
search_docs(query, k) -> list[dict]  [M3, SEARCH_DOCS_SCHEMA, MAX_K=5]
    │
    ├──► run_agent(...) combining with get_quote  [M4]
    │
    ├──► evaluate(EVAL_SET, index, k, search)  [M6, recall@k/precision@k]
    │
    └──► cite_source / handle_no_results / apply_threshold  [M7]

RAW_DOCS (new version, with changes)
    │  reingest(conn, current_docs)  [M5, content_hash/doc_hash]
    ▼
updated sqlite3 chunk store, duplicating and losing nothing

Notice something important about this diagram: it isn't a single-pass linear chain. search_docs feeds three different consumers — the agent (M4), the evaluation harness (M6), and the operational layer (M7) — and the incremental chunk store (M5) updates the corpus independently, on its own cycle, without the rest of the pipeline having to stop. This is the real shape of a production system: not a pipe that runs once and finishes, but pieces reused from different angles over the same underlying data.

Hard execution rule (carried over from M1-M7)

The full pipeline IS executed. Every "What to expect" block in this module is real output from Python 3.14.0 + numpy 2.5.1, run over the canonical 57-chunk corpus — no exceptions, no numbers tuned to look better.
Lesson 06's evaluation score is the same one Module 6 already measured: 1/6 in the top-1, honestly. This module doesn't rerun the evaluation hoping for a different number — it reruns the same harness over the pipeline assembled here, and confirms the result didn't change.
The LLM's decision stays conceptual. No lesson in this module calls a real API; wherever claude-sonnet-5's decision shows up, it's a turn script (model_script) that realistically represents what the model would decide — the same pattern from M3 and M4.
No random, no datetime.now(). INGESTION_DATE = "2026-01-15" remains the only date in play, inherited from Module 1.
BM25 is always labeled real lexical retrieval, never semantic — this module's Lesson 07, the ecosystem-closing one, is precisely where the real semantic alternative gets named, without re-explaining it.
Current models only (claude-sonnet-5/claude-opus-5); never claude-3/gpt-*.

Prerequisites

Required knowledge:

✅ Modules 1-7 complete — this project doesn't reintroduce any piece from scratch, it assembles them.
✅ The EVAL_SET's six anchor queries (Module 6) and their expected doc_id — the project reuses them unchanged.
✅ agent-fundamentals-and-tool-calling's price anchors: Focus 2500¢/h, Studio 4000¢/h, Boardroom 8000¢/h, pro discount *80//100.

NOT required:

❌ You don't need to reread any BM25 mathematical derivation — Module 2 already covered it; here the formula is imported, not re-explained.
❌ You don't need any API key or a real model call.

Environment:

✅ Python 3.14.0 + numpy 2.5.1 (already installed, $0, no network) + stdlib (re, html.parser, hashlib, sqlite3, collections, dataclasses, math, concurrent.futures, json). Disposable working directory, mktemp -d.

Module roadmap

Lesson 01 — Module introduction (this one)

The full map of the eight modules, the list of reused signatures, and the diagram of the single pipeline the following lessons assemble.

Lesson 02 — Assembling the ingestion pipeline

reservo_corpus.py, the RAW_DOCS + full chunker block, copied verbatim from Module 1/DESIGN, run over the 13 documents: 57 chunks, with the breakdown by document and format.

Lesson 03 — The index and the tool

reservo_index.py (BM25, k1=1.5/b=0.75) and search_docs_tool.py (SEARCH_DOCS_SCHEMA, search_docs), run against the six anchor queries with real scores.

Lesson 04 — Agentic retrieval end to end

M4's runner (run_agent/dispatch_parallel/check_grounding) combining search_docs with get_quote in the same turn — the project's central demo: Focus pro 3 hours = 6000 cents, cited with grounding.

Lesson 05 — Incremental reingest in the capstone

Module 5's chunk_store.py, run three times over three corpus versions: initial ingestion (57), identical reingest (0 duplicates), and a run that modifies one document and deletes another at the same time.

Lesson 06 — The evaluation score

Module 6's full harness run over the pipeline assembled here: recall@1=0.1667, recall@3=0.6667, recall@5=0.8333, recall@6=1.0 — the same honest number, not forced to look better.

Lesson 07 — What your RAG still needs

Ecosystem close: which guide to follow for real reranking/hybrid search (advanced-rag-techniques-guide), context window budgeting (context-engineering-guide), conversation memory (agent-memory-and-state-guide), and the full semantic theory/infrastructure (embeddings-deep-dive-guide/vector-databases-fundamentals-guide).

Lesson 08 — Project: shipping `search_docs`

The complete pipeline, end to end, in a single script — the entire guide's final deliverable, with an executed verification checklist.

Evidence of success

Before considering this guide finished, you should be able to, with your own terminal:

✅ Rebuild the full canonical corpus (57 chunks, 13 documents) from RAW_DOCS and confirm the count didn't change.
✅ Run search_docs against the six anchor queries and explain, with the real scores, why only one hits the expected doc_id in first place.
✅ Run an agent that combines search_docs with get_quote in the same turn, and confirm with check_grounding that the final answer has no number without backing.
✅ Reingest the corpus twice with no duplication, and a third time with one document modified and another deleted at the same time.
✅ Report real recall@k and precision@k on the EVAL_SET, with no number tuned to look better.
✅ Name, for every limit this system deliberately leaves open, the exact ecosystem guide that genuinely solves it.

Summary

This module builds nothing new — it assembles M1-M7's seven pieces into a single pipeline, with each one's real signature, and runs them end to end over the full canonical corpus.
The pipeline connects search_docs to three different consumers (M4's agent, M6's harness, M7's operational layer) and to M5's incremental reingestion, which runs on its own independent cycle.
The hard rule holds: all the engineering genuinely runs, with real output; the LLM's decision stays conceptual, with claude-sonnet-5 as the only named model.

Next lesson: 02 — Assembling the ingestion pipeline. The project's first step: rebuild, verbatim, Module 1's RAW_DOCS + chunker block, and confirm it produces exactly the same 57 chunks the rest of the guide runs on.

Additional resources

production-rag-and-document-ingestion-guide — Modules 1-7 complete: the exact source of every signature reused in this project.
Anthropic — Tool use (function calling) overview — the reference for the contract search_docs follows throughout the pipeline.
Python 3.14 — What's New — the version this project's entire codebase runs on.
agent-fundamentals-and-tool-calling-guide — the source of get_quote/book_room/cancel_booking and the run_agent/dispatch_parallel runner this project combines with search_docs.

← PreviousMini-project: operating `search_docs`Next →Assembling the ingestion pipeline