Assembling the ingestion pipeline

Description

Everything that follows in this project — the index, the tool, the agent, reingestion, evaluation, the operational layer — depends on one single thing: the starting corpus being exactly the same, character for character, in every piece. This lesson lays that foundation in your own working directory: the RAW_DOCS block plus the full chunker, copied verbatim from Module 1, not a comma different. There's no new engineering to learn here — you already built and tested every parser, every cleaning rule, every chunking strategy in its own module — this lesson's job is the discipline of reproducing it exactly, and confirming with your own run that the result is still 57 chunks over 13 documents.

Connection to the module

This is the project pipeline's first piece: reservo_corpus.py is the foundation the following six lessons build on, without exception. A single word's error here propagates through the rest of the project — it would change BM25 scores, search_docs results, and even Lesson 06's evaluation score — so this lesson closes with an explicit check before moving on.

Analogy: the document intake room, day one

Of the seven people from the full-front-desk analogy, this lesson is the first to start work: the one who receives Reservo's raw documents — the markdown policies, the HTML manuals, the operations manual with the typical mess of an automated extraction — and leaves them clean, chunked, with their metadata card, ready for the rest of the team to use. If this person mislabels a document or loses a paragraph during cleanup, no one working after them — not the person who builds the catalog, not the one staffing the desk, not the one auditing quality — can fix that error downstream. That's why today's work ends, before anything else, with an explicit check that intake was done right.

Step 1: a disposable working directory

mkdir -p "$(mktemp -d)/reservo-capstone" && cd "$_"
pwd

What to expect: a new path under the system's temp directory, empty — the same disposable-directory pattern you already used in the Module 2, 4, 5, and 6 mini-projects.

Step 2: reservo_corpus.py, verbatim

Save this entire file, as is, in your working directory. It's exactly Module 1's canonical block — 13 raw documents (RAW_DOCS) and the full chunker (parse_markdown, parse_html, clean_text, sectionize_cleaned, chunk_by_sentence, chunk_by_structure, ingest_document, build_corpus) — unchanged:

# reservo_corpus.py
# =============================================================================
# RESERVO CANONICAL CORPUS -- single, self-contained, runnable block.
# Python 3.14.0 + stdlib (re, html.parser, dataclasses) -- no numpy, no network.
# INGESTION_DATE fixed (no datetime.now(), no random).
# =============================================================================
import re
from dataclasses import dataclass
from html.parser import HTMLParser

INGESTION_DATE = "2026-01-15"

RAW_DOCS: dict[str, tuple[str, str]] = {}  # doc_id -> (fmt, raw_text)

RAW_DOCS["cancellation-policy"] = ("md", """# Cancellation Policy

## Basic Tier Cancellation Window

Basic members can cancel a booking up to 24 hours before the reserved start time with no penalty. Cancellations made less than 24 hours in advance forfeit the full booking amount.

## Pro Tier Cancellation Window

Pro members get a shorter, friendlier window: cancellations up to 4 hours before the reserved start time are free of charge. This is one of the perks of the pro tier, alongside the 20% discount on hourly rates.

## How to Cancel

Cancellations go through the same booking system used to reserve the room. There is no phone line for cancellations; the system timestamp is what determines whether the cancellation was made in time.

## Related Policies

See `refund-policy` for what happens to the money once a cancellation is processed, and `no-show-policy` for what happens if you simply do not show up without cancelling.
""")

RAW_DOCS["no-show-policy"] = ("md", """# No-Show Policy

## What Counts as a No-Show

A no-show is a booking where the member never checks in during the reserved hours and never cancelled beforehand. This is different from a late cancellation, which is covered in `cancellation-policy`.

## What Happens on a No-Show

No-shows are charged the full amount of the booking. Unlike a late cancellation, there is no partial leniency: the no-show fee equals the entire reserved price, and it is never refunded under `refund-policy`.

## Repeated No-Shows

Members with three or more no-shows in a rolling 30-day window lose the ability to book same-day reservations; all future bookings must be made at least 24 hours in advance until the pattern clears.

## Why This Policy Exists

Rooms held for a no-show cannot be re-offered to another member during that window, so the fee reflects real lost capacity, not a punitive charge.
""")

RAW_DOCS["refund-policy"] = ("md", """# Refund Policy

## What Qualifies for a Refund

A refund applies when a booking is cancelled within the free window described in `cancellation-policy`, or when Reservo cancels a confirmed booking because of a facility issue, such as a maintenance problem or a power outage.

## Refund Amount and Timing

Eligible refunds return the full amount charged for the booking to the original payment method. Processing takes up to 5 business days once the cancellation is confirmed in the booking system.

## What Does Not Qualify for a Refund

Cancellations made outside the free window are not eligible; the booking amount is forfeited under `cancellation-policy`. No-shows are never refunded, regardless of membership tier; see `no-show-policy` for that separate case.

## How to Request a Refund

Eligible refunds are issued automatically once a qualifying cancellation is processed; members do not need to submit a separate request. Questions about a specific refund go to member support through the booking system.
""")

RAW_DOCS["booking-faq"] = ("md", """# Booking FAQ

## How Do I Book a Room?

Bookings are made through the Reservo booking system by choosing a room, a date, and a start and end time. A confirmation appears immediately, and the room is held exclusively for that window.

## Is a Deposit Required to Book?

Yes. A deposit equal to the full session amount is charged at the time of booking, through the payment method on file; see `payment-methods-faq` for what is accepted.

## Can I Book More Than One Room at a Time?

Yes, a member can hold bookings in multiple rooms at once, as long as the times do not overlap for the same member. Each room booking is billed and cancelled separately.

## How Far in Advance Can I Book a Room?

Rooms can be booked up to 60 days in advance. Same-day booking is allowed, subject to the same-day restriction described in `no-show-policy` for members with repeated no-shows.
""")

RAW_DOCS["membership-tiers-faq"] = ("md", """# Membership Tiers FAQ

## What Is the Difference Between Basic and Pro?

Basic is the default tier for every new member, with no monthly fee. Pro is a paid upgrade that adds a shorter cancellation window, see `cancellation-policy`, and a discount on every booking.

## How Much Discount Does the Pro Tier Get?

Pro members receive a 20% discount on the hourly rate of every room, applied automatically at checkout. No code or coupon is needed; the discount is tied to the membership tier on the account.

## How Do I Upgrade to Pro?

Upgrading to Pro takes effect immediately from the account settings page. The new cancellation window and the 20% discount apply starting with the very next booking made after the upgrade.

## Can I Downgrade Back to Basic?

Yes, at any time from the account settings page. Downgrading takes effect on the next booking; any booking already confirmed under Pro keeps its Pro-tier terms.
""")

RAW_DOCS["payment-methods-faq"] = ("md", """# Payment Methods FAQ

## What Payment Methods Does Reservo Accept?

Reservo accepts major credit and debit cards on file with the account. The same card charged for a booking deposit is used automatically for any no-show or late-cancellation charge.

## Does Reservo Accept Cash?

No. All bookings, deposits, and no-show charges are processed electronically through the payment method on file.

## What Happens if a Card Is Declined?

A declined card cancels the booking hold immediately; the room is released back to the schedule. The member is notified and can retry with the same card or add a different one.

## Can I Split a Payment Between Two Cards?

No, a single booking can only be charged to one card on file at a time. Members who want to change which card is used should update the default payment method before booking.
""")

RAW_DOCS["wifi-and-equipment-faq"] = ("md", """# Wifi and Equipment FAQ

## Is Wifi Included in Every Room?

Yes, building-wide wifi reaches every room, from Phonebooth to Boardroom, at no extra cost. Coverage is the same in every room regardless of size or hourly rate.

## What Is the Wifi Network Name and Password?

The network name and password are posted on a card inside each room and also shown on the booking confirmation screen. The password rotates monthly for security.

## What Common Equipment Is Available Outside the Rooms?

The building shares a printer and a water station on the ground floor, available to any member with an active booking. Room-specific equipment is listed in each room manual.

## Who Do I Contact if the Wifi Is Down?

Report a wifi outage through the booking system's support option; staff follow the internal reset procedure and typically restore the connection within a few minutes.
""")

RAW_DOCS["focus-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Focus Room Manual</title></head>
<body>
<h1>Focus Room</h1>
<p>Focus is Reservo's single-occupancy room, designed for calls and deep work
that needs a closed door. It is the smallest and least expensive room in the
building.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 1 person. The room has one desk, one chair, and a soundproofed
door. There is no window, by design, to minimize visual distraction.</p>
<h2>Equipment</h2>
<ul>
<li>27-inch monitor with HDMI input</li>
<li>Adjustable desk lamp</li>
<li>Wall outlet with two USB-C ports</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $25.00 per hour (2500 cents), the lowest rate in the building.
Pro members receive the standard 20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Focus is not soundproof against phone ringtones; members are asked to keep
devices on silent. Food is allowed but no hot meals, due to the room's small
size and lack of ventilation.</p>
</body>
</html>
""")

RAW_DOCS["studio-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Studio Room Manual</title></head>
<body>
<h1>Studio Room</h1>
<p>Studio is Reservo's small-team room, built for a working session that
needs a table and a whiteboard rather than a single desk. It sits between
Focus and Boardroom in both size and price.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 4 people. The room has a round table, four chairs, and a
wall-mounted whiteboard. A large window faces the courtyard.</p>
<h2>Equipment</h2>
<ul>
<li>43-inch monitor with HDMI and USB-C input</li>
<li>Wall-mounted whiteboard with markers</li>
<li>Conference speakerphone</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $40.00 per hour (4000 cents). Pro members receive the standard
20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Studio can be booked for up to 4 consecutive hours per reservation. The
whiteboard must be wiped clean before the next booking begins.</p>
</body>
</html>
""")

RAW_DOCS["boardroom-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Boardroom Room Manual</title></head>
<body>
<h1>Boardroom</h1>
<p>Boardroom is Reservo's largest room, reserved for formal meetings,
client presentations, and full-team gatherings. It is the only room with a
dedicated presentation screen.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 10 people. The room has a long table, ten chairs, and a
wall-mounted presentation screen at the head of the table.</p>
<h2>Equipment</h2>
<ul>
<li>75-inch presentation screen with HDMI and wireless casting</li>
<li>Conference speakerphone with ceiling microphones</li>
<li>Wall-mounted whiteboard with markers</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $80.00 per hour (8000 cents), the highest rate in the
building. Pro members receive the standard 20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Boardroom receives a full clean every evening regardless of usage,
because of its size. Food is allowed only during bookings longer than 2
hours, and must be cleared before the room is released.</p>
</body>
</html>
""")

RAW_DOCS["lounge-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Lounge Room Manual</title></head>
<body>
<h1>Lounge</h1>
<p>Lounge is Reservo's informal meeting room, built for a relaxed
conversation rather than a formal presentation. It is open-plan, with no
door separating it from the hallway.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 6 people. The room has two low sofas, a coffee table, and
extra chairs stacked against the wall for larger groups.</p>
<h2>Equipment</h2>
<ul>
<li>32-inch monitor with HDMI input</li>
<li>Bluetooth speaker</li>
<li>Coffee and tea station</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $50.00 per hour (5000 cents). Pro members receive the standard
20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Lounge is open-plan and does not require a keypad code, unlike Focus,
Phonebooth, and Boardroom. Because there is no door, members should keep
calls at conversational volume.</p>
</body>
</html>
""")

RAW_DOCS["phonebooth-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Phonebooth Room Manual</title></head>
<body>
<h1>Phonebooth</h1>
<p>Phonebooth is Reservo's smallest room, built for a single short call
rather than a working session. It is the only room designed to be used
standing up.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 1 person. The room has a narrow shelf-desk and a single stool,
with just enough space to stand and pace during a call.</p>
<h2>Equipment</h2>
<ul>
<li>Wall-mounted phone charging dock</li>
<li>Small desk fan</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $15.00 per hour (1500 cents), the lowest rate in the building
alongside its small size. Pro members receive the standard 20% discount on
every booking.</p>
<h2>House Rules</h2>
<p>Phonebooth bookings are capped at 1 hour per reservation, since the room
is designed for short calls. Rooms with a physical door -- Focus,
Phonebooth, and Boardroom -- use a keypad code that rotates weekly.</p>
</body>
</html>
""")

# operations-manual-raw.txt: deliberately DIRTY (M1 practices parsing/cleaning
# on this one). Built so that clean_text(raw) reproduces M1's exact verbatim
# cleaned output (len(raw)=1473, len(clean)=1172, header repeated 4x, 5
# hyphen-breaks: week-\nends, ope-\nning, run-\nning, min-\nutes, re-\nset).
_HEADER = "RESERVO OPERATIONS MANUAL - INTERNAL - CONFIDENTIAL"


def _insert_break(text: str, word: str, nth: int = 1) -> str:
    """Split the nth occurrence of `word` at its midpoint into 'wo-\\nrd'."""
    start = 0
    count = 0
    idx = -1
    while True:
        idx = text.find(word, start)
        if idx == -1:
            raise ValueError(f"word {word!r} not found (occurrence {nth})")
        count += 1
        if count == nth:
            break
        start = idx + 1
    mid = len(word) // 2
    return text[:idx] + word[:mid] + "-\n" + word[mid:] + text[idx + len(word):]


_body1 = ("The building opens at 07:00 and closes at 22:00 on weekdays. On weekends the "
          "building opens at 09:00 and closes at 18:00. Staff must complete a "
          "walkthrough of every room before opening to confirm no equipment was left "
          "running overnight.")
_body1 = _insert_break(_body1, "weekends", 1)
_body1 = _insert_break(_body1, "opening", 1)
_body1 = _insert_break(_body1, "running", 1)

_body2 = ("Rooms are cleaned between every booking when the gap is 30 minutes or longer. "
          "For back-to-back bookings under 30 minutes, cleaning is limited to wiping the "
          "table and checking for left-behind belongings. Boardroom receives a full "
          "clean every evening regardless of usage, because of its size.")
_body2 = _insert_break(_body2, "minutes", 1)  # only the FIRST "minutes" breaks

_body3 = ("Rooms with a physical door - Focus, Phonebooth, and Boardroom - use a keypad "
          "code that rotates weekly. Studio and Lounge are open-plan and do not require "
          "a code. Staff must update the keypad codes every Monday before 07:00 and log "
          "the change in the access log.")
# page 3 has no hyphen-break in the source

_body4 = ("If a member reports the wifi is down, staff should first check the router in "
          "the utility closet before escalating. A full reset takes approximately 3 "
          "minutes and drops every room's connection at once, so it should only be done "
          "between bookings, never during an active reservation.")
_body4 = _insert_break(_body4, "reset", 1)

RAW_DOCS["operations-manual-raw"] = ("txt", (
    _HEADER + "\n" + "Page 1 of 4" + "\n" + ("\n" * 3) +
    "Opening and Closing Procedures" + "\n\n" + _body1 + ("\n" * 10) +
    _HEADER + "\n" + "Page 2 of 4" + "\n" + ("\n" * 3) +
    "Cleaning Procedures" + "\n\n" + _body2 + ("\n" * 10) +
    _HEADER + "\n" + "Page 3 of 4" + "\n" + ("\n" * 3) +
    "Key and Access Handling" + "\n\n" + _body3 + ("\n" * 9) +
    _HEADER + "\n" + "Page 4 of 4" + "\n" + ("\n" * 3) +
    "Wifi Reset Procedure" + "\n\n" + _body4
))


@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    doc_id: str
    title: str
    section: str
    position: int
    text: str


# --- parse HTML (M1 L03) ----------------------------------------------------
WS_RE = re.compile(r"\s+")


class RoomManualParser(HTMLParser):
    """Extracts (title, [(section, text), ...]) from a room manual."""

    CAPTURE_TAGS = {"p", "li", "h1", "h2", "title"}

    def __init__(self):
        super().__init__()
        self.title = ""
        self.sections: list[tuple[str, list[str]]] = []
        self._current_section = "Overview"
        self._section_index: dict[str, int] = {}
        self._tag_stack: list[str] = []
        self._buffer: list[str] = []

    def handle_starttag(self, tag, attrs):
        self._tag_stack.append(tag)
        if tag in self.CAPTURE_TAGS:
            self._buffer = []

    def handle_endtag(self, tag):
        if self._tag_stack and self._tag_stack[-1] == tag:
            self._tag_stack.pop()
        text = "".join(self._buffer).strip()
        if tag == "title":
            self.title = text
        elif tag == "h2":
            self._current_section = text
        elif tag == "p" and text:
            self._add(text)
        elif tag == "li" and text:
            self._add(f"- {text}")

    def handle_data(self, data):
        if self._tag_stack and self._tag_stack[-1] in self.CAPTURE_TAGS:
            self._buffer.append(data)

    def _add(self, text):
        if self._current_section not in self._section_index:
            self._section_index[self._current_section] = len(self.sections)
            self.sections.append((self._current_section, []))
        idx = self._section_index[self._current_section]
        self.sections[idx][1].append(text)


def parse_html(raw_html: str) -> tuple[str, list[tuple[str, str]]]:
    parser = RoomManualParser()
    parser.feed(raw_html)
    sections = []
    for heading, paras in parser.sections:
        clean_paras = [WS_RE.sub(" ", p).strip() for p in paras]
        sep = "; " if all(p.startswith("- ") for p in clean_paras) else " "
        sections.append((heading, sep.join(clean_paras)))
    return parser.title, sections


# --- parse markdown (M1 L04) -------------------------------------------------
H2_SPLIT_RE = re.compile(r"(?m)^##\s+(.+)$")


def parse_markdown(raw_md: str) -> tuple[str, list[tuple[str, str]]]:
    lines = raw_md.strip("\n").split("\n")
    title = ""
    if lines and lines[0].startswith("# "):
        title = lines[0][2:].strip()
        lines = lines[1:]
    body = "\n".join(lines)

    parts = H2_SPLIT_RE.split(body)
    sections: list[tuple[str, str]] = []

    preamble = parts[0].strip()
    if preamble:
        sections.append(("Overview", preamble))

    for i in range(1, len(parts), 2):
        heading = parts[i].strip()
        content = parts[i + 1].strip() if i + 1 < len(parts) else ""
        content = re.sub(r"\s+", " ", content)
        sections.append((heading, content))

    return title, sections


# --- clean dirty .txt (M1 L04) -----------------------------------------------
HEADER_FOOTER_RE = re.compile(r"^RESERVO OPERATIONS MANUAL.*$\n?", re.MULTILINE)
PAGE_MARKER_RE = re.compile(r"^Page \d+ of \d+\s*$\n?", re.MULTILINE)
HYPHEN_BREAK_RE = re.compile(r"(\w)-\n(\w)")
BLANK_RUN_RE = re.compile(r"\n{2,}")
INNER_WS_RE = re.compile(r"[ \t]+")


def clean_text(raw: str) -> str:
    """Undoes the damage of a naive PDF/OCR-style text extraction."""
    text = HEADER_FOOTER_RE.sub("", raw)
    text = PAGE_MARKER_RE.sub("", text)
    text = HYPHEN_BREAK_RE.sub(r"\1\2", text)
    text = BLANK_RUN_RE.sub("\n\n", text)
    paragraphs = [p.replace("\n", " ").strip()
                  for p in text.split("\n\n") if p.strip()]
    paragraphs = [INNER_WS_RE.sub(" ", p) for p in paragraphs]
    return "\n\n".join(paragraphs)


def sectionize_cleaned(cleaned: str) -> list[tuple[str, str]]:
    """Cleaned text alternates short heading and body, one pair per old 'page'."""
    paragraphs = cleaned.split("\n\n")
    sections = []
    i = 0
    while i < len(paragraphs):
        heading = paragraphs[i]
        body = paragraphs[i + 1] if i + 1 < len(paragraphs) else ""
        sections.append((heading, body))
        i += 2
    return sections


# --- chunking strategies (M1 L05) --------------------------------------------
SENTENCE_RE = re.compile(r"(?<=[.!?])\s+(?=[A-Z(`])")


def split_sentences(text: str) -> list[str]:
    return [s.strip() for s in SENTENCE_RE.split(text) if s.strip()]


def chunk_by_sentence(text: str, max_size: int = 400) -> list[str]:
    """Packs complete sentences into chunks of up to max_size characters."""
    sentences = split_sentences(text)
    chunks: list[str] = []
    current: list[str] = []
    current_len = 0
    for sent in sentences:
        extra = len(sent) + (1 if current else 0)
        if current and current_len + extra > max_size:
            chunks.append(" ".join(current))
            current, current_len = [], 0
        current.append(sent)
        current_len += len(sent) + (1 if len(current) > 1 else 0)
    if current:
        chunks.append(" ".join(current))
    return chunks


def chunk_by_structure(
    sections: list[tuple[str, str]], max_size: int = 400
) -> list[tuple[str, str]]:
    """Respects section boundaries; only sub-splits a section past max_size."""
    out: list[tuple[str, str]] = []
    for heading, body in sections:
        if len(body) <= max_size:
            out.append((heading, body))
        else:
            for piece in chunk_by_sentence(body, max_size=max_size):
                out.append((heading, piece))
    return out


# --- metadata + ingestion (M1 L07/L08) ---------------------------------------
def ingest_document(doc_id: str, fmt: str, raw_text: str, max_size: int = 400) -> list[Chunk]:
    """Parses + (cleans) + chunks one document, attaching metadata."""
    if fmt == "md":
        title, sections = parse_markdown(raw_text)
    elif fmt == "html":
        title, sections = parse_html(raw_text)
    elif fmt == "txt":
        cleaned = clean_text(raw_text)
        title = "Operations Manual (raw, cleaned)"
        sections = sectionize_cleaned(cleaned)
    else:
        raise ValueError(f"unknown format: {fmt}")

    section_chunks = chunk_by_structure(sections, max_size=max_size)
    chunks = []
    for position, (section, text) in enumerate(section_chunks):
        chunk_id = f"{doc_id}-{position:03d}"
        chunks.append(Chunk(chunk_id, doc_id, title, section, position, text))
    return chunks


def build_corpus(max_size: int = 400) -> list[Chunk]:
    """Ingests all 13 RAW_DOCS into the single canonical list[Chunk] -- 57 chunks."""
    all_chunks: list[Chunk] = []
    for doc_id, (fmt, raw_text) in RAW_DOCS.items():
        all_chunks.extend(ingest_document(doc_id, fmt, raw_text, max_size=max_size))
    return all_chunks

The block above is the full canonical corpus: all 13 documents in RAW_DOCS, none abbreviated, plus the entire chunker — identical, character for character, to what this guide's DESIGN.md fixes in its "🛑 CANONICAL CORPUS — EXACT BLOCK" section. There's no need to go fetch it from another module: copied as is, it's ready in your working directory for the rest of the project.

Worked example: rebuilding the 57 chunks

With reservo_corpus.py saved in full (all 13 documents, none abbreviated), confirm it produces exactly the canonical corpus:

from reservo_corpus import build_corpus, RAW_DOCS

chunks = build_corpus()
assert len(chunks) == 57, f"expected 57, got {len(chunks)}"

print("documents:", len(RAW_DOCS), " total chunks:", len(chunks))
print()

by_doc = {}
for c in chunks:
    by_doc.setdefault(c.doc_id, []).append(c)

for doc_id, (fmt, _) in RAW_DOCS.items():
    doc_chunks = by_doc[doc_id]
    print(f"  {doc_id:26s} .{fmt:4s} -> {len(doc_chunks)} chunks")

sizes = [len(c.text) for c in chunks]
print()
print(f"chunk size -> min={min(sizes)} max={max(sizes)} avg={sum(sizes)/len(sizes):.1f}")

What to expect (run):

documents: 13  total chunks: 57

  cancellation-policy        .md  -> 4 chunks
  no-show-policy              .md  -> 4 chunks
  refund-policy                .md  -> 4 chunks
  booking-faq                   .md  -> 4 chunks
  membership-tiers-faq           .md  -> 4 chunks
  payment-methods-faq             .md  -> 4 chunks
  wifi-and-equipment-faq           .md  -> 4 chunks
  focus-room-manual                 .html -> 5 chunks
  studio-room-manual                .html -> 5 chunks
  boardroom-room-manual              .html -> 5 chunks
  lounge-room-manual                  .html -> 5 chunks
  phonebooth-room-manual               .html -> 5 chunks
  operations-manual-raw                 .txt -> 4 chunks

chunk size -> min=103 max=290 avg=174.4

These numbers — 57 chunks, 7 markdown documents with 4 chunks each, 5 HTML manuals with 5 chunks each, and operations-manual-raw (already cleaned) with 4 — are identical to what Module 1 first reported and to what this guide's DESIGN.md pinned as the single source of truth. 7×4 + 5×5 + 1×4 = 28 + 25 + 4 = 57. No module after M1 — not Module 2's index, not Module 3's tool, not this capstone — regenerates a different corpus: all of them import or reproduce this same block.

Additional check: chunk_id is 0-indexed, with zero-padding

sample_ids = [c.chunk_id for c in chunks if c.doc_id == "no-show-policy"]
print("no-show-policy chunk_ids:", sample_ids)

no_show_001 = next(c for c in chunks if c.chunk_id == "no-show-policy-001")
print("no-show-policy-001 ->", no_show_001.section, "| position =", no_show_001.position)

What to expect:

no-show-policy chunk_ids: ['no-show-policy-000', 'no-show-policy-001', 'no-show-policy-002', 'no-show-policy-003']
no-show-policy-001 -> What Happens on a No-Show | position = 1

This confirms the convention fixed since Module 1 and verified by execution in DESIGN.md: chunk_id = f"{doc_id}-{position:03d}", 0-indexed — no-show-policy-000 is the first chunk ("What Counts as a No-Show"), no-show-policy-001 is the second ("What Happens on a No-Show", position=1), not the first. Any piece of this project that assumes 1-indexing — or that's missing the zero-padding ("no-show-policy-1" instead of "no-show-policy-001") — isn't compatible with the rest of the pipeline.

Common mistakes

1. Abbreviating RAW_DOCS "to save space" and ending up with fewer than 13 documents. This lesson's assert len(chunks) == 57 exists exactly to catch this mistake early — if you copied 10 documents instead of 13, the assert fails loudly right here, not silently three lessons later with an evaluation score that matches nothing.

2. Changing a word of the text "because it doesn't look important". This corpus feeds a lexical index — a single character's difference in a document can change which chunk wins a close query later in the project (Lessons 03, 04, and 06). Copy the exact RAW_DOCS block, not a paraphrase.

3. Confusing max_size with a hard character limit per chunk. chunk_by_structure respects section boundaries first — it only sub-splits with chunk_by_sentence when an entire section exceeds max_size=400. In this corpus, no section exceeds it, so every (section, text) pair becomes exactly one Chunk — that's why the final count (57) matches exactly the sum of sections across the 13 documents.

4. Reordering RAW_DOCS and expecting the same result. The dictionary's insertion order determines build_corpus()'s order, which in turn determines the order of chunk_ids Module 2's index sees. It doesn't change the final count or BM25 scores (which don't depend on order), but it can change the order in which tied-score results appear — use the original block's same insertion order for exact reproducibility.

Exercises

Exercise 1: Confirm one format's breakdown (Easy)

Without running anything new: according to this lesson's "What to expect" table, how many total chunks do the 5 HTML manuals contribute? And the 7 markdown documents? Verify your answer by summing len(by_doc[doc_id]) for each group.

html_docs = [d for d, (fmt, _) in RAW_DOCS.items() if fmt == "html"]
md_docs = [d for d, (fmt, _) in RAW_DOCS.items() if fmt == "md"]

html_total = sum(len(by_doc[d]) for d in html_docs)
md_total = sum(len(by_doc[d]) for d in md_docs)

print(f"HTML: {len(html_docs)} documents -> {html_total} chunks")
print(f"Markdown: {len(md_docs)} documents -> {md_total} chunks")

HTML: 5 documents -> 25 chunks
Markdown: 7 documents -> 28 chunks

Exercise 2: Verify clean_text on the dirty document (Medium)

Confirm, in your own directory, that clean_text(RAW_DOCS["operations-manual-raw"][1]) produces exactly len(raw)=1473 for the input and len(clean)=1172 for the output — the two numbers Module 1 verified by execution and that DESIGN.md pinned again.

from reservo_corpus import clean_text

raw = RAW_DOCS["operations-manual-raw"][1]
cleaned = clean_text(raw)

print("len(raw):", len(raw))
print("len(clean):", len(cleaned))
print("header count in raw:", raw.count("RESERVO OPERATIONS MANUAL"))
print("header count in clean:", cleaned.count("RESERVO OPERATIONS MANUAL"))

len(raw): 1473
len(clean): 1172
header count in raw: 4
header count in clean: 0

Exercise 3: Predict each document's chunk count before running anything (Hard)

Without running code: for each of the 13 documents, predict how many chunks it will produce, based only on the number of ## headings (for markdown) or <h2> tags (for HTML) it has, and on the fact that no section in this corpus exceeds max_size=400. Then run it and compare your prediction against the real result.

predicted = {}
for doc_id, (fmt, raw) in RAW_DOCS.items():
    if fmt == "md":
        predicted[doc_id] = raw.count("\n## ")
    elif fmt == "html":
        predicted[doc_id] = raw.count("<h2>") + 1  # +1 for the implicit Overview
    else:
        predicted[doc_id] = 4  # operations-manual-raw: 4 fixed "pages"

for doc_id in RAW_DOCS:
    actual = len(by_doc[doc_id])
    match = "OK" if predicted[doc_id] == actual else "MISMATCH"
    print(f"  [{match}] {doc_id:26s} predicted={predicted[doc_id]}  actual={actual}")

  [OK] cancellation-policy        predicted=4  actual=4
  [OK] no-show-policy              predicted=4  actual=4
  [OK] refund-policy                predicted=4  actual=4
  [OK] booking-faq                   predicted=4  actual=4
  [OK] membership-tiers-faq           predicted=4  actual=4
  [OK] payment-methods-faq             predicted=4  actual=4
  [OK] wifi-and-equipment-faq           predicted=4  actual=4
  [OK] focus-room-manual                 predicted=5  actual=5
  [OK] studio-room-manual                predicted=5  actual=5
  [OK] boardroom-room-manual              predicted=5  actual=5
  [OK] lounge-room-manual                  predicted=5  actual=5
  [OK] phonebooth-room-manual               predicted=5  actual=5
  [OK] operations-manual-raw                predicted=4  actual=4

Summary and next step

• reservo_corpus.py is now assembled in your working directory, verbatim from Module 1: 13 raw documents, the full chunker, and build_corpus() producing exactly 57 chunks.
• You confirmed, by running it, the breakdown by format (7 markdown × 4, 5 HTML × 5, 1 cleaned txt × 4) and the 0-indexed, zero-padded chunk_id convention.
• This file is the foundation for the rest of the project — every following lesson imports Chunk and build_corpus from here, without touching a line of the chunker again.

Next lesson: 03 — The index and the tool. You build the BM25 index over these 57 chunks (k1=1.5, b=0.75) and wrap it in search_docs, the tool the rest of the project calls.

Additional resources

1. production-rag-and-document-ingestion-guide — Module 1, Lesson 08 (mini-project): the full RAW_DOCS block, with all 13 documents unabbreviated, to copy into your own directory.
2. production-rag-and-document-ingestion-guide — DESIGN.md, "🛑 CANONICAL CORPUS — EXACT BLOCK" section: the single source of truth that fixes the count (57), the chunk_id scheme, and Chunk's six fields.
3. Python — html.parser and Python — re — the full stdlib foundation behind parse_html/parse_markdown/clean_text.
4. Python — dataclasses — the @dataclass(frozen=True) behind Chunk, immutable by design.