Documents change in production

Description

Module 1 ingested Reservo's corpus exactly once: 13 documents, 57 chunks, and that's where the story ended — Lesson 08 of that module never ran the pipeline again on those same files. This lesson does what no previous module did: run the ingestion pipeline twice on the same corpus, without changing a single line of text between one run and the next, and look at what happens when there's no mechanism designed for that.

The answer, up front: something bad happens. If ingestion simply inserts every chunk it produces, without ever asking whether it already existed, running it twice doesn't give you the same corpus twice — it gives you double. This lesson reproduces that failure with real numbers, before Lessons 03-07 fix it piece by piece.

Connection to the module

This lesson reintroduces, verbatim, the canonical corpus and the full chunker from Module 1 — the reservo_corpus.py file the rest of this module imports without pasting it again. It then uses that corpus to run, end to end, the scenario that motivates everything that follows: reingesting with no idempotency control and watching the corpus duplicate.

Analogy: the first day with no registry

Picking up the clerk from the module introduction: this lesson is their first day on the job, before any fingerprint registry exists. They're handed the tray with Reservo's 13 documents and file them, one by one, as 57 new pages. Well done — the archive is complete.

The next morning, someone puts the same tray with the same 13 documents back on their desk (nobody told them these were already filed yesterday). The clerk, faithful to their one known procedure — "file what's in the tray" — re-photocopies and files the 57 pages again. The archive now has 114 pages: every document, duplicated. Nothing changed in the real world — no policy was updated, no manual was deleted — but the archive no longer reflects reality: it has double what it should. This is exactly the bug you're about to reproduce in code next, before Lesson 03 gives the clerk their first fingerprint registry.

The corpus and chunker you reuse from M1 (verbatim)

Everything that follows in this module starts from this same file, reservo_corpus.py — the canonical 13-document corpus and the full Chunk/ingest_document/build_corpus chunker, exactly as they stood at the end of Module 1 (Lessons 03-08). No lesson in this module changes a line of this block; we reproduce it in full here exactly once because it's the starting point for everything Module 5 builds, and from here on every lesson does from reservo_corpus import ... instead of repeating it.

# =============================================================================
# RESERVO CANONICAL CORPUS -- single, self-contained, runnable block.
# Python 3.14.0 + stdlib (re, html.parser, dataclasses) -- no numpy, no network.
# INGESTION_DATE fixed (no datetime.now(), no random).
# =============================================================================
import re
from dataclasses import dataclass
from html.parser import HTMLParser

INGESTION_DATE = "2026-01-15"

# -----------------------------------------------------------------------
# RAW_DOCS: Reservo's 13 raw documents, as they "exist" in
# corpus/ -- 7 .md (policies/FAQ), 5 .html (room manuals), 1 dirty .txt.
# -----------------------------------------------------------------------
RAW_DOCS: dict[str, tuple[str, str]] = {}  # doc_id -> (fmt, raw_text)

RAW_DOCS["cancellation-policy"] = ("md", """# Cancellation Policy

## Basic Tier Cancellation Window

Basic members can cancel a booking up to 24 hours before the reserved start time with no penalty. Cancellations made less than 24 hours in advance forfeit the full booking amount.

## Pro Tier Cancellation Window

Pro members get a shorter, friendlier window: cancellations up to 4 hours before the reserved start time are free of charge. This is one of the perks of the pro tier, alongside the 20% discount on hourly rates.

## How to Cancel

Cancellations go through the same booking system used to reserve the room. There is no phone line for cancellations; the system timestamp is what determines whether the cancellation was made in time.

## Related Policies

See `refund-policy` for what happens to the money once a cancellation is processed, and `no-show-policy` for what happens if you simply do not show up without cancelling.
""")

RAW_DOCS["no-show-policy"] = ("md", """# No-Show Policy

## What Counts as a No-Show

A no-show is a booking where the member never checks in during the reserved hours and never cancelled beforehand. This is different from a late cancellation, which is covered in `cancellation-policy`.

## What Happens on a No-Show

No-shows are charged the full amount of the booking. Unlike a late cancellation, there is no partial leniency: the no-show fee equals the entire reserved price, and it is never refunded under `refund-policy`.

## Repeated No-Shows

Members with three or more no-shows in a rolling 30-day window lose the ability to book same-day reservations; all future bookings must be made at least 24 hours in advance until the pattern clears.

## Why This Policy Exists

Rooms held for a no-show cannot be re-offered to another member during that window, so the fee reflects real lost capacity, not a punitive charge.
""")

RAW_DOCS["refund-policy"] = ("md", """# Refund Policy

## What Qualifies for a Refund

A refund applies when a booking is cancelled within the free window described in `cancellation-policy`, or when Reservo cancels a confirmed booking because of a facility issue, such as a maintenance problem or a power outage.

## Refund Amount and Timing

Eligible refunds return the full amount charged for the booking to the original payment method. Processing takes up to 5 business days once the cancellation is confirmed in the booking system.

## What Does Not Qualify for a Refund

Cancellations made outside the free window are not eligible; the booking amount is forfeited under `cancellation-policy`. No-shows are never refunded, regardless of membership tier; see `no-show-policy` for that separate case.

## How to Request a Refund

Eligible refunds are issued automatically once a qualifying cancellation is processed; members do not need to submit a separate request. Questions about a specific refund go to member support through the booking system.
""")

RAW_DOCS["booking-faq"] = ("md", """# Booking FAQ

## How Do I Book a Room?

Bookings are made through the Reservo booking system by choosing a room, a date, and a start and end time. A confirmation appears immediately, and the room is held exclusively for that window.

## Is a Deposit Required to Book?

Yes. A deposit equal to the full session amount is charged at the time of booking, through the payment method on file; see `payment-methods-faq` for what is accepted.

## Can I Book More Than One Room at a Time?

Yes, a member can hold bookings in multiple rooms at once, as long as the times do not overlap for the same member. Each room booking is billed and cancelled separately.

## How Far in Advance Can I Book a Room?

Rooms can be booked up to 60 days in advance. Same-day booking is allowed, subject to the same-day restriction described in `no-show-policy` for members with repeated no-shows.
""")

RAW_DOCS["membership-tiers-faq"] = ("md", """# Membership Tiers FAQ

## What Is the Difference Between Basic and Pro?

Basic is the default tier for every new member, with no monthly fee. Pro is a paid upgrade that adds a shorter cancellation window, see `cancellation-policy`, and a discount on every booking.

## How Much Discount Does the Pro Tier Get?

Pro members receive a 20% discount on the hourly rate of every room, applied automatically at checkout. No code or coupon is needed; the discount is tied to the membership tier on the account.

## How Do I Upgrade to Pro?

Upgrading to Pro takes effect immediately from the account settings page. The new cancellation window and the 20% discount apply starting with the very next booking made after the upgrade.

## Can I Downgrade Back to Basic?

Yes, at any time from the account settings page. Downgrading takes effect on the next booking; any booking already confirmed under Pro keeps its Pro-tier terms.
""")

RAW_DOCS["payment-methods-faq"] = ("md", """# Payment Methods FAQ

## What Payment Methods Does Reservo Accept?

Reservo accepts major credit and debit cards on file with the account. The same card charged for a booking deposit is used automatically for any no-show or late-cancellation charge.

## Does Reservo Accept Cash?

No. All bookings, deposits, and no-show charges are processed electronically through the payment method on file.

## What Happens if a Card Is Declined?

A declined card cancels the booking hold immediately; the room is released back to the schedule. The member is notified and can retry with the same card or add a different one.

## Can I Split a Payment Between Two Cards?

No, a single booking can only be charged to one card on file at a time. Members who want to change which card is used should update the default payment method before booking.
""")

RAW_DOCS["wifi-and-equipment-faq"] = ("md", """# Wifi and Equipment FAQ

## Is Wifi Included in Every Room?

Yes, building-wide wifi reaches every room, from Phonebooth to Boardroom, at no extra cost. Coverage is the same in every room regardless of size or hourly rate.

## What Is the Wifi Network Name and Password?

The network name and password are posted on a card inside each room and also shown on the booking confirmation screen. The password rotates monthly for security.

## What Common Equipment Is Available Outside the Rooms?

The building shares a printer and a water station on the ground floor, available to any member with an active booking. Room-specific equipment is listed in each room manual.

## Who Do I Contact if the Wifi Is Down?

Report a wifi outage through the booking system's support option; staff follow the internal reset procedure and typically restore the connection within a few minutes.
""")

RAW_DOCS["focus-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Focus Room Manual</title></head>
<body>
<h1>Focus Room</h1>
<p>Focus is Reservo's single-occupancy room, designed for calls and deep work
that needs a closed door. It is the smallest and least expensive room in the
building.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 1 person. The room has one desk, one chair, and a soundproofed
door. There is no window, by design, to minimize visual distraction.</p>
<h2>Equipment</h2>
<ul>
<li>27-inch monitor with HDMI input</li>
<li>Adjustable desk lamp</li>
<li>Wall outlet with two USB-C ports</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $25.00 per hour (2500 cents), the lowest rate in the building.
Pro members receive the standard 20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Focus is not soundproof against phone ringtones; members are asked to keep
devices on silent. Food is allowed but no hot meals, due to the room's small
size and lack of ventilation.</p>
</body>
</html>
""")

RAW_DOCS["studio-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Studio Room Manual</title></head>
<body>
<h1>Studio Room</h1>
<p>Studio is Reservo's small-team room, built for a working session that
needs a table and a whiteboard rather than a single desk. It sits between
Focus and Boardroom in both size and price.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 4 people. The room has a round table, four chairs, and a
wall-mounted whiteboard. A large window faces the courtyard.</p>
<h2>Equipment</h2>
<ul>
<li>43-inch monitor with HDMI and USB-C input</li>
<li>Wall-mounted whiteboard with markers</li>
<li>Conference speakerphone</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $40.00 per hour (4000 cents). Pro members receive the standard
20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Studio can be booked for up to 4 consecutive hours per reservation. The
whiteboard must be wiped clean before the next booking begins.</p>
</body>
</html>
""")

RAW_DOCS["boardroom-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Boardroom Room Manual</title></head>
<body>
<h1>Boardroom</h1>
<p>Boardroom is Reservo's largest room, reserved for formal meetings,
client presentations, and full-team gatherings. It is the only room with a
dedicated presentation screen.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 10 people. The room has a long table, ten chairs, and a
wall-mounted presentation screen at the head of the table.</p>
<h2>Equipment</h2>
<ul>
<li>75-inch presentation screen with HDMI and wireless casting</li>
<li>Conference speakerphone with ceiling microphones</li>
<li>Wall-mounted whiteboard with markers</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $80.00 per hour (8000 cents), the highest rate in the
building. Pro members receive the standard 20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Boardroom receives a full clean every evening regardless of usage,
because of its size. Food is allowed only during bookings longer than 2
hours, and must be cleared before the room is released.</p>
</body>
</html>
""")

RAW_DOCS["lounge-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Lounge Room Manual</title></head>
<body>
<h1>Lounge</h1>
<p>Lounge is Reservo's informal meeting room, built for a relaxed
conversation rather than a formal presentation. It is open-plan, with no
door separating it from the hallway.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 6 people. The room has two low sofas, a coffee table, and
extra chairs stacked against the wall for larger groups.</p>
<h2>Equipment</h2>
<ul>
<li>32-inch monitor with HDMI input</li>
<li>Bluetooth speaker</li>
<li>Coffee and tea station</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $50.00 per hour (5000 cents). Pro members receive the standard
20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Lounge is open-plan and does not require a keypad code, unlike Focus,
Phonebooth, and Boardroom. Because there is no door, members should keep
calls at conversational volume.</p>
</body>
</html>
""")

RAW_DOCS["phonebooth-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Phonebooth Room Manual</title></head>
<body>
<h1>Phonebooth</h1>
<p>Phonebooth is Reservo's smallest room, built for a single short call
rather than a working session. It is the only room designed to be used
standing up.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 1 person. The room has a narrow shelf-desk and a single stool,
with just enough space to stand and pace during a call.</p>
<h2>Equipment</h2>
<ul>
<li>Wall-mounted phone charging dock</li>
<li>Small desk fan</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $15.00 per hour (1500 cents), the lowest rate in the building
alongside its small size. Pro members receive the standard 20% discount on
every booking.</p>
<h2>House Rules</h2>
<p>Phonebooth bookings are capped at 1 hour per reservation, since the room
is designed for short calls. Rooms with a physical door -- Focus,
Phonebooth, and Boardroom -- use a keypad code that rotates weekly.</p>
</body>
</html>
""")

# operations-manual-raw.txt: deliberately DIRTY (M1 practices parsing/cleaning
# on this one). Built so that clean_text(raw) reproduces M1's exact verbatim
# cleaned output (len(raw)=1473, len(clean)=1172, header repeated 4x, 5
# hyphen-breaks: week-\nends, ope-\nning, run-\nning, min-\nutes, re-\nset).
_HEADER = "RESERVO OPERATIONS MANUAL - INTERNAL - CONFIDENTIAL"


def _insert_break(text: str, word: str, nth: int = 1) -> str:
    """Split the nth occurrence of `word` at its midpoint into 'wo-\\nrd'."""
    start = 0
    count = 0
    idx = -1
    while True:
        idx = text.find(word, start)
        if idx == -1:
            raise ValueError(f"word {word!r} not found (occurrence {nth})")
        count += 1
        if count == nth:
            break
        start = idx + 1
    mid = len(word) // 2
    return text[:idx] + word[:mid] + "-\n" + word[mid:] + text[idx + len(word):]


_body1 = ("The building opens at 07:00 and closes at 22:00 on weekdays. On weekends the "
          "building opens at 09:00 and closes at 18:00. Staff must complete a "
          "walkthrough of every room before opening to confirm no equipment was left "
          "running overnight.")
_body1 = _insert_break(_body1, "weekends", 1)
_body1 = _insert_break(_body1, "opening", 1)
_body1 = _insert_break(_body1, "running", 1)

_body2 = ("Rooms are cleaned between every booking when the gap is 30 minutes or longer. "
          "For back-to-back bookings under 30 minutes, cleaning is limited to wiping the "
          "table and checking for left-behind belongings. Boardroom receives a full "
          "clean every evening regardless of usage, because of its size.")
_body2 = _insert_break(_body2, "minutes", 1)  # only the FIRST "minutes" breaks

_body3 = ("Rooms with a physical door - Focus, Phonebooth, and Boardroom - use a keypad "
          "code that rotates weekly. Studio and Lounge are open-plan and do not require "
          "a code. Staff must update the keypad codes every Monday before 07:00 and log "
          "the change in the access log.")
# page 3 has no hyphen-break in the source

_body4 = ("If a member reports the wifi is down, staff should first check the router in "
          "the utility closet before escalating. A full reset takes approximately 3 "
          "minutes and drops every room's connection at once, so it should only be done "
          "between bookings, never during an active reservation.")
_body4 = _insert_break(_body4, "reset", 1)

RAW_DOCS["operations-manual-raw"] = ("txt", (
    _HEADER + "\n" + "Page 1 of 4" + "\n" + ("\n" * 3) +
    "Opening and Closing Procedures" + "\n\n" + _body1 + ("\n" * 10) +
    _HEADER + "\n" + "Page 2 of 4" + "\n" + ("\n" * 3) +
    "Cleaning Procedures" + "\n\n" + _body2 + ("\n" * 10) +
    _HEADER + "\n" + "Page 3 of 4" + "\n" + ("\n" * 3) +
    "Key and Access Handling" + "\n\n" + _body3 + ("\n" * 9) +
    _HEADER + "\n" + "Page 4 of 4" + "\n" + ("\n" * 3) +
    "Wifi Reset Procedure" + "\n\n" + _body4
))


# =============================================================================
# THE CANONICAL CHUNKER -- same pattern M1 teaches (module-01-parsing-and-
# chunking-documents, lessons 03-07): parse by format -> clean_text (only
# .txt) -> chunk_by_structure (structure-aware, L05) -> Chunk (L07).
# =============================================================================


@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    doc_id: str
    title: str
    section: str
    position: int
    text: str


# --- parse HTML (M1 L03) ----------------------------------------------------
WS_RE = re.compile(r"\s+")


class RoomManualParser(HTMLParser):
    """Extracts (title, [(section, text), ...]) from a room manual."""

    CAPTURE_TAGS = {"p", "li", "h1", "h2", "title"}

    def __init__(self):
        super().__init__()
        self.title = ""
        self.sections: list[tuple[str, list[str]]] = []
        self._current_section = "Overview"
        self._section_index: dict[str, int] = {}
        self._tag_stack: list[str] = []
        self._buffer: list[str] = []

    def handle_starttag(self, tag, attrs):
        self._tag_stack.append(tag)
        if tag in self.CAPTURE_TAGS:
            self._buffer = []

    def handle_endtag(self, tag):
        if self._tag_stack and self._tag_stack[-1] == tag:
            self._tag_stack.pop()
        text = "".join(self._buffer).strip()
        if tag == "title":
            self.title = text
        elif tag == "h2":
            self._current_section = text
        elif tag == "p" and text:
            self._add(text)
        elif tag == "li" and text:
            self._add(f"- {text}")

    def handle_data(self, data):
        if self._tag_stack and self._tag_stack[-1] in self.CAPTURE_TAGS:
            self._buffer.append(data)

    def _add(self, text):
        if self._current_section not in self._section_index:
            self._section_index[self._current_section] = len(self.sections)
            self.sections.append((self._current_section, []))
        idx = self._section_index[self._current_section]
        self.sections[idx][1].append(text)


def parse_html(raw_html: str) -> tuple[str, list[tuple[str, str]]]:
    parser = RoomManualParser()
    parser.feed(raw_html)
    sections = []
    for heading, paras in parser.sections:
        clean_paras = [WS_RE.sub(" ", p).strip() for p in paras]
        sep = "; " if all(p.startswith("- ") for p in clean_paras) else " "
        sections.append((heading, sep.join(clean_paras)))
    return parser.title, sections


# --- parse markdown (M1 L04) -------------------------------------------------
H2_SPLIT_RE = re.compile(r"(?m)^##\s+(.+)$")


def parse_markdown(raw_md: str) -> tuple[str, list[tuple[str, str]]]:
    lines = raw_md.strip("\n").split("\n")
    title = ""
    if lines and lines[0].startswith("# "):
        title = lines[0][2:].strip()
        lines = lines[1:]
    body = "\n".join(lines)

    parts = H2_SPLIT_RE.split(body)
    sections: list[tuple[str, str]] = []

    preamble = parts[0].strip()
    if preamble:
        sections.append(("Overview", preamble))

    for i in range(1, len(parts), 2):
        heading = parts[i].strip()
        content = parts[i + 1].strip() if i + 1 < len(parts) else ""
        content = re.sub(r"\s+", " ", content)
        sections.append((heading, content))

    return title, sections


# --- clean dirty .txt (M1 L04) -----------------------------------------------
HEADER_FOOTER_RE = re.compile(r"^RESERVO OPERATIONS MANUAL.*$\n?", re.MULTILINE)
PAGE_MARKER_RE = re.compile(r"^Page \d+ of \d+\s*$\n?", re.MULTILINE)
HYPHEN_BREAK_RE = re.compile(r"(\w)-\n(\w)")
BLANK_RUN_RE = re.compile(r"\n{2,}")
INNER_WS_RE = re.compile(r"[ \t]+")


def clean_text(raw: str) -> str:
    """Undoes the damage of a naive PDF/OCR-style text extraction."""
    text = HEADER_FOOTER_RE.sub("", raw)
    text = PAGE_MARKER_RE.sub("", text)
    text = HYPHEN_BREAK_RE.sub(r"\1\2", text)
    text = BLANK_RUN_RE.sub("\n\n", text)
    paragraphs = [p.replace("\n", " ").strip()
                  for p in text.split("\n\n") if p.strip()]
    paragraphs = [INNER_WS_RE.sub(" ", p) for p in paragraphs]
    return "\n\n".join(paragraphs)


def sectionize_cleaned(cleaned: str) -> list[tuple[str, str]]:
    """Cleaned text alternates short heading and body, one pair per old 'page'."""
    paragraphs = cleaned.split("\n\n")
    sections = []
    i = 0
    while i < len(paragraphs):
        heading = paragraphs[i]
        body = paragraphs[i + 1] if i + 1 < len(paragraphs) else ""
        sections.append((heading, body))
        i += 2
    return sections


# --- chunking strategies (M1 L05) --------------------------------------------
SENTENCE_RE = re.compile(r"(?<=[.!?])\s+(?=[A-Z(`])")


def split_sentences(text: str) -> list[str]:
    return [s.strip() for s in SENTENCE_RE.split(text) if s.strip()]


def chunk_by_sentence(text: str, max_size: int = 400) -> list[str]:
    """Packs complete sentences into chunks of up to max_size characters."""
    sentences = split_sentences(text)
    chunks: list[str] = []
    current: list[str] = []
    current_len = 0
    for sent in sentences:
        extra = len(sent) + (1 if current else 0)
        if current and current_len + extra > max_size:
            chunks.append(" ".join(current))
            current, current_len = [], 0
        current.append(sent)
        current_len += len(sent) + (1 if len(current) > 1 else 0)
    if current:
        chunks.append(" ".join(current))
    return chunks


def chunk_by_structure(
    sections: list[tuple[str, str]], max_size: int = 400
) -> list[tuple[str, str]]:
    """Respects section boundaries; only sub-splits a section past max_size."""
    out: list[tuple[str, str]] = []
    for heading, body in sections:
        if len(body) <= max_size:
            out.append((heading, body))
        else:
            for piece in chunk_by_sentence(body, max_size=max_size):
                out.append((heading, piece))
    return out


# --- metadata + ingestion (M1 L07/L08) ---------------------------------------
def ingest_document(doc_id: str, fmt: str, raw_text: str, max_size: int = 400) -> list[Chunk]:
    """Parses + (cleans) + chunks one document, attaching metadata."""
    if fmt == "md":
        title, sections = parse_markdown(raw_text)
    elif fmt == "html":
        title, sections = parse_html(raw_text)
    elif fmt == "txt":
        cleaned = clean_text(raw_text)
        title = "Operations Manual (raw, cleaned)"
        sections = sectionize_cleaned(cleaned)
    else:
        raise ValueError(f"unknown format: {fmt}")

    section_chunks = chunk_by_structure(sections, max_size=max_size)
    chunks = []
    for position, (section, text) in enumerate(section_chunks):
        chunk_id = f"{doc_id}-{position:03d}"
        chunks.append(Chunk(chunk_id, doc_id, title, section, position, text))
    return chunks


def build_corpus(max_size: int = 400) -> list[Chunk]:
    """Ingests all 13 RAW_DOCS into the single canonical list[Chunk] -- 57 chunks."""
    all_chunks: list[Chunk] = []
    for doc_id, (fmt, raw_text) in RAW_DOCS.items():
        all_chunks.extend(ingest_document(doc_id, fmt, raw_text, max_size=max_size))
    return all_chunks


if __name__ == "__main__":
    chunks = build_corpus()
    assert len(chunks) == 57, len(chunks)
    print("documents:", len(RAW_DOCS), " total chunks:", len(chunks))

What to expect (run, python3 reservo_corpus.py):

documents: 13  total chunks: 57

This block doesn't repeat anything from Module 1: it's the same RAW_DOCS, the same Chunk, the same ingest_document, the same INGESTION_DATE = "2026-01-15" — copied as is. The only difference from Lesson 08 of Module 1 is what we do with it next.

The problem, run: reingesting without control duplicates everything

With reservo_corpus.py available, put together the simplest possible version of an ingestion: create a sqlite3 table and dump every chunk as a row, without ever asking whether that row already existed. It's exactly what someone who never ran into the idempotency problem would do — and it works perfectly the first time.

import sqlite3
from reservo_corpus import build_corpus

conn = sqlite3.connect(":memory:")
conn.execute("""
    CREATE TABLE chunks_naive (
        chunk_id TEXT, doc_id TEXT, title TEXT, section TEXT,
        position INTEGER, text TEXT
    )
""")


def naive_ingest() -> None:
    """Reingests the full corpus with no idempotency check whatsoever --
    every call just re-parses everything and INSERTs every chunk again."""
    for c in build_corpus():
        conn.execute(
            "INSERT INTO chunks_naive VALUES (?, ?, ?, ?, ?, ?)",
            (c.chunk_id, c.doc_id, c.title, c.section, c.position, c.text),
        )
    conn.commit()


print("--- run 1: first ingestion ---")
naive_ingest()
total_1 = conn.execute("SELECT COUNT(*) FROM chunks_naive").fetchone()[0]
print("rows in chunks_naive:", total_1)

print("--- run 2: same corpus, no documents changed ---")
naive_ingest()
total_2 = conn.execute("SELECT COUNT(*) FROM chunks_naive").fetchone()[0]
print("rows in chunks_naive:", total_2)

print()
print("duplicated chunk_id rows (should be zero in a correct pipeline):")
dupes = conn.execute("""
    SELECT chunk_id, COUNT(*) AS n
    FROM chunks_naive
    GROUP BY chunk_id
    HAVING n > 1
    ORDER BY chunk_id
    LIMIT 3
""").fetchall()
for chunk_id, n in dupes:
    print(f"  {chunk_id}: {n} copies")

What to expect (run):

--- run 1: first ingestion ---
rows in chunks_naive: 57
--- run 2: same corpus, no documents changed ---
rows in chunks_naive: 114

duplicated chunk_id rows (should be zero in a correct pipeline):
  boardroom-room-manual-000: 2 copies
  boardroom-room-manual-001: 2 copies
  boardroom-room-manual-002: 2 copies

The first run does exactly what's expected: 57 rows, one per chunk. The second run — on the same corpus, with no document actually changed in the real world — duplicates everything: 114 rows, double the correct figure. boardroom-room-manual-000 isn't an isolated case; all 57 chunk_id values have exactly two copies each, because naive_ingest() has no way of knowing it already did this work yesterday.

Why this isn't a rare case

It's tempting to think "well, just don't run the script on the same corpus again" — but that's not a solution, it's avoiding the problem. In a real production system, ingestion fires recurrently (every night, every time a new document lands in a bucket) with no one manually checking whether something changed since last time. Three scenarios where naive_ingest() runs twice on unchanged data, all of them common:

1. A cron that runs every night. If none of the 13 documents changed in the last 24 hours, the cron still triggers ingestion — and without idempotency, it duplicates the 57 chunks that night, and the next, and the next.
2. A retry after a partial failure. If ingestion crashes halfway through (say, after writing 40 of the 57 chunks) and the system simply reruns it from the start, the first 40 chunks already written get duplicated.
3. Two instances of the same pipeline running by mistake. A duplicated deployment, a job scheduled twice by accident — any scenario where the same ingestion fires more than once against the same corpus state.

In all three cases, Reservo's actual corpus didn't change — but the chunk store ended up with double the rows it should have. Multiply that across months of daily runs and the chunk store stops being a trustworthy source of truth: it has duplicate entries, the index built on top of it (Module 2) returns repeated results, and no one can trust a COUNT(*) to know how many documents are actually ingested.

The fix isn't "run ingestion more carefully" — it's making ingestion able to recognize what it already processed, no matter how many times it's invoked. That's exactly what "idempotent" means: applying the same operation once or a hundred times produces the same final result. Lessons 03-05 build that property, starting with the smallest piece: a fingerprint per chunk.

Common mistakes

1. Thinking the problem is "forgetting not to run it twice". The point of this lesson is that in production you will run it more than once, on purpose or by accident — the fix is never human discipline ("remember not to reingest"), it's that the code itself be safe to invoke any number of times.
2. Confusing "duplicated rows" with "duplicated information". The 114 rows in chunks_naive have no new content — they're literally 57 pairs of identical rows. This is worse than corrupted data: it's correct data, repeated, with no signal at all distinguishing the original copy from the duplicate.
3. Using chunk_id with no uniqueness constraint on the table. chunks_naive declared no primary key and no UNIQUE column on chunk_id — that's why sqlite3 accepted the second batch of 57 INSERTs without complaint. Lesson 03 fixes exactly this.
4. Assuming the problem disappears if the corpus is small. With 57 chunks, 114 duplicated rows is still easy to notice with a COUNT(*). With a production corpus of tens of thousands of chunks, ingested daily for months with no idempotency, the number of duplicated rows can far exceed the number of real rows, and no one notices until the search index starts returning the same result three or four times per query.

Exercises

Exercise 1: Three runs instead of two (Easy)

Call naive_ingest() three times in a row instead of two (on a freshly created chunks_naive table). How many total rows do you expect? How many copies of no-show-policy-000 should show up? Confirm by running it.

naive_ingest()
naive_ingest()
naive_ingest()
print("after 3 runs:", conn.execute("SELECT COUNT(*) FROM chunks_naive").fetchone()[0])

count_noshow = conn.execute(
    "SELECT COUNT(*) FROM chunks_naive WHERE doc_id='no-show-policy'"
).fetchone()[0]
print("no-show-policy rows after 3 runs:", count_noshow)

after 3 runs: 171
no-show-policy rows after 3 runs: 12

Exercise 2: Counting the row excess with SQL (Medium)

Without using Python to count by hand, write a single SQL query that returns how many extra rows chunks_naive has compared to the 57 it should have — that is, the total row count minus the number of distinct chunk_id values. Run it against the table after the 3 runs from Exercise 1.

excess = conn.execute("""
    SELECT COUNT(*) - (SELECT COUNT(DISTINCT chunk_id) FROM chunks_naive)
    FROM chunks_naive
""").fetchone()[0]
print("excess rows over the correct 57 distinct chunk_ids:", excess)

distinct = conn.execute("SELECT COUNT(DISTINCT chunk_id) FROM chunks_naive").fetchone()[0]
print("distinct chunk_id:", distinct)

excess rows over the correct 57 distinct chunk_ids: 114
distinct chunk_id: 57

Exercise 3: Why isn't INSERT OR IGNORE alone enough? (Hard)

sqlite3 has an INSERT OR IGNORE variant that doesn't fail if a row violates a uniqueness constraint — it simply doesn't insert it. If you declared chunk_id as a primary key on chunks_naive and swapped every INSERT for INSERT OR IGNORE, would the 114-row problem disappear? Think about what would happen if refund-policy's text changed between run 1 and run 2, before looking at the solution.

Summary and next step

• You reproduced, with real code, this module's central problem: reingesting the same corpus with no idempotency control duplicates every chunk — from 57 rows to 114 with a single extra run, with no real document ever having changed.
• This isn't a rare lab case: a nightly cron, a retry after a partial failure, or two instances of the same pipeline running by mistake all produce exactly this situation in a real system.
• INSERT OR IGNORE on a primary key fixes the duplication but not real idempotency: it silently discards new content when a document actually did change. What's needed is a way to distinguish "already filed this, no change" from "this is different from what I filed" — the work of Lesson 03.
• reservo_corpus.py, with Module 1's full corpus and chunker, stays available unchanged for the rest of this module.

Next lesson: 03 — Content hashing for idempotency. Each chunk's fingerprint, with hashlib.sha256, and the chunk store in sqlite3 that uses it to write only what genuinely changed.

Additional resources

1. Python — sqlite3 — connect(), execute(), commit(), the basis for this lesson's chunks_naive table.
2. SQLite — INSERT — covers INSERT OR IGNORE and the other conflict-resolution variants Exercise 3 explores.
3. Wikipedia — Idempotence (computer science) — the exact property this lesson shows broken, and that the rest of the module builds.
4. production-rag-and-document-ingestion-guide — Module 1, Lesson 08 (08-mini-project-ingest-the-reservo-corpus.md): the original source of reservo_corpus.py, ingested for the first and only time before this module.