Los documentos cambian en producción

Descripción

El Módulo 1 ingirió el corpus de Reservo exactamente una vez: 13 documentos, 57 chunks, y ahí terminó la historia — la Lección 08 de ese módulo nunca volvió a ejecutar el pipeline sobre esos mismos archivos. Esta lección hace lo que ningún módulo anterior hizo: correr el pipeline de ingestión dos veces sobre el mismo corpus, sin cambiar una sola línea de texto entre una corrida y la otra, y mirar qué pasa si no hay ningún mecanismo pensado para eso.

La respuesta, adelantada: pasa algo malo. Si la ingestión simplemente inserta cada chunk que produce, sin preguntarse si ya estaba, correrla dos veces no te da el mismo corpus dos veces — te da el doble. Esta lección reproduce esa falla con números reales, antes de que las lecciones 03-07 la resuelvan pieza por pieza.

Conexión con el módulo

Esta lección reintroduce, verbatim, el corpus canónico y el chunker completo del Módulo 1 — el archivo reservo_corpus.py que el resto de este módulo importa sin volver a pegarlo. Después usa ese corpus para ejecutar, de punta a punta, el escenario que motiva todo lo que sigue: reingestar sin ningún control de idempotencia y ver el corpus duplicarse.

Analogía: el primer día sin registro

Retomando al archivista de la introducción del módulo: esta lección es su primer día en el trabajo, antes de que exista cualquier registro de huellas digitales. Le entregan la bandeja con los 13 documentos de Reservo y los archiva, uno por uno, en 57 páginas nuevas. Bien hecho — el archivo queda completo.

A la mañana siguiente, alguien vuelve a poner la misma bandeja con los mismos 13 documentos sobre su escritorio (nadie le avisó que ya los había archivado ayer). El archivista, fiel a su único procedimiento conocido —"archivar lo que está en la bandeja"—, vuelve a fotocopiar y archivar las 57 páginas. El archivo ahora tiene 114 páginas: cada documento, duplicado. Nada cambió en el mundo real —ninguna política se modificó, ningún manual se borró— pero el archivo ya no refleja la realidad: tiene el doble de lo que debería. Este es exactamente el bug que vas a reproducir en código a continuación, antes de que la Lección 03 le dé al archivista su primer registro de huellas.

El corpus y el chunker que reusas de M1 (verbatim)

Todo lo que sigue en este módulo parte de este mismo archivo, reservo_corpus.py — el corpus canónico de 13 documentos y el chunker completo de Chunk/ingest_document/build_corpus, exactamente como quedaron en el Módulo 1 (Lecciones 03-08). Ninguna lección de este módulo cambia una línea de este bloque; lo reproducimos completo aquí una única vez porque es el punto de partida de todo lo que construye el Módulo 5, y de aquí en adelante cada lección hace from reservo_corpus import ... en vez de repetirlo.

# =============================================================================
# CORPUS CANÓNICO DE RESERVO -- bloque único, auto-contenido y ejecutable.
# Python 3.14.0 + stdlib (re, html.parser, dataclasses) -- sin numpy, sin red.
# INGESTION_DATE fija (sin datetime.now(), sin random).
# =============================================================================
import re
from dataclasses import dataclass
from html.parser import HTMLParser

INGESTION_DATE = "2026-01-15"

# -----------------------------------------------------------------------
# RAW_DOCS: los 13 documentos crudos de Reservo, tal como "existen" en
# corpus/ -- 7 .md (politicas/FAQ), 5 .html (manuales de sala), 1 .txt sucio.
# -----------------------------------------------------------------------
RAW_DOCS: dict[str, tuple[str, str]] = {}  # doc_id -> (fmt, raw_text)

RAW_DOCS["cancellation-policy"] = ("md", """# Cancellation Policy

## Basic Tier Cancellation Window

Basic members can cancel a booking up to 24 hours before the reserved start time with no penalty. Cancellations made less than 24 hours in advance forfeit the full booking amount.

## Pro Tier Cancellation Window

Pro members get a shorter, friendlier window: cancellations up to 4 hours before the reserved start time are free of charge. This is one of the perks of the pro tier, alongside the 20% discount on hourly rates.

## How to Cancel

Cancellations go through the same booking system used to reserve the room. There is no phone line for cancellations; the system timestamp is what determines whether the cancellation was made in time.

## Related Policies

See `refund-policy` for what happens to the money once a cancellation is processed, and `no-show-policy` for what happens if you simply do not show up without cancelling.
""")

RAW_DOCS["no-show-policy"] = ("md", """# No-Show Policy

## What Counts as a No-Show

A no-show is a booking where the member never checks in during the reserved hours and never cancelled beforehand. This is different from a late cancellation, which is covered in `cancellation-policy`.

## What Happens on a No-Show

No-shows are charged the full amount of the booking. Unlike a late cancellation, there is no partial leniency: the no-show fee equals the entire reserved price, and it is never refunded under `refund-policy`.

## Repeated No-Shows

Members with three or more no-shows in a rolling 30-day window lose the ability to book same-day reservations; all future bookings must be made at least 24 hours in advance until the pattern clears.

## Why This Policy Exists

Rooms held for a no-show cannot be re-offered to another member during that window, so the fee reflects real lost capacity, not a punitive charge.
""")

RAW_DOCS["refund-policy"] = ("md", """# Refund Policy

## What Qualifies for a Refund

A refund applies when a booking is cancelled within the free window described in `cancellation-policy`, or when Reservo cancels a confirmed booking because of a facility issue, such as a maintenance problem or a power outage.

## Refund Amount and Timing

Eligible refunds return the full amount charged for the booking to the original payment method. Processing takes up to 5 business days once the cancellation is confirmed in the booking system.

## What Does Not Qualify for a Refund

Cancellations made outside the free window are not eligible; the booking amount is forfeited under `cancellation-policy`. No-shows are never refunded, regardless of membership tier; see `no-show-policy` for that separate case.

## How to Request a Refund

Eligible refunds are issued automatically once a qualifying cancellation is processed; members do not need to submit a separate request. Questions about a specific refund go to member support through the booking system.
""")

RAW_DOCS["booking-faq"] = ("md", """# Booking FAQ

## How Do I Book a Room?

Bookings are made through the Reservo booking system by choosing a room, a date, and a start and end time. A confirmation appears immediately, and the room is held exclusively for that window.

## Is a Deposit Required to Book?

Yes. A deposit equal to the full session amount is charged at the time of booking, through the payment method on file; see `payment-methods-faq` for what is accepted.

## Can I Book More Than One Room at a Time?

Yes, a member can hold bookings in multiple rooms at once, as long as the times do not overlap for the same member. Each room booking is billed and cancelled separately.

## How Far in Advance Can I Book a Room?

Rooms can be booked up to 60 days in advance. Same-day booking is allowed, subject to the same-day restriction described in `no-show-policy` for members with repeated no-shows.
""")

RAW_DOCS["membership-tiers-faq"] = ("md", """# Membership Tiers FAQ

## What Is the Difference Between Basic and Pro?

Basic is the default tier for every new member, with no monthly fee. Pro is a paid upgrade that adds a shorter cancellation window, see `cancellation-policy`, and a discount on every booking.

## How Much Discount Does the Pro Tier Get?

Pro members receive a 20% discount on the hourly rate of every room, applied automatically at checkout. No code or coupon is needed; the discount is tied to the membership tier on the account.

## How Do I Upgrade to Pro?

Upgrading to Pro takes effect immediately from the account settings page. The new cancellation window and the 20% discount apply starting with the very next booking made after the upgrade.

## Can I Downgrade Back to Basic?

Yes, at any time from the account settings page. Downgrading takes effect on the next booking; any booking already confirmed under Pro keeps its Pro-tier terms.
""")

RAW_DOCS["payment-methods-faq"] = ("md", """# Payment Methods FAQ

## What Payment Methods Does Reservo Accept?

Reservo accepts major credit and debit cards on file with the account. The same card charged for a booking deposit is used automatically for any no-show or late-cancellation charge.

## Does Reservo Accept Cash?

No. All bookings, deposits, and no-show charges are processed electronically through the payment method on file.

## What Happens if a Card Is Declined?

A declined card cancels the booking hold immediately; the room is released back to the schedule. The member is notified and can retry with the same card or add a different one.

## Can I Split a Payment Between Two Cards?

No, a single booking can only be charged to one card on file at a time. Members who want to change which card is used should update the default payment method before booking.
""")

RAW_DOCS["wifi-and-equipment-faq"] = ("md", """# Wifi and Equipment FAQ

## Is Wifi Included in Every Room?

Yes, building-wide wifi reaches every room, from Phonebooth to Boardroom, at no extra cost. Coverage is the same in every room regardless of size or hourly rate.

## What Is the Wifi Network Name and Password?

The network name and password are posted on a card inside each room and also shown on the booking confirmation screen. The password rotates monthly for security.

## What Common Equipment Is Available Outside the Rooms?

The building shares a printer and a water station on the ground floor, available to any member with an active booking. Room-specific equipment is listed in each room manual.

## Who Do I Contact if the Wifi Is Down?

Report a wifi outage through the booking system's support option; staff follow the internal reset procedure and typically restore the connection within a few minutes.
""")

RAW_DOCS["focus-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Focus Room Manual</title></head>
<body>
<h1>Focus Room</h1>
<p>Focus is Reservo's single-occupancy room, designed for calls and deep work
that needs a closed door. It is the smallest and least expensive room in the
building.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 1 person. The room has one desk, one chair, and a soundproofed
door. There is no window, by design, to minimize visual distraction.</p>
<h2>Equipment</h2>
<ul>
<li>27-inch monitor with HDMI input</li>
<li>Adjustable desk lamp</li>
<li>Wall outlet with two USB-C ports</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $25.00 per hour (2500 cents), the lowest rate in the building.
Pro members receive the standard 20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Focus is not soundproof against phone ringtones; members are asked to keep
devices on silent. Food is allowed but no hot meals, due to the room's small
size and lack of ventilation.</p>
</body>
</html>
""")

RAW_DOCS["studio-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Studio Room Manual</title></head>
<body>
<h1>Studio Room</h1>
<p>Studio is Reservo's small-team room, built for a working session that
needs a table and a whiteboard rather than a single desk. It sits between
Focus and Boardroom in both size and price.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 4 people. The room has a round table, four chairs, and a
wall-mounted whiteboard. A large window faces the courtyard.</p>
<h2>Equipment</h2>
<ul>
<li>43-inch monitor with HDMI and USB-C input</li>
<li>Wall-mounted whiteboard with markers</li>
<li>Conference speakerphone</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $40.00 per hour (4000 cents). Pro members receive the standard
20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Studio can be booked for up to 4 consecutive hours per reservation. The
whiteboard must be wiped clean before the next booking begins.</p>
</body>
</html>
""")

RAW_DOCS["boardroom-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Boardroom Room Manual</title></head>
<body>
<h1>Boardroom</h1>
<p>Boardroom is Reservo's largest room, reserved for formal meetings,
client presentations, and full-team gatherings. It is the only room with a
dedicated presentation screen.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 10 people. The room has a long table, ten chairs, and a
wall-mounted presentation screen at the head of the table.</p>
<h2>Equipment</h2>
<ul>
<li>75-inch presentation screen with HDMI and wireless casting</li>
<li>Conference speakerphone with ceiling microphones</li>
<li>Wall-mounted whiteboard with markers</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $80.00 per hour (8000 cents), the highest rate in the
building. Pro members receive the standard 20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Boardroom receives a full clean every evening regardless of usage,
because of its size. Food is allowed only during bookings longer than 2
hours, and must be cleared before the room is released.</p>
</body>
</html>
""")

RAW_DOCS["lounge-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Lounge Room Manual</title></head>
<body>
<h1>Lounge</h1>
<p>Lounge is Reservo's informal meeting room, built for a relaxed
conversation rather than a formal presentation. It is open-plan, with no
door separating it from the hallway.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 6 people. The room has two low sofas, a coffee table, and
extra chairs stacked against the wall for larger groups.</p>
<h2>Equipment</h2>
<ul>
<li>32-inch monitor with HDMI input</li>
<li>Bluetooth speaker</li>
<li>Coffee and tea station</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $50.00 per hour (5000 cents). Pro members receive the standard
20% discount on every booking.</p>
<h2>House Rules</h2>
<p>Lounge is open-plan and does not require a keypad code, unlike Focus,
Phonebooth, and Boardroom. Because there is no door, members should keep
calls at conversational volume.</p>
</body>
</html>
""")

RAW_DOCS["phonebooth-room-manual"] = ("html", """<!DOCTYPE html>
<html>
<head><title>Phonebooth Room Manual</title></head>
<body>
<h1>Phonebooth</h1>
<p>Phonebooth is Reservo's smallest room, built for a single short call
rather than a working session. It is the only room designed to be used
standing up.</p>
<h2>Capacity &amp; Layout</h2>
<p>Capacity: 1 person. The room has a narrow shelf-desk and a single stool,
with just enough space to stand and pace during a call.</p>
<h2>Equipment</h2>
<ul>
<li>Wall-mounted phone charging dock</li>
<li>Small desk fan</li>
<li>Building-wide wifi (see wifi-and-equipment-faq)</li>
</ul>
<h2>Booking &amp; Rate</h2>
<p>Base rate: $15.00 per hour (1500 cents), the lowest rate in the building
alongside its small size. Pro members receive the standard 20% discount on
every booking.</p>
<h2>House Rules</h2>
<p>Phonebooth bookings are capped at 1 hour per reservation, since the room
is designed for short calls. Rooms with a physical door -- Focus,
Phonebooth, and Boardroom -- use a keypad code that rotates weekly.</p>
</body>
</html>
""")

# operations-manual-raw.txt: deliberately DIRTY (M1 practices parsing/cleaning
# on this one). Built so that clean_text(raw) reproduces M1's exact verbatim
# cleaned output (len(raw)=1473, len(clean)=1172, header repeated 4x, 5
# hyphen-breaks: week-\nends, ope-\nning, run-\nning, min-\nutes, re-\nset).
_HEADER = "RESERVO OPERATIONS MANUAL - INTERNAL - CONFIDENTIAL"


def _insert_break(text: str, word: str, nth: int = 1) -> str:
    """Split the nth occurrence of `word` at its midpoint into 'wo-\\nrd'."""
    start = 0
    count = 0
    idx = -1
    while True:
        idx = text.find(word, start)
        if idx == -1:
            raise ValueError(f"word {word!r} not found (occurrence {nth})")
        count += 1
        if count == nth:
            break
        start = idx + 1
    mid = len(word) // 2
    return text[:idx] + word[:mid] + "-\n" + word[mid:] + text[idx + len(word):]


_body1 = ("The building opens at 07:00 and closes at 22:00 on weekdays. On weekends the "
          "building opens at 09:00 and closes at 18:00. Staff must complete a "
          "walkthrough of every room before opening to confirm no equipment was left "
          "running overnight.")
_body1 = _insert_break(_body1, "weekends", 1)
_body1 = _insert_break(_body1, "opening", 1)
_body1 = _insert_break(_body1, "running", 1)

_body2 = ("Rooms are cleaned between every booking when the gap is 30 minutes or longer. "
          "For back-to-back bookings under 30 minutes, cleaning is limited to wiping the "
          "table and checking for left-behind belongings. Boardroom receives a full "
          "clean every evening regardless of usage, because of its size.")
_body2 = _insert_break(_body2, "minutes", 1)  # only the FIRST "minutes" breaks

_body3 = ("Rooms with a physical door - Focus, Phonebooth, and Boardroom - use a keypad "
          "code that rotates weekly. Studio and Lounge are open-plan and do not require "
          "a code. Staff must update the keypad codes every Monday before 07:00 and log "
          "the change in the access log.")
# page 3 has no hyphen-break in the source

_body4 = ("If a member reports the wifi is down, staff should first check the router in "
          "the utility closet before escalating. A full reset takes approximately 3 "
          "minutes and drops every room's connection at once, so it should only be done "
          "between bookings, never during an active reservation.")
_body4 = _insert_break(_body4, "reset", 1)

RAW_DOCS["operations-manual-raw"] = ("txt", (
    _HEADER + "\n" + "Page 1 of 4" + "\n" + ("\n" * 3) +
    "Opening and Closing Procedures" + "\n\n" + _body1 + ("\n" * 10) +
    _HEADER + "\n" + "Page 2 of 4" + "\n" + ("\n" * 3) +
    "Cleaning Procedures" + "\n\n" + _body2 + ("\n" * 10) +
    _HEADER + "\n" + "Page 3 of 4" + "\n" + ("\n" * 3) +
    "Key and Access Handling" + "\n\n" + _body3 + ("\n" * 9) +
    _HEADER + "\n" + "Page 4 of 4" + "\n" + ("\n" * 3) +
    "Wifi Reset Procedure" + "\n\n" + _body4
))


# =============================================================================
# EL CHUNKER CANONICO -- mismo patron que M1 ensena (module-01-parsing-and-
# chunking-documents, lecciones 03-07): parse por formato -> clean_text (solo
# .txt) -> chunk_by_structure (consciente de estructura, L05) -> Chunk (L07).
# =============================================================================


@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    doc_id: str
    title: str
    section: str
    position: int
    text: str


# --- parse HTML (M1 L03) ----------------------------------------------------
WS_RE = re.compile(r"\s+")


class RoomManualParser(HTMLParser):
    """Extracts (title, [(section, text), ...]) from a room manual."""

    CAPTURE_TAGS = {"p", "li", "h1", "h2", "title"}

    def __init__(self):
        super().__init__()
        self.title = ""
        self.sections: list[tuple[str, list[str]]] = []
        self._current_section = "Overview"
        self._section_index: dict[str, int] = {}
        self._tag_stack: list[str] = []
        self._buffer: list[str] = []

    def handle_starttag(self, tag, attrs):
        self._tag_stack.append(tag)
        if tag in self.CAPTURE_TAGS:
            self._buffer = []

    def handle_endtag(self, tag):
        if self._tag_stack and self._tag_stack[-1] == tag:
            self._tag_stack.pop()
        text = "".join(self._buffer).strip()
        if tag == "title":
            self.title = text
        elif tag == "h2":
            self._current_section = text
        elif tag == "p" and text:
            self._add(text)
        elif tag == "li" and text:
            self._add(f"- {text}")

    def handle_data(self, data):
        if self._tag_stack and self._tag_stack[-1] in self.CAPTURE_TAGS:
            self._buffer.append(data)

    def _add(self, text):
        if self._current_section not in self._section_index:
            self._section_index[self._current_section] = len(self.sections)
            self.sections.append((self._current_section, []))
        idx = self._section_index[self._current_section]
        self.sections[idx][1].append(text)


def parse_html(raw_html: str) -> tuple[str, list[tuple[str, str]]]:
    parser = RoomManualParser()
    parser.feed(raw_html)
    sections = []
    for heading, paras in parser.sections:
        clean_paras = [WS_RE.sub(" ", p).strip() for p in paras]
        sep = "; " if all(p.startswith("- ") for p in clean_paras) else " "
        sections.append((heading, sep.join(clean_paras)))
    return parser.title, sections


# --- parse markdown (M1 L04) -------------------------------------------------
H2_SPLIT_RE = re.compile(r"(?m)^##\s+(.+)$")


def parse_markdown(raw_md: str) -> tuple[str, list[tuple[str, str]]]:
    lines = raw_md.strip("\n").split("\n")
    title = ""
    if lines and lines[0].startswith("# "):
        title = lines[0][2:].strip()
        lines = lines[1:]
    body = "\n".join(lines)

    parts = H2_SPLIT_RE.split(body)
    sections: list[tuple[str, str]] = []

    preamble = parts[0].strip()
    if preamble:
        sections.append(("Overview", preamble))

    for i in range(1, len(parts), 2):
        heading = parts[i].strip()
        content = parts[i + 1].strip() if i + 1 < len(parts) else ""
        content = re.sub(r"\s+", " ", content)
        sections.append((heading, content))

    return title, sections


# --- clean dirty .txt (M1 L04) -----------------------------------------------
HEADER_FOOTER_RE = re.compile(r"^RESERVO OPERATIONS MANUAL.*$\n?", re.MULTILINE)
PAGE_MARKER_RE = re.compile(r"^Page \d+ of \d+\s*$\n?", re.MULTILINE)
HYPHEN_BREAK_RE = re.compile(r"(\w)-\n(\w)")
BLANK_RUN_RE = re.compile(r"\n{2,}")
INNER_WS_RE = re.compile(r"[ \t]+")


def clean_text(raw: str) -> str:
    """Undoes the damage of a naive PDF/OCR-style text extraction."""
    text = HEADER_FOOTER_RE.sub("", raw)
    text = PAGE_MARKER_RE.sub("", text)
    text = HYPHEN_BREAK_RE.sub(r"\1\2", text)
    text = BLANK_RUN_RE.sub("\n\n", text)
    paragraphs = [p.replace("\n", " ").strip()
                  for p in text.split("\n\n") if p.strip()]
    paragraphs = [INNER_WS_RE.sub(" ", p) for p in paragraphs]
    return "\n\n".join(paragraphs)


def sectionize_cleaned(cleaned: str) -> list[tuple[str, str]]:
    """Cleaned text alternates short heading and body, one pair per old 'page'."""
    paragraphs = cleaned.split("\n\n")
    sections = []
    i = 0
    while i < len(paragraphs):
        heading = paragraphs[i]
        body = paragraphs[i + 1] if i + 1 < len(paragraphs) else ""
        sections.append((heading, body))
        i += 2
    return sections


# --- chunking strategies (M1 L05) --------------------------------------------
SENTENCE_RE = re.compile(r"(?<=[.!?])\s+(?=[A-Z(`])")


def split_sentences(text: str) -> list[str]:
    return [s.strip() for s in SENTENCE_RE.split(text) if s.strip()]


def chunk_by_sentence(text: str, max_size: int = 400) -> list[str]:
    """Packs complete sentences into chunks of up to max_size characters."""
    sentences = split_sentences(text)
    chunks: list[str] = []
    current: list[str] = []
    current_len = 0
    for sent in sentences:
        extra = len(sent) + (1 if current else 0)
        if current and current_len + extra > max_size:
            chunks.append(" ".join(current))
            current, current_len = [], 0
        current.append(sent)
        current_len += len(sent) + (1 if len(current) > 1 else 0)
    if current:
        chunks.append(" ".join(current))
    return chunks


def chunk_by_structure(
    sections: list[tuple[str, str]], max_size: int = 400
) -> list[tuple[str, str]]:
    """Respects section boundaries; only sub-splits a section past max_size."""
    out: list[tuple[str, str]] = []
    for heading, body in sections:
        if len(body) <= max_size:
            out.append((heading, body))
        else:
            for piece in chunk_by_sentence(body, max_size=max_size):
                out.append((heading, piece))
    return out


# --- metadata + ingestion (M1 L07/L08) ---------------------------------------
def ingest_document(doc_id: str, fmt: str, raw_text: str, max_size: int = 400) -> list[Chunk]:
    """Parses + (cleans) + chunks one document, attaching metadata."""
    if fmt == "md":
        title, sections = parse_markdown(raw_text)
    elif fmt == "html":
        title, sections = parse_html(raw_text)
    elif fmt == "txt":
        cleaned = clean_text(raw_text)
        title = "Operations Manual (raw, cleaned)"
        sections = sectionize_cleaned(cleaned)
    else:
        raise ValueError(f"unknown format: {fmt}")

    section_chunks = chunk_by_structure(sections, max_size=max_size)
    chunks = []
    for position, (section, text) in enumerate(section_chunks):
        chunk_id = f"{doc_id}-{position:03d}"
        chunks.append(Chunk(chunk_id, doc_id, title, section, position, text))
    return chunks


def build_corpus(max_size: int = 400) -> list[Chunk]:
    """Ingests all 13 RAW_DOCS into the single canonical list[Chunk] -- 57 chunks."""
    all_chunks: list[Chunk] = []
    for doc_id, (fmt, raw_text) in RAW_DOCS.items():
        all_chunks.extend(ingest_document(doc_id, fmt, raw_text, max_size=max_size))
    return all_chunks


if __name__ == "__main__":
    chunks = build_corpus()
    assert len(chunks) == 57, len(chunks)
    print("documentos:", len(RAW_DOCS), " chunks totales:", len(chunks))

Qué esperar (ejecutado, python3 reservo_corpus.py):

documentos: 13  chunks totales: 57

Este bloque no repite nada del Módulo 1: es el mismo RAW_DOCS, el mismo Chunk, el mismo ingest_document, la misma INGESTION_DATE = "2026-01-15" — copiado tal cual. La única diferencia con la Lección 08 del Módulo 1 es lo que hacemos con él a continuación.

El problema, ejecutado: reingestar sin control duplica todo

Con reservo_corpus.py disponible, arma la versión más simple posible de una ingestión: crear una tabla sqlite3 y volcar cada chunk como una fila, sin preguntarte nunca si esa fila ya existía. Es exactamente lo que haría alguien que nunca se topó con el problema de la idempotencia — funciona perfecto la primera vez.

import sqlite3
from reservo_corpus import build_corpus

conn = sqlite3.connect(":memory:")
conn.execute("""
    CREATE TABLE chunks_naive (
        chunk_id TEXT, doc_id TEXT, title TEXT, section TEXT,
        position INTEGER, text TEXT
    )
""")


def naive_ingest() -> None:
    """Reingests the full corpus with no idempotency check whatsoever --
    every call just re-parses everything and INSERTs every chunk again."""
    for c in build_corpus():
        conn.execute(
            "INSERT INTO chunks_naive VALUES (?, ?, ?, ?, ?, ?)",
            (c.chunk_id, c.doc_id, c.title, c.section, c.position, c.text),
        )
    conn.commit()


print("--- run 1: first ingestion ---")
naive_ingest()
total_1 = conn.execute("SELECT COUNT(*) FROM chunks_naive").fetchone()[0]
print("rows in chunks_naive:", total_1)

print("--- run 2: same corpus, no documents changed ---")
naive_ingest()
total_2 = conn.execute("SELECT COUNT(*) FROM chunks_naive").fetchone()[0]
print("rows in chunks_naive:", total_2)

print()
print("duplicated chunk_id rows (should be zero in a correct pipeline):")
dupes = conn.execute("""
    SELECT chunk_id, COUNT(*) AS n
    FROM chunks_naive
    GROUP BY chunk_id
    HAVING n > 1
    ORDER BY chunk_id
    LIMIT 3
""").fetchall()
for chunk_id, n in dupes:
    print(f"  {chunk_id}: {n} copies")

Qué esperar (ejecutado):

--- run 1: first ingestion ---
rows in chunks_naive: 57
--- run 2: same corpus, no documents changed ---
rows in chunks_naive: 114

duplicated chunk_id rows (should be zero in a correct pipeline):
  boardroom-room-manual-000: 2 copies
  boardroom-room-manual-001: 2 copies
  boardroom-room-manual-002: 2 copies

La primera corrida hace exactamente lo que se espera: 57 filas, una por chunk. La segunda corrida —sobre el mismo corpus, sin que ningún documento haya cambiado en el mundo real— duplica todo: 114 filas, el doble de la cifra correcta. boardroom-room-manual-000 no es un caso aislado; los 57 chunk_id tienen exactamente dos copias cada uno, porque naive_ingest() no tiene ninguna forma de saber que ya había hecho este trabajo ayer.

Por qué esto no es un caso raro

Es tentador pensar "bueno, no vuelvas a correr el script sobre el mismo corpus" — pero eso no es una solución, es evitar el problema. En un sistema de producción real, la ingestión se dispara de forma recurrente (cada noche, cada vez que se sube un documento nuevo a un bucket) sin que nadie verifique a mano si algo cambió desde la última vez. Tres escenarios donde naive_ingest() corre dos veces sobre datos sin cambios, todos comunes:

1. Un cron que corre cada noche. Si de los 13 documentos ninguno cambió en las últimas 24 horas, el cron igual dispara la ingestión — y sin idempotencia, duplica los 57 chunks esa noche, y la siguiente, y la siguiente.
2. Un reintento después de un fallo parcial. Si la ingestión se cae a mitad de camino (por ejemplo, después de escribir 40 de los 57 chunks) y el sistema simplemente la vuelve a correr desde el principio, los primeros 40 chunks ya escritos se duplican.
3. Dos instancias del mismo pipeline corriendo por error. Un despliegue duplicado, un job programado dos veces por accidente — cualquier escenario donde la misma ingestión se dispara más de una vez sobre el mismo estado del corpus.

En los tres casos, el corpus real de Reservo no cambió — pero el chunk store terminó con el doble de filas de las que debería tener. Multiplica esto por meses de corridas diarias y el chunk store deja de ser una fuente confiable de verdad: tiene entradas duplicadas, el índice que se construye sobre él (Módulo 2) devuelve resultados repetidos, y nadie puede confiar en un COUNT(*) para saber cuántos documentos hay realmente ingeridos.

La solución no es "correr la ingestión con más cuidado" — es que la ingestión sepa reconocer lo que ya procesó, sin importar cuántas veces se la invoque. Eso es exactamente lo que la palabra "idempotente" significa: aplicar la misma operación una vez o cien veces produce el mismo resultado final. Las Lecciones 03-05 construyen esa propiedad, empezando por la pieza más pequeña: una huella digital por chunk.

Errores comunes

1. Pensar que el problema es "olvidarse de no correrlo dos veces". El punto de esta lección es que en producción vas a correrlo más de una vez, a propósito o por accidente — la solución nunca es disciplina humana ("acuérdate de no reingestar"), es que el código mismo sea seguro de invocar cualquier cantidad de veces.
2. Confundir "duplicar filas" con "duplicar información". Las 114 filas de chunks_naive no tienen contenido nuevo — son literalmente 57 pares de filas idénticas. Esto es peor que tener datos corruptos: es tener datos correctos, repetidos, sin ninguna señal que distinga la copia original de la duplicada.
3. Usar chunk_id sin ninguna restricción de unicidad en la tabla. chunks_naive no declaró ninguna clave primaria ni columna UNIQUE sobre chunk_id — por eso sqlite3 aceptó la segunda tanda de 57 INSERT sin protestar. La Lección 03 corrige exactamente esto.
4. Suponer que el problema desaparece si el corpus es pequeño. Con 57 chunks, 114 filas duplicadas todavía es fácil de notar con un COUNT(*). Con un corpus de producción de decenas de miles de chunks, ingerido diariamente durante meses sin idempotencia, la cantidad de filas duplicadas puede superar ampliamente la cantidad de filas reales, y nadie lo nota hasta que el índice de búsqueda empieza a devolver el mismo resultado tres o cuatro veces por query.

Ejercicios

Ejercicio 1: Tres corridas en vez de dos (Fácil)

Llama a naive_ingest() tres veces seguidas en vez de dos (sobre una tabla chunks_naive recién creada). ¿Cuántas filas totales esperas? ¿Cuántas copias de no-show-policy-000 deberían aparecer? Confirma ejecutando.

naive_ingest()
naive_ingest()
naive_ingest()
print("after 3 runs:", conn.execute("SELECT COUNT(*) FROM chunks_naive").fetchone()[0])

count_noshow = conn.execute(
    "SELECT COUNT(*) FROM chunks_naive WHERE doc_id='no-show-policy'"
).fetchone()[0]
print("no-show-policy rows after 3 runs:", count_noshow)

after 3 runs: 171
no-show-policy rows after 3 runs: 12

Ejercicio 2: Contar el exceso de filas con SQL (Medio)

Sin usar Python para contar a mano, escribe una única consulta SQL que devuelva cuántas filas de más tiene chunks_naive respecto de las 57 que debería tener — es decir, el total de filas menos la cantidad de chunk_id distintos. Ejecútala sobre la tabla después de las 3 corridas del Ejercicio 1.

excess = conn.execute("""
    SELECT COUNT(*) - (SELECT COUNT(DISTINCT chunk_id) FROM chunks_naive)
    FROM chunks_naive
""").fetchone()[0]
print("excess rows over the correct 57 distinct chunk_ids:", excess)

distinct = conn.execute("SELECT COUNT(DISTINCT chunk_id) FROM chunks_naive").fetchone()[0]
print("distinct chunk_id:", distinct)

excess rows over the correct 57 distinct chunk_ids: 114
distinct chunk_id: 57

Ejercicio 3: ¿Por qué no alcanza con INSERT OR IGNORE sin más? (Difícil)

sqlite3 tiene una variante INSERT OR IGNORE que no falla si una fila viola una restricción de unicidad — simplemente no la inserta. Si declararas chunk_id como clave primaria en chunks_naive y cambiaras cada INSERT por INSERT OR IGNORE, ¿el problema de las 114 filas desaparecería? Piensa qué pasaría si refund-policy cambia de texto entre la corrida 1 y la corrida 2, antes de mirar la solución.

Resumen y siguiente paso

• Reprodujiste, con código real, el problema central de este módulo: reingestar el mismo corpus sin ningún control de idempotencia duplica cada chunk — de 57 filas a 114 con una sola corrida extra, sin que ningún documento real haya cambiado.
• Este no es un caso raro de laboratorio: un cron nocturno, un reintento después de un fallo parcial, o dos instancias del mismo pipeline corriendo por error producen exactamente esta situación en un sistema real.
• INSERT OR IGNORE sobre una clave primaria resuelve la duplicación pero no la idempotencia real: descarta contenido nuevo sin avisar cuando un documento sí cambió. Lo que hace falta es una forma de distinguir "ya archivé esto, sin cambios" de "esto es distinto a lo que archivé" — el trabajo de la Lección 03.
• reservo_corpus.py, con el corpus y el chunker completo del Módulo 1, queda disponible sin cambios para el resto de este módulo.

Siguiente lección: 03 — Hashing de contenido para idempotencia. La huella digital de cada chunk, con hashlib.sha256, y el chunk store en sqlite3 que la usa para escribir solo lo que genuinamente cambió.

Recursos adicionales

1. Python — sqlite3 — connect(), execute(), commit(), la base de la tabla chunks_naive de esta lección.
2. SQLite — INSERT — incluye INSERT OR IGNORE y las demás variantes de resolución de conflictos que el Ejercicio 3 explora.
3. Wikipedia — Idempotencia (ciencias de la computación) — la propiedad exacta que esta lección muestra rota, y que el resto del módulo construye.
4. production-rag-and-document-ingestion-guide — Módulo 1, Lección 08 (08-mini-project-ingest-the-reservo-corpus.md): la fuente original de reservo_corpus.py, ingerido por primera y única vez antes de este módulo.