Metadata for Citation

Description

So far, every chunk you produced in lessons 05 and 06 was a loose string of text: "No-shows are charged the full amount of the booking.", nothing more. It's correct text, cut with real criteria — but if a user asks the Reservo agent "where did you get that from?", that string of text has no answer. It doesn't know which document it came from, which section of the document it was in, or what position it held relative to the other chunks from the same document. This lesson solves exactly that: it turns every chunk into an object with metadata — doc_id, title, section, position — that makes it citable, traceable, and reconstructible. It's the last piece before the mini-project: with this, Reservo's complete corpus is ready for Module 2 to index.

Connection to the module

Lessons 03 and 04 parsed the three formats; lessons 05 and 06 cut the text into chunks and reasoned about their size. This lesson joins both things into a single per-document ingestion function, and adds the one thing that was missing: each chunk's identity. Lesson 08 (the mini-project) takes exactly this function and runs it over all 13 documents of the complete corpus.

Analogy: the unlabeled meal-prep container

If you've ever prepped meals for the whole week and stored them in identical, unlabeled containers, you already know the problem: three days later, you open the freezer and have ten identical containers, with no idea which one is the lentil stew and which is the leftover tomato sauce. The content is perfectly fine — the problem is that it lost its identity the moment you stored it without writing anything down. A text chunk with no metadata is exactly that container: the text can be precise and well-cut, but if you don't know which document it came from, you can't cite it, you can't verify it, and you can't show the user where the full source is if they need more context. Labeling the container — with what it is, from which recipe, from which day — is exactly what this lesson does with every chunk.

The Chunk dataclass: the four fields you need to cite something

from dataclasses import dataclass


@dataclass(frozen=True)
class Chunk:
    chunk_id: str    # unique identifier, e.g. "cancellation-policy-002"
    doc_id: str       # which document it came from, e.g. "cancellation-policy"
    title: str        # human-readable document title, e.g. "Cancellation Policy"
    section: str      # section within the document, e.g. "How to Cancel"
    position: int      # order within the document (0, 1, 2, ...)
    text: str          # the chunk's actual text

frozen=True makes a Chunk, once created, impossible to modify — consistent with what it is: a fragment of a document that's already been processed, not something a caller should be able to accidentally mutate later in the pipeline (Module 2's index, Module 3's tool). Every field, besides text, exists for a concrete reason:

chunk_id  -> identifies THIS chunk unambiguously (needed to update/delete in M5)
doc_id    -> which source document it came from (needed to cite "according to cancellation-policy...")
title     -> the document's readable name (needed to cite something a HUMAN understands,
             not a technical identifier)
section   -> which part of the document it lived in (needed to cite with precision: not just
             "according to cancellation-policy" but "in the Pro Tier Cancellation Window section")
position  -> the order within the document (needed to reconstruct context: "show me
             the chunk before and after this one, from the same document")

Without doc_id, a chunk is anonymous text. Without section, you can say which document it came from but not which part. Without position, there's no way to ask for "the context around this chunk" — every chunk stays isolated from its neighbors, with no reconstructible order.

Worked example: from raw document to a list of Chunk

Let's join everything we built in lessons 03-06 into a single function. ingest_document takes the doc_id, the format, and the raw text, picks the right parser for the format, chunks with chunk_by_structure (lesson 05's structure-aware strategy, the one that never mixes sections), and wraps each result in a Chunk with its metadata:

def ingest_document(doc_id: str, fmt: str, raw_text: str, max_size: int = 400) -> list[Chunk]:
    """Parse + (clean) + chunk a document, attaching metadata."""
    if fmt == "md":
        title, sections = parse_markdown(raw_text)
    elif fmt == "html":
        title, sections = parse_html(raw_text)
    elif fmt == "txt":
        cleaned = clean_text(raw_text)
        title = "Operations Manual (raw, cleaned)"
        sections = sectionize_cleaned(cleaned)
    else:
        raise ValueError(f"unknown format: {fmt}")

    section_chunks = chunk_by_structure(sections, max_size=max_size)
    chunks = []
    for position, (section, text) in enumerate(section_chunks):
        chunk_id = f"{doc_id}-{position:03d}"
        chunks.append(Chunk(chunk_id, doc_id, title, section, position, text))
    return chunks

Notice the chunk_id: it's built as f"{doc_id}-{position:03d}" — doc_id first, so two chunks from different documents can never collide, and position zero-padded (003, not 3) so the IDs sort alphabetically in the same order they appear in the document. Let's run it on three documents, one of each format:

fmt, raw = "md", Path("corpus/cancellation-policy.md").read_text()
for c in ingest_document("cancellation-policy", fmt, raw):
    print(c)

What to expect:

Chunk(chunk_id='cancellation-policy-000', doc_id='cancellation-policy', title='Cancellation Policy', section='Basic Tier Cancellation Window', position=0, text='Basic members can cancel a booking up to 24 hours before the reserved start time with no penalty. Cancellations made less than 24 hours in advance forfeit the full booking amount.')
Chunk(chunk_id='cancellation-policy-001', doc_id='cancellation-policy', title='Cancellation Policy', section='Pro Tier Cancellation Window', position=1, text='Pro members get a shorter, friendlier window: cancellations up to 4 hours before the reserved start time are free of charge. This is one of the perks of the pro tier, alongside the 20% discount on hourly rates.')
Chunk(chunk_id='cancellation-policy-002', doc_id='cancellation-policy', title='Cancellation Policy', section='How to Cancel', position=2, text='Cancellations go through the same booking system used to reserve the room. There is no phone line for cancellations; the system timestamp is what determines whether the cancellation was made in time.')
Chunk(chunk_id='cancellation-policy-003', doc_id='cancellation-policy', title='Cancellation Policy', section='Related Policies', position=3, text='See `refund-policy` for what happens to the money once a cancellation is processed, and `no-show-policy` for what happens if you simply do not show up without cancelling.')

fmt, raw = "html", Path("corpus/focus-room-manual.html").read_text()
for c in ingest_document("focus-room-manual", fmt, raw):
    print(c)

What to expect:

Chunk(chunk_id='focus-room-manual-000', doc_id='focus-room-manual', title='Focus Room Manual', section='Overview', position=0, text="Focus is Reservo's single-occupancy room, designed for calls and deep work that needs a closed door. It is the smallest and least expensive room in the building.")
Chunk(chunk_id='focus-room-manual-001', doc_id='focus-room-manual', title='Focus Room Manual', section='Capacity & Layout', position=1, text='Capacity: 1 person. The room has one desk, one chair, and a soundproofed door. There is no window, by design, to minimize visual distraction.')
Chunk(chunk_id='focus-room-manual-002', doc_id='focus-room-manual', title='Focus Room Manual', section='Equipment', position=2, text='- 27-inch monitor with HDMI input; - Adjustable desk lamp; - Wall outlet with two USB-C ports; - Building-wide wifi (see wifi-and-equipment-faq)')
Chunk(chunk_id='focus-room-manual-003', doc_id='focus-room-manual', title='Focus Room Manual', section='Booking & Rate', position=3, text='Base rate: $25.00 per hour (2500 cents), the lowest rate in the building. Pro members receive the standard 20% discount on every booking.')
Chunk(chunk_id='focus-room-manual-004', doc_id='focus-room-manual', title='Focus Room Manual', section='House Rules', position=4, text="Focus is not soundproof against phone ringtones; members are asked to keep devices on silent. Food is allowed but no hot meals, due to the room's small size and lack of ventilation.")

fmt, raw = "txt", Path("corpus/operations-manual-raw.txt").read_text()
for c in ingest_document("operations-manual-raw", fmt, raw):
    print(c)

What to expect:

Chunk(chunk_id='operations-manual-raw-000', doc_id='operations-manual-raw', title='Operations Manual (raw, cleaned)', section='Opening and Closing Procedures', position=0, text='The building opens at 07:00 and closes at 22:00 on weekdays. On weekends the building opens at 09:00 and closes at 18:00. Staff must complete a walkthrough of every room before opening to confirm no equipment was left running overnight.')
Chunk(chunk_id='operations-manual-raw-001', doc_id='operations-manual-raw', title='Operations Manual (raw, cleaned)', section='Cleaning Procedures', position=1, text='Rooms are cleaned between every booking when the gap is 30 minutes or longer. For back-to-back bookings under 30 minutes, cleaning is limited to wiping the table and checking for left-behind belongings. Boardroom receives a full clean every evening regardless of usage, because of its size.')
Chunk(chunk_id='operations-manual-raw-002', doc_id='operations-manual-raw', title='Operations Manual (raw, cleaned)', section='Key and Access Handling', position=2, text='Rooms with a physical door - Focus, Phonebooth, and Boardroom - use a keypad code that rotates weekly. Studio and Lounge are open-plan and do not require a code. Staff must update the keypad codes every Monday before 07:00 and log the change in the access log.')
Chunk(chunk_id='operations-manual-raw-003', doc_id='operations-manual-raw', title='Operations Manual (raw, cleaned)', section='Wifi Reset Procedure', position=3, text="If a member reports the wifi is down, staff should first check the router in the utility closet before escalating. A full reset takes approximately 3 minutes and drops every room's connection at once, so it should only be done between bookings, never during an active reservation.")

The three formats — markdown, HTML, already-cleaned dirty text — converge on the same object shape. That uniformity is exactly what lets Module 2 index all 13 documents with the same code, regardless of which format each chunk came from.

From Chunk to citation

With the metadata in hand, formatting a citation is trivial:

def format_citation(c: Chunk) -> str:
    return f"[{c.doc_id} - {c.section}] {c.text}"

sample = ingest_document("no-show-policy", "md",
                          Path("corpus/no-show-policy.md").read_text())[1]
print(format_citation(sample))

What to expect:

[no-show-policy - What Happens on a No-Show] No-shows are charged the full amount of the booking. Unlike a late cancellation, there is no partial leniency: the no-show fee equals the entire reserved price, and it is never refunded under `refund-policy`.

That line is exactly what an agent (Module 3 onward) can show a user or cite in its answer: not just the text, but exactly where it came from, down to the section.

Why position matters: reconstructing the context around a chunk

This is where position stops being a bookkeeping detail and becomes useful. If a chunk alone isn't enough to answer well, position lets you ask for its neighbors — the previous and next chunk from the SAME document — to give the agent more context without having to return the whole document:

def neighbors(chunks: list[Chunk], chunk_id: str) -> list[Chunk]:
    """Return the requested chunk along with its previous and next neighbor, same doc_id."""
    target = next(c for c in chunks if c.chunk_id == chunk_id)
    same_doc = sorted((c for c in chunks if c.doc_id == target.doc_id), key=lambda c: c.position)
    idx = next(i for i, c in enumerate(same_doc) if c.chunk_id == chunk_id)
    return same_doc[max(0, idx - 1):min(len(same_doc), idx + 2)]


chunks = ingest_document("no-show-policy", "md", Path("corpus/no-show-policy.md").read_text())
for c in neighbors(chunks, "no-show-policy-001"):
    marker = ">>" if c.chunk_id == "no-show-policy-001" else "  "
    print(marker, c.chunk_id, "-", c.section)

What to expect:

   no-show-policy-000 - What Counts as a No-Show
>> no-show-policy-001 - What Happens on a No-Show
   no-show-policy-002 - Repeated No-Shows

If chunk [001] only got as far as saying "the full amount gets charged" but the user asked "and what exactly counts as a no-show?", neighbor [000] — retrievable only because position exists — has exactly that definition. Without position, this function would have no way of knowing which chunk is "the previous one": the chunks would be an unordered bag of text, not a reconstructible sequence.

Common mistakes

1. Building chunk_id without the doc_id prefix. If you used only position as the id ("000", "001", ...), every document in the corpus would have chunks with the same id — a guaranteed collision as soon as you have more than one document. The doc_id prefix isn't cosmetic: it's what makes chunk_id unique across the WHOLE corpus, not just within one document.

2. Using doc_id instead of title to show something to the user. doc_id ("no-show-policy") is perfect for internal logic (comparing, indexing, updating) but ugly to show a person. title ("No-Show Policy") is the one that goes into a human-facing answer. Mixing the two — showing a user doc_id, or using title as a comparison key — is a typical source of subtle bugs (two documents with the same title but a different doc_id would break any logic that compared by title).

3. Forgetting that frozen=True means you can't reassign a field after creating the Chunk. If you need to "fix" a chunk after creating it (say, normalizing its text one more time), the only way is to create a new Chunk with dataclasses.replace(chunk, text=new_text) — trying chunk.text = new_text directly raises FrozenInstanceError. It's a deliberate restriction, not an accident: it protects against accidental mutations once a chunk has already traveled into the index.

4. Assuming position is a global ID. position only makes sense WITHIN a doc_id — chunk [000] of no-show-policy and chunk [000] of refund-policy are two completely different chunks that share the position number by pure coincidence (both are the first chunk of their document). Looking up by position alone, without filtering by doc_id first, mixes different documents as if they were comparable.

Exercises

Exercise 1: Cite the right chunk (Easy)

Ingest payment-methods-faq.md with ingest_document and use format_citation to print the citation for the chunk that answers "does Reservo accept cash?" (look for the chunk whose section matches that question).

chunks = ingest_document("payment-methods-faq", "md",
                          Path("corpus/payment-methods-faq.md").read_text())
target = next(c for c in chunks if "Cash" in c.section)
print(format_citation(target))

[payment-methods-faq - Does Reservo Accept Cash?] No. All bookings, deposits, and no-show charges are processed electronically through the payment method on file.

Exercise 2: Verify there's no chunk_id collision across two documents (Medium)

Ingest cancellation-policy.md AND refund-policy.md separately, combine their chunks into a single list, and verify with code (not by eyeballing it) that no chunk_id repeats.

c1 = ingest_document("cancellation-policy", "md", Path("corpus/cancellation-policy.md").read_text())
c2 = ingest_document("refund-policy", "md", Path("corpus/refund-policy.md").read_text())
all_chunks = c1 + c2

ids = [c.chunk_id for c in all_chunks]
print("total chunks:", len(ids))
print("unique ids:", len(set(ids)))
print("no collisions:", len(ids) == len(set(ids)))

total chunks: 8
unique ids: 8
no collisions: True

Exercise 3: Does a fifth metadata field make sense? (Hard)

A colleague suggests adding a char_count: int field to the Chunk dataclass, computed as len(text), so it doesn't have to be recalculated every time it's needed. Argue whether you think it's a good field to add to the dataclass, considering that frozen=True means that field would have to stay in sync with text forever (what happens if someone creates a Chunk with a char_count that doesn't match len(text)?).

@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    doc_id: str
    title: str
    section: str
    position: int
    text: str

    @property
    def char_count(self) -> int:
        return len(self.text)

Summary and next step

• The Chunk dataclass (chunk_id, doc_id, title, section, position, text, frozen=True) turns a loose piece of text into something citable, traceable, and reconstructible.
• ingest_document joins everything from before (per-format parsing, cleanup when it applies, structure-aware chunking) into a single function that returns list[Chunk], tested on one document from each of the corpus's three formats.
• format_citation turns any Chunk into a citable line: [doc_id - section] text — and position lets you reconstruct the context around a chunk by requesting its neighbors in the same document, something impossible without that field.
• The most common mistakes are about identity: chunk_id without the doc_id prefix collides across documents; position only makes sense within a doc_id, never as a global identifier.

Next lesson: 08 — Mini-project: ingest the Reservo corpus. We run ingest_document over all 13 documents in the complete corpus, and close the module with the full list of Chunk objects with metadata — exactly what Module 2 is going to index.

Additional resources

1. Python — dataclasses — The complete reference for @dataclass, including frozen=True and dataclasses.replace().
2. Python — property — The decorator used in Exercise 3 to expose a derived value without risking it drifting out of sync with the source data.
3. Python — f-strings and formatting — The foundation for format_citation and the f"{doc_id}-{position:03d}" that builds every chunk_id.
4. advanced-rag-techniques-guide — How a production system with budget for reranking uses these exact same metadata fields (doc_id, section) to explain why a result got reordered; out of scope for this guide.