Content hashing for idempotency

Description

Lesson 02 showed the problem with real numbers: reingesting with no control at all duplicates every chunk, run after run. This lesson builds the first piece of the fix — each chunk's fingerprint — and uses it to write a chunk store in sqlite3 that knows, before writing any row, whether that exact content is already saved.

The core idea fits in one sentence: a chunk with the same content always produces the same hash, no matter when or how many times you compute it. That property — determinism — is what turns a hash into a trustworthy content identifier, and what lets the chunk store answer, unambiguously, the question "do I already have this stored?".

Connection to the module

This lesson directly solves the problem Lesson 02 left open: it replaces the blind INSERT of naive_ingest() with an upsert_chunk() that first asks, using the hash, whether anything needs to be written at all. The chunk store built here (create_chunk_store, the chunks table with its content_hash column) is the foundation on top of which Lesson 04 adds document-level detection.

Analogy: the fingerprint, not the name

Back to the clerk from earlier lessons: until now, when a page arrived, the only question they knew how to ask was "do I already have a page with this same filename?" — and not even that: naive_ingest() from Lesson 02 didn't ask any question at all, it just filed straight away. This lesson hands the clerk a magnifying glass and a fingerprint notebook. Before filing any page, they scan it end to end and compute a short number that summarizes exactly that content — changing a single comma on the page changes the number completely. If that exact fingerprint is already in the notebook, there's no need to file anything new: the page they have is identical, byte for byte, to the one that arrived. If the fingerprint is different — even if the document's name is the same one as always — something changed, and the new version does need to be filed.

That magnifying glass is hashlib.sha256. That notebook is the content_hash column of the chunks table.

content_hash: a chunk's fingerprint

hashlib.sha256 takes any sequence of bytes and produces a 256-bit digest (32 bytes, represented as 64 hexadecimal characters with .hexdigest()). Two properties make it perfect for this:

1. Deterministic. The same input content always produces exactly the same hash — today, tomorrow, on a different machine, it doesn't matter. There's no random or clock-dependent component.
2. Sensitive to any change. Changing a single character of the content produces a completely different hash (the so-called "avalanche effect") — there's no way for a small change in the text to produce a small change in the hash; the new hash bears no resemblance to the old one.

import hashlib
from reservo_corpus import Chunk


def content_hash(chunk: Chunk) -> str:
    """A stable fingerprint of a chunk's content: same content -> same hash,
    always, on any machine, on any run, forever. Nothing time-based goes in."""
    payload = f"{chunk.doc_id}\x1f{chunk.section}\x1f{chunk.position}\x1f{chunk.text}"
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

The hash isn't computed over chunk.text alone — it's computed over doc_id, section, position, and text together, separated by \x1f (the "unit separator" control character, designed exactly for joining fields without risking one of the real values containing the separator and producing an ambiguous payload). The reason: two chunks with the same text but from different documents, or the same text in a different section, should be treated as distinct identities by the store — Exercise 3 in this lesson explores what would happen if we hashed only text.

from reservo_corpus import build_corpus

chunks = build_corpus()
sample = next(c for c in chunks if c.chunk_id == "no-show-policy-001")

h1 = content_hash(sample)
h2 = content_hash(sample)
print("chunk_id:", sample.chunk_id)
print("content_hash:", h1)
print("length:", len(h1), "hex characters")
print("same hash computed twice:", h1 == h2)

other = next(c for c in chunks if c.chunk_id == "refund-policy-001")
print("a different chunk has a different hash:", content_hash(other) != h1)

What to expect (run):

chunk_id: no-show-policy-001
content_hash: 61f4991efe35cd7375e2bd07bc1ed837c7196e68bb3843e82053e4904c629d0b
length: 64 hex characters
same hash computed twice: True
a different chunk has a different hash: True

no-show-policy-001 — the chunk that reads, verbatim, "No-shows are charged the full amount of the booking... the no-show fee equals the entire reserved price" — always produces that same 64-character hash. Computed once or a thousand times, today or a year from now, the result is identical as long as the chunk's content doesn't change by even one character. That's exactly the property "detecting changes" needs: if the stored hash matches the freshly computed hash, the content is — with total practical certainty — the same.

The chunk store: a sqlite3 table with content_hash

The store needs, for every chunk, the same six fields Chunk already has (chunk_id, doc_id, title, section, position, text) plus two new ones: content_hash (the fingerprint that row was saved with) and ingested_at (when it was written — always using INGESTION_DATE, never datetime.now()). chunk_id is the primary key: by design of ingest_document (Module 1), every chunk_id identifies a unique position within a document (f"{doc_id}-{i:03d}"), so there can't be two legitimate rows with the same chunk_id.

import sqlite3


def create_chunk_store(path: str) -> sqlite3.Connection:
    conn = sqlite3.connect(path)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS chunks (
            chunk_id TEXT PRIMARY KEY,
            doc_id TEXT NOT NULL,
            title TEXT NOT NULL,
            section TEXT NOT NULL,
            position INTEGER NOT NULL,
            text TEXT NOT NULL,
            content_hash TEXT NOT NULL,
            ingested_at TEXT NOT NULL
        )
    """)
    conn.commit()
    return conn

path accepts any file path, but also the special value ":memory:" — a database that lives only in RAM, never touches disk, and disappears when the process ends. This entire module uses ":memory:" because the chunk store, here, is a self-contained, disposable exercise; in a real production system that path would point to a persistent file or a database server, but the logic above wouldn't change by a single line.

PRIMARY KEY on chunk_id is what the chunks_naive table from Lesson 02 didn't have — it's the structural constraint that makes it impossible, at the database level, for two rows to share the same chunk_id. But a primary key alone only prevents exact duplicates of the whole row if you attempt a plain INSERT (it would fail with an error); what's needed is deciding what to do when a row with that chunk_id already exists: ignore it if the content is the same, overwrite it if it changed. That's exactly what upsert_chunk does.

upsert_chunk: write only when needed

from reservo_corpus import INGESTION_DATE


def upsert_chunk(conn: sqlite3.Connection, chunk: Chunk) -> bool:
    """Writes `chunk` only if it's new or its content_hash changed since the
    last write. Returns True if a write happened, False if it was a no-op."""
    h = content_hash(chunk)
    row = conn.execute(
        "SELECT content_hash FROM chunks WHERE chunk_id = ?", (chunk.chunk_id,)
    ).fetchone()
    if row is not None and row[0] == h:
        return False  # identical content already stored -- idempotent no-op

    conn.execute("""
        INSERT INTO chunks (chunk_id, doc_id, title, section, position, text, content_hash, ingested_at)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?)
        ON CONFLICT(chunk_id) DO UPDATE SET
            title=excluded.title, section=excluded.section, position=excluded.position,
            text=excluded.text, content_hash=excluded.content_hash, ingested_at=excluded.ingested_at
    """, (chunk.chunk_id, chunk.doc_id, chunk.title, chunk.section, chunk.position,
          chunk.text, h, INGESTION_DATE))
    return True

Three steps, in order:

1. Compute the hash of the chunk you want to write. This is the hash of the new content, not yet compared to anything.
2. Look for a row with that chunk_id, and if there is one, compare its stored hash against the new one. If they match, no INSERT or UPDATE is needed at all — the store already has exactly this content, and touching the row would be work (and a disk write) with no benefit.
3. If they don't match (or the row didn't exist), write. INSERT ... ON CONFLICT(chunk_id) DO UPDATE SET ... is SQLite's upsert clause: it tries to insert a new row, and if one with that primary key already exists, instead of failing, it updates its columns with the new values. It covers both cases — new chunk and modified chunk — with a single statement.

if __name__ == "__main__":
    chunks = build_corpus()
    conn = create_chunk_store(":memory:")

    writes_run1 = sum(upsert_chunk(conn, c) for c in chunks)
    total_run1 = conn.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
    print(f"run 1: {writes_run1} writes, {total_run1} rows in the store")

    writes_run2 = sum(upsert_chunk(conn, c) for c in chunks)
    total_run2 = conn.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
    print(f"run 2: {writes_run2} writes, {total_run2} rows in the store")

What to expect (run):

run 1: 57 writes, 57 rows in the store
run 2: 0 writes, 57 rows in the store

This is the result Lesson 02 couldn't get: run 2, on the same corpus, writes not a single row — upsert_chunk computed the hash of each of the 57 chunks, found it matched exactly what was already stored, and touched nothing — and the total stays at 57, not 114. sum(upsert_chunk(conn, c) for c in chunks) sums True/False as 1/0, so that number is literally the count of rows that genuinely changed.

Common mistakes

1. Hashing the whole chunk as a Python object (hash(chunk)), not its content with hashlib. Python's builtin hash() function is not deterministic across processes for several types (for security, Python randomizes the hash of strings between different interpreter runs, unless you fix PYTHONHASHSEED). A content identifier for a production system needs to be stable across processes and machines — that's why hashlib.sha256, never hash().
2. Comparing the new hash against the previous run's hash held in memory, instead of against what's stored in the table. If the process restarts (the normal case: ingestion runs as a fresh job each time, not as a process that lives forever), any in-memory state from the previous run disappears. The comparison always has to be against the persistent store (SELECT content_hash FROM chunks WHERE chunk_id = ?), never against a variable from the current run.
3. Using a plain INSERT instead of INSERT ... ON CONFLICT DO UPDATE. A plain INSERT on a primary key that already exists fails with sqlite3.IntegrityError — it doesn't update the row. Without the ON CONFLICT clause, you'd have to wrap every INSERT in a try/except and do a manual UPDATE in the except; the upsert does both in a single atomic statement.
4. Forgetting that upsert_chunk returning False isn't an error — it's valuable information. It's tempting to treat any False as "something went wrong". Here it means exactly the opposite: the store was already correct, and doing nothing was the right decision. Counting how many upsert_chunk calls returned True in a run (as writes_run1/writes_run2 do above) is, in fact, the most useful metric for knowing how much real work an ingestion run did.

Exercises

Exercise 1: Another chunk's hash (Easy)

Compute the content_hash of chunk payment-methods-faq-001 (the "Does Reservo Accept Cash?" section). Confirm it's different from no-show-policy-001's hash seen in the worked example, and that computing it twice gives the same result.

sample = next(c for c in chunks if c.chunk_id == "payment-methods-faq-001")
h = content_hash(sample)
print("chunk_id:", sample.chunk_id)
print("section:", sample.section)
print("content_hash:", h)
print("same hash twice:", h == content_hash(sample))

chunk_id: payment-methods-faq-001
section: Does Reservo Accept Cash?
content_hash: fc3bb46622971a8a8e2285a5346e8e2df5288f86120aa0beda320c4078a45a67
same hash twice: True

Exercise 2: One character changes everything (Medium)

Using dataclasses.replace, create a copy of payment-methods-faq-001 with a single change: the last character of text (a period .) replaced with an exclamation mark !. Compute the copy's content_hash and confirm it bears no resemblance to the original, beyond that single changed character.

from dataclasses import replace

sample = next(c for c in chunks if c.chunk_id == "payment-methods-faq-001")
mutated = replace(sample, text=sample.text[:-1] + "!")

print("original last char:", repr(sample.text[-1]))
print("mutated last char:", repr(mutated.text[-1]))
print("original hash:", content_hash(sample))
print("mutated hash: ", content_hash(mutated))
print("hashes differ:", content_hash(sample) != content_hash(mutated))

original last char: '.'
mutated last char: '!'
original hash: fc3bb46622971a8a8e2285a5346e8e2df5288f86120aa0beda320c4078a45a67
mutated hash:  16b2fd0d73732c9fb4df5c642d2b0f96f2b3c3100b5761508136ae7381ca86c5
hashes differ: True

Exercise 3: Why not hash only text? (Hard)

content_hash builds its payload from doc_id, section, position, and text, not just text. Imagine two hypothetical chunks, from different documents, that by pure coincidence had exactly the same text (for example, two room manuals that happen to share the exact sentence "Building-wide wifi is included at no extra cost." in their equipment section). If content_hash computed the hash over text alone, what problem would show up when saving them to the chunk store? Think through the answer before looking at the solution, then confirm with code that no pair of identical texts exists in the real 57-chunk corpus.

from collections import Counter

texts = Counter(c.text for c in chunks)
duplicated_texts = [text for text, n in texts.items() if n > 1]
print("identical text pairs in the real corpus:", len(duplicated_texts))

identical text pairs in the real corpus: 0

Summary and next step

• content_hash(chunk) uses hashlib.sha256 over doc_id, section, position, and text to produce a deterministic 64-character hexadecimal fingerprint: same content, same hash, always.
• The chunk store is a sqlite3 table (chunks, with chunk_id as primary key and a content_hash column) created by create_chunk_store.
• upsert_chunk compares the new hash against the stored hash before writing anything: if they match, it does nothing (idempotent); if not, it uses INSERT ... ON CONFLICT DO UPDATE to write in a single statement, whether the chunk was new or already existed.
• Run twice on the full, unchanged corpus: the first run writes 57 rows, the second writes zero — exactly Lesson 02's problem, solved.

What's still missing: upsert_chunk works chunk by chunk, but efficient reingestion needs to know, at the document level, which of the 13 documents changed — so the ones that didn't don't have to be reparsed and rechunked. That's exactly the next lesson's job.

Next lesson: 04 — Detecting new, modified, and deleted documents. A documents table holding the hash of each raw text, and a function that classifies every doc_id without reparsing anything that didn't change.

Additional resources

1. Python — hashlib — sha256(), .hexdigest(), and why the builtin hash() doesn't work for this (randomized string hashing across processes).
2. Python — dataclasses.replace — the function used in Exercise 2 to create a copy of a Chunk with a single field modified.
3. SQLite — Upsert (INSERT ... ON CONFLICT) — the exact clause behind upsert_chunk.
4. Python — sqlite3 — connect(":memory:"), PRIMARY KEY, and the rest of the API used in this lesson's chunk store.