Module 4: The System's Data Model

7. Idempotent RAG ingestion

Next →

Description

By the end of this lesson you'll be able to apply this module's deduplication pattern to new territory: document ingestion for RAG. You're going to understand why re-embedding a document you already processed is a problem —it costs compute and dirties the vector store with copies— and how to avoid it with the same dedup store you already know, changing only the key: instead of an order_id, a hash of the file's content. You're going to see what an embedding and a vector store are in simple terms, how the Starter Kit gives you Qdrant and Ollama to do all of this locally and at zero cost, and why the content hash (not the file name) is the right key for deciding whether a document changed or is the same one as always.

This matters for two reasons. The practical one: embedding documents costs —compute time, and in many setups, money per model call—, and doing it again over something that didn't change is pure waste; worse, if you insert the same chunks into the vector store twice, your searches start returning duplicates and answer quality drops. The conceptual one, which is the most valuable: this lesson demonstrates that what you learned isn't a charge-processing trick. It's a general pattern —check before acting on a costly effect— serving anywhere repeating causes harm. The charge and the embedding are the same shape with different skin.

Connection to the module: this lesson takes lesson 5's dedup store and changes its domain. The uniqueness constraint, ON CONFLICT DO NOTHING, "act only if you inserted": all identical. The only thing that changes is what goes in the key column —a content hash instead of an order identifier— and what the "effect" you're protecting is —embedding and inserting into Qdrant instead of creating a charge—. It uses Qdrant and Ollama, the two Starter Kit services you met in lesson 6. And it closes the module by demonstrating the pattern's generality, right before lesson 8 consolidates it into a project.

Photocopying what's already filed

Let's start with the analogy, because it makes the waste obvious.

Imagine your job is feeding a filing cabinet with documents. Every document that arrives, you read it, pull out a summary card —so you can find it later by topic— and file it in the cabinet. It's useful work: thanks to those cards, anyone can search "return policies" and find the right document in seconds.

Now imagine that, every morning, someone hands you yesterday's same stack of documents, plus a couple new ones. If you process the whole stack without checking, you re-read, re-summarize, and re-file documents that were already filed. Two problems come from that. One: you spent hours pulling out cards that already existed —work thrown in the trash—. Two, and worse: now the cabinet has two cards for the same document, and when someone searches "return policies," they're going to get duplicates, confusing them about which one is correct.

RAG ingestion is exactly that filing cabinet. Every document —a Cumbre product sheet, a supplier's catalog, a shipping policy— gets processed so a support agent can search it by meaning. And if you reprocess documents that were already there, you waste extra compute and fill the vector store with duplicate chunks that degrade searches. This lesson's job is to give the cabinet a doorman saying: "this document is already filed, skip it; this one's new, let it through." That doorman is the module's dedup store, with a different key.

What embedding is and what a vector store is, simply

Before deduplicating, it's worth understanding what the "costly effect" we're protecting is, because it's probably new. Let's go with two analogy-driven definitions.

An embedding is the translation of a text's meaning into a list of numbers. A model reads a chunk —"our policy allows returns within 30 days"— and produces a long list of numbers (a "vector") capturing what that text is about. The magic is that texts with similar meaning produce vectors close to each other, even if they use different words: "a product can be returned within the first month" would land near the previous one, even though it shares almost no words with it. Think of it as assigning every text a coordinate on a map of meanings: similar topics land in the same neighborhood, and searching "how do I return something" takes you to the right neighborhood with no need for the exact words to match.

A vector store is the cabinet where you keep those vectors so you can search by closeness. Qdrant —one of the Starter Kit's services— is a vector store: it stores every chunk's vectors and, when you give it a question's vector, quickly finds the closest chunks on the map of meanings. That's what makes RAG work: retrieving the document pieces relevant to a question to hand them to a language model that drafts the answer.

Whoever produces the embeddings, locally, is Ollama. The Starter Kit's other service runs models on your machine, with no paid API. There are dedicated models for embeddings; as of this guide's writing —July 2026—, two widely used ones in Ollama's library are nomic-embed-text (768 dimensions, good at handling long texts) and mxbai-embed-large (1024 dimensions). The model list changes over time and new ones show up frequently, so check Ollama's current library when you set this up; the dedup pattern taught here doesn't depend on which model you choose.

With that clear, the "costly effect" we don't want to repeat is: taking a document, splitting it into chunks, asking Ollama for each chunk's embedding, and storing them in Qdrant. Repeating it over a document that didn't change wastes compute and duplicates chunks. Idempotency is not doing it again if it's already been done.

Why "costly," if Ollama is local and free? Because "free" isn't "instant": embedding dozens or hundreds of chunks takes processor time —and if in your real setup you used a paid API embedding model instead of Ollama, every reprocessed chunk would also be, on top of that, money—. The pattern you learn here protects you in both scenarios: locally, it saves you time; in the cloud, it saves you the bill. And in both, it prevents duplicates in the store, which is a quality cost independent of the compute one.

The right key: the content hash, not the file name

Here's the design decision making all the difference, and it connects directly to lesson 4's false duplicate: which key identifies "I already processed this document"?

The temptation is to use the file name: "if I've already seen return-policy.pdf, skip it." But that fails in both directions. If the file got updated —the policy changed, but the name is the same— the name would make you skip it, and your vector store would be stuck with the old version forever: a duplicate slipping past in reverse, a document that should have been reprocessed and wasn't. And if the same content arrives with two different names, you'd process it twice.

The right key is a hash of the file's content. A hash is a fingerprint: a function taking the complete content and producing a short, fixed string representing it. Its key property is that the same content always produces the same hash, and the tiniest change in the content produces a completely different hash. Think of it as a document's fingerprint: change a single word and the fingerprint changes; if it's identical byte for byte, the fingerprint is identical, no matter what the file is called.

With the content hash as the key, deduplication does exactly the right thing:

  • Document identical to one already processed → same hash → already in the store → skip it (don't waste compute, don't duplicate).
  • Document that changed (even with the same name) → different hash → new key → process it (embed it, update the knowledge).
  • Same content under a different name → same hash → already there → skip it (don't duplicate over a rename).

It's the same "identify the work, not the attempt" logic from module 2. The "work" here is embedding this exact content; the content hash is what identifies it unambiguously.

The dedup store for documents

The table is identical in shape to lesson 5's processed_orders; only the names change to reflect the domain:

CREATE TABLE IF NOT EXISTS processed_documents (
  content_hash   TEXT          PRIMARY KEY,
  source_name    TEXT          NOT NULL,
  processed_at   TIMESTAMPTZ   NOT NULL DEFAULT now()
);

content_hash is the primary key —unique, the locker lock—. source_name stores the file's name or origin, just so you can look at the table and understand what each row is (same as order_id in the orders store: it isn't the key, it's the readable label). And processed_at marks when it was ingested.

The gate is, word for word, the same statement you already know, with the key and the table changed:

INSERT INTO processed_documents (content_hash, source_name)
VALUES ($1, $2)
ON CONFLICT (content_hash) DO NOTHING
RETURNING content_hash;

If it returns a row, the document is new: embed it. If it returns empty, it was already there: skip it. You recognize the pattern because it's the pattern. We invented nothing new; we moved the same mechanism to a different door.

Computing the content hash with crypto

You compute the hash in a Code node with crypto, same as idempotency_key. But there's an n8n 2.0 restriction worth facing head-on here, because it affects how the content reaches the code.

The Code node can't read files from disk. It has no file system access. So the document's content isn't read by your code: it's read by a dedicated node —for example, a node that reads a binary file, or the RAG flow's own document loader— and the content reaches the Code node as part of the item, either as text or as binary data. Your code only computes the hash of what arrives, it doesn't go fetch the file.

// ============================================================
// Node: Code — "Compute content hash"
// Mode: Run Once for Each Item
//
// INPUT:   an item with the document CONTENT already read by an
//          earlier node (the Code node does NOT read files from disk in n8n 2.0).
// OUTPUT:  the same item, with a computed content_hash.
// NOTE:    crypto IS allowed in n8n 2.0's Code node.
// ============================================================

const crypto = require('crypto');

// The document text arrives in the item, put there by a previous node
// (a file reader, a document loader, etc.).
const documentText = $input.item.json.text;

// The content fingerprint: same content → same hash,
// any change → different hash. That is why it identifies "this exact document".
const contentHash = crypto.createHash('sha256').update(documentText).digest('hex');

return {
  json: {
    ...$input.item.json,
    content_hash: contentHash,
    source_name: $input.item.json.source_name,
  },
};

An honest nuance: exactly which field the content comes from depends on how you loaded it —it can be text, it can come in a binary field you first convert to text, it can be a loader's output—. The idea that doesn't change is that the content already arrives in the item, put there by a node that can read it, and the Code node just hashes it. Adjust the field name to your flow; check on the panel what each item carries.

Worked example: ingesting Cumbre's knowledge base

Cumbre wants its support agent to be able to answer questions about policies, products, and suppliers. For that, it ingests a folder of documents into Qdrant. The flow, with the dedup gate, looks like this:

Trigger (manual or scheduled)
  └─► Read the document list          ← a node that brings in the files/contents
        └─► Code: "Compute content hash"     ← fingerprint of each document
              └─► Postgres: "Doc dedup gate"  (INSERT ... ON CONFLICT ... RETURNING)
                    └─► IF: "Is it new?"
                          ├─ true  (returned a row → new content)
                          │    └─► Embeddings (Ollama) → Qdrant: Insert Documents
                          │
                          └─ false (empty → already ingested)
                               └─► skip (do not embed, do not insert)

The gate sits before embedding, not after. That's the key to the savings: if a document is already there, you don't even call Ollama to embed it. The expensive compute only happens on the "new" branch.

What to expect. The first time you run the ingestion with, say, twenty documents, the processed_documents store is empty, so all twenty pass through true, get embedded, and get inserted into Qdrant. You see twenty new rows in processed_documents and the corresponding chunks in Qdrant. The second time you run the ingestion with the same twenty documents plus two new ones, twenty collide with ON CONFLICT and go to false —they get skipped, with no call to Ollama—, and only the two new ones get embedded. processed_documents now has twenty-two rows, and there isn't a single duplicate chunk in Qdrant. You ran the full ingestion twice and the expensive work only happened on what was genuinely new. That's idempotency in ingestion: running the pipeline a thousand times leaves the same result as running it once, with no waste and no duplicates.

And if a document changed between the first and second run —the return policy got edited— its new content produces a new hash, so the gate treats it as new and reprocesses it. Exactly what you want: skip what's identical, reprocess what changed.

Granularity: the document changed, what about the old chunks?

There's a detail separating a toy idempotent ingestion from one that works for real, and it's worth facing head-on because it's where a lot of people trip up.

A document doesn't get embedded whole in one shot: it first gets split into chunks —paragraphs, sections, manageable-sized pieces— and each chunk gets embedded separately and stored as a point in Qdrant. A single policy PDF can turn into fifteen chunks. This is normal and necessary, because searches want to return the relevant piece, not the whole document.

Here's the nuance. We're deduplicating at the document level —the key is the full file's content hash—, not at the chunk level. That's fine for the common case: if the document is identical, you skip all its chunks at once, which is exactly the savings you're after. The problem shows up when a document changes: its new hash makes the gate treat it as new and reprocess all its chunks —correct—, but the previous version's old chunks stay in Qdrant. If you only insert the new ones, you end up with both versions coexisting: the old and the new, and your searches return a mix of both.

The solution is making reprocessing a document also idempotent in the store: before inserting the new chunks, delete the old ones from that same document. To be able to do that, when you insert each chunk you tag it with a label saying which document it came from —its source_name or, better, a stable document identifier—, so you can tell Qdrant "delete every point from this document" before putting the new ones in. The flow for a document that changed looks like this:

Document changed (new hash → passes the gate as new)
   └─► Delete points in Qdrant tagged with this document   ← cleans up the old
         └─► Embed the new chunks with Ollama
               └─► Insert the new chunks into Qdrant          ← only the new stays

An honest nuance about capabilities: whether your version's Qdrant node lets you delete points by a tag or filter depends on the version and the available operation, so check it on the panel; if it doesn't expose it directly, the idea —cleaning up the old before putting in the new— is still correct, and you implement it with whatever delete operation your version does offer. For this module's most common case —documents almost always identical between runs— the document-level gate already gives you 90% of the value: it doesn't reprocess what didn't change. Fine-grained update handling is the extra mile, and it's worth knowing it exists so you don't get caught off guard by old versions coexisting.

A nuance about Qdrant and the honesty of checking

There's a second way to seek idempotency in a vector store, and it's worth mentioning with honesty about its limits. Some vector stores, Qdrant included, let you give every point a deterministic identifier —derived, for example, from the content hash—, so inserting the same point twice overwrites instead of duplicating. That would be idempotency on the store's side: even if you tried inserting twice, there'd be no duplicate because the second one overwrites the first.

The honest nuance: whether n8n's Qdrant node lets you fix that point identifier depends on the version and the operation, and the node's documentation, as of this guide's writing, doesn't clearly detail it for the document-insertion operation. That's why the pattern taught here —the Postgres dedup gate before embedding— is the most reliable and the one that doesn't depend on that capability: it works with any vector store, it saves you the embedding's compute (which the overwrite approach doesn't save you, because you still embed before inserting either way), and you can audit it with a SELECT. If your version of the Qdrant node does expose deterministic point identifiers, you can add it as a second line of defense; but don't depend only on that, and check on the panel what your version allows. Where the documentation doesn't confirm something, teach the concept and verify on your own installation: never assume a capability you haven't seen work.

Common mistakes

Deduplicating by file name instead of by content (conceptual, the big one). What happens: the file name gets used as the key, and when a document gets updated while keeping the name, the system skips it and the vector store is stuck with the old version forever. Why it happens: the name is the most visible thing and looks like it identifies the document. How to spot it: if you update a file's content, run the ingestion again, and searches keep returning the old stuff, this is it. How to fix it: use a content hash as the key; that way a content change produces a new key and the document gets reprocessed, while identical content gets skipped.

Putting the gate after embedding (practical). What happens: every document gets embedded and only afterward is it checked which ones were already there, so the expensive compute gets spent even on the ones about to be skipped. Why it happens: "process and then filter" seems natural. How to spot it: if your ingestion takes the same time the second run as the first even with no new documents, the gate is in the wrong place. How to fix it: put the dedup gate before calling Ollama; the goal is not embedding what's already there, and that's only achieved by deciding before the embedding.

Inserting with no deduplication and degrading searches (practical). What happens: the ingestion runs several times with no dedup at all, Qdrant accumulates copies of the same chunks, and the agent's answers get worse because searches return duplicates. Why it happens: with no dedup, every run inserts everything again. How to spot it: if searching a topic returns repeated chunks, or the point count in Qdrant grows every time you re-ingest the same thing, this is it. How to fix it: the content-hash dedup gate prevents inserting what's already there; and if you already dirtied the store, you may need to clean it up and re-ingest with the gate in place.

Trying to read the file from the Code node (practical). What happens: code gets written trying to open the file to hash it, and it fails because the Code node has no file system access in n8n 2.0. Why it happens: it comes naturally to think "I read the file and hash it in the same node." How to spot it: if your Code node tries to read from disk and can't, this is it. How to fix it: have a dedicated node read the file and pass its content in the item; the Code node only computes the hash of what it receives. crypto is allowed for hashing; reading files, it isn't.

Assuming the Qdrant node deduplicates on its own (conceptual). What happens: it gets assumed inserting the same document twice "Qdrant surely handles it" and no dedup gets set up, trusting an overwrite the node's version might not do. Why it happens: a capability is taken for granted with no verification. How to spot it: if your only defense against duplicates is an assumption about the Qdrant node, and you didn't check it, this is it. How to fix it: don't depend on unverified capabilities; the Postgres gate is reliable, auditable, and saves the embedding. Check what your Qdrant node actually does, and if it exposes deterministic IDs, use it as reinforcement, not as your only defense.

Exercises

Exercise 1 — Choose the key. For each situation in Cumbre's ingestion, say what the dedup key should be and why, in one sentence.

(a) The same folder gets re-ingested every night; almost everything is identical to the day before. (b) The return policy got edited, but the file is still called the same thing. (c) A supplier sends their catalog by email twice, with different file names, same content.

See solution

(a) Content hash. Identical documents produce the same hash and get skipped with no compute spent; only what genuinely changed gets reprocessed. Exactly the case content dedup exists for.

(b) Content hash, and here's where you see why the name doesn't work. The name didn't change, but the content did, so the hash is different and the document gets reprocessed. If you deduplicated by name, you'd be stuck with the old policy forever.

(c) Content hash. Two different names, same content → same hash → processed once. If you deduplicated by name, you'd process it twice and duplicate the chunks in Qdrant.

Why this works: all three cases point to the same thing: what identifies "the same document" is its content, not its name. The content hash is the only key getting all three right, because it answers the right question —"is this the exact same content?"— and not an approximation —"is it called the same thing?"—.

Exercise 2 — Locate the gate. You're given two ingestion designs. In design A, the order is: read documents → embed all of them with Ollama → dedup gate → insert the new ones into Qdrant. In design B: read documents → hash → dedup gate → embed only the new ones → insert into Qdrant. Which one saves the expensive compute and why? What does the other one waste?

See solution

Design B saves the expensive compute. The gate goes before embedding, so Ollama only gets called for the documents that turned out to be new. The ones already there get discarded before touching the model.

Design A wastes the embedding. It embeds every document —including ones already there— and only afterward decides which to insert. The expensive step (asking Ollama for every chunk's embedding) already got paid for all of them, even though most are about to be discarded. Design A's dedup does prevent duplicates in Qdrant, yes, but it saves no compute at all, which is usually the pipeline's most expensive part.

Why this works: the principle is the same as lesson 3's ledger and lesson 5's gate —decide before acting on the costly effect—. Here the costly effect is embedding, so the decision has to come before the embedding, not after. Placing the gate is placing where the expense you want to avoid sits.

Exercise 3 — Connect the pattern. In a short explanation, describe how deduplicating charges (lesson 5) and deduplicating document ingestion (this lesson) are alike and how they differ. The goal is for you to see it's the same pattern.

See solution

A reference version:

How they're alike —which is almost everything—: the mechanism is identical. A table with a key column marked unique, and an INSERT ... ON CONFLICT DO NOTHING RETURNING deciding, in a single atomic step, whether it's the first time (returns a row → act) or a duplicate (returns empty → skip). In both cases the gate goes before the costly effect, and in both the key must identify "the work" and not "the attempt."

How they differ —only two things—: first, the key. In charges it's an order_id or a hash of order fields; in ingestion it's a hash of the file's content. Second, the effect you're protecting. In charges it's creating a charge in the CRM (costs money, is irreversible); in ingestion it's embedding with Ollama and inserting into Qdrant (costs compute and dirties the store with duplicates). Different key, different effect, same skeleton.

The conclusion: idempotency isn't a charge-processing technique. It's a pattern —persistent check before a costly effect you don't want to repeat— and it serves equally for money, for vectors, or for any other action with consequences.

Why this works: seeing the pattern above its two applications is what lets you recognize it in a third case you haven't seen yet. Someone who only learned "how not to charge twice" solves one problem; someone who learned "how not to repeat a costly effect" solves a whole family.

Summary and next step

In this lesson you carried the module's deduplication into RAG ingestion and confirmed it's the same pattern with different skin. Re-embedding a document you already processed wastes compute and fills the vector store with duplicate chunks that degrade searches —like re-photocopying and re-filing a document already in the cabinet—. The solution is lesson 5's dedup store, with a processed_documents table, the same INSERT ... ON CONFLICT DO NOTHING RETURNING gate, and a key change.

That key change is the central decision: the right key is a hash of the file's content, not its name. The hash is the content's fingerprint —same content, same hash; any change, different hash—, so it skips what's identical, reprocesses what changed even under the same name, and doesn't get fooled by a rename. You learned about embeddings (translating meaning into numbers) and vector stores (the cabinet searching by closeness), with Qdrant and Ollama from the Starter Kit doing it locally at zero cost, with models like nomic-embed-text whose list is worth checking. And you saw the golden rule: the gate goes before embedding, so the expensive compute only happens on the new stuff.

And you saw the extra mile: since a document gets split into chunks, when one changes it's worth deleting its old chunks from Qdrant before inserting the new ones, so you don't leave two versions coexisting. The document-level gate gives you most of the value; that fine-grained update handling is what takes it to production.

Before moving on you should be able to: explain why the content hash is a better key than the file name; say why the gate goes before the embedding; and articulate how this resembles deduplicating charges.

Lesson 8 closes the module with the project: you build the ledger and the dedup store in local Postgres, connect them to a webhook that fires twice, and demonstrate —with the tables and the CRM in view— that the second trigger gets discarded before the effect. Everything you designed, set up, and applied across the seven lessons comes together in a deliverable you can defend: the table schema plus the flow using it, tested end to end against the duplicate.

Resources

← Previous6. The local stack for state, at zero costNext →8. Project: a deduplication ledger for a webhook that fires double