1. Introduction: where the system's truth lives

Description

By the end of this lesson you'll be able to explain why idempotency and deduplication —the two topics you worked on in the previous modules— need a physical place to live, and why that place can't be the workflow's memory. You're going to have the full map of this module's eight lessons, you're going to know what you're going to build at the end (a run ledger and a deduplication store in a real database), and you're going to meet the local, zero-cost stack you're going to build everything on: the Postgres the Self-Hosted AI Starter Kit v2 already ships with.

This matters for a very concrete reason. In module 2 you made a record-creating step idempotent: before acting, the workflow asked itself "did I already do this?" But that question makes no sense if there's nowhere to store the answer. A workflow that asks itself "did I already process this order?" with no persistent memory is like a cashier who decides whether they already charged you by looking only at what they remember from this morning: the moment they go to lunch, they forget everything. Idempotency across separate executions —the real case, the one where the webhook fires twice ten minutes apart— demands a place where the system's truth gets written down and survives past the execution ending. This module builds that place.

Connection to the module: this lesson is the map, not the territory. Here you lay out the problem (idempotency needs persistent state and the workflow doesn't reliably have it), you meet the case you're going to solve, and you get the plan for the seven lessons that follow. Lesson 2 is the argument's heart: why Static Data and variables aren't enough. Lesson 3 designs the run ledger. Lessons 4 and 5 build the deduplication store and the pattern that makes it atomic. Lesson 6 connects everything to the Starter Kit's local Postgres. Lesson 7 applies the same pattern to a RAG pipeline. And lesson 8 pulls it together into a project: a deduplication ledger for a webhook that fires double. A scope note from the start: this module doesn't teach you SQL from scratch or how to operate a database in production —that's the data ecosystem and the operations guide—. Here you use Postgres as the tool that sustains the system's correctness, with the minimum SQL that job needs, explained piece by piece.

Why the workflow's memory isn't a good place for the truth

Let's start with the problem, because it's what gives everything else meaning.

Think of two ways of keeping track of who you've already delivered a package to. The first: you remember it while doing your morning route. It works well for a while —you know you already left the box for the lady in 3B—. But once you finish your shift, close up the van, and go home, that memory clears. Tomorrow you start from zero. If by mistake you're assigned a repeat of yesterday's delivery, you have no way of knowing it: your only source of truth was your head, and your head already reset.

The second way: you write down every delivery in a notebook, with the date and the recipient's signature. The notebook doesn't clear when your shift ends. The next day, before dropping off a package, you can open it and ask: "did this address already receive this order?" If the answer is written there, you don't repeat. The notebook survives your shifts; your memory doesn't.

The exact same thing happens in n8n, and it's the point organizing this whole module. Every time a workflow fires, a new execution starts. That execution has its own memory: the items flowing between nodes, the variables you compute, what a Code node stores in a constant. All of that lives while the execution runs and disappears when it ends. It's the courier's morning memory. It works for what happens inside an execution, and it's useless for anything that needs to be remembered between executions.

This connects directly to the execution model you saw in module 1. There it became clear that every trigger produces an isolated execution, with its own history in n8n, and that two executions of the same workflow don't know about each other: there's no shared variable one writes and the other reads. What was, in module 1, an observation about how n8n runs, here becomes the central problem. Because the isolation that makes n8n robust —that a failure in one execution doesn't contaminate another— is the same isolation that stops one execution from telling the next "hey, I already charged this order." That isolation is a virtue of the engine, and the price of that virtue is that you have to provide the shared memory yourself, outside.

And here's the problem, because the case that matters to us lives right on the wrong side of that boundary.

Worked example: the order that gets processed twice

Let's put on the table the case you're going to solve throughout this module. It's the same one from the previous guides: Cumbre, the coffee and tea wholesale distributor, and its order-triage workflow.

order-triage does three things, in order:

Webhook  ──►  AI Agent  ──►  HTTP Request: "Create charge in CRM"
(an order      (classifies      (creates a real charge for the order)
 comes in)      the order)

The order comes in through a webhook. An AI Agent classifies it —priority, route, whatever—. And at the end, an HTTP Request node calls Cumbre's CRM and creates a charge. That last step is an effect: it moves real money in the real world. It isn't a read you can repeat with no consequences; it's an action that, repeated, charges twice.

Now the scenario that breaks everything. The webhook fires twice for the same order. It can happen for many reasons you already saw in module 1: the calling system retried because it didn't get a response in time, the network duplicated the message, someone double-clicked. The point is that two triggers arrive with the same order_id, say ORD-2041, a few minutes apart.

Each trigger starts a different execution. And here's the detail that makes the problem so hard: the first execution has already finished by the time the second arrives. Its memory —everything it "knew"— has already been erased. The second execution has no way of knowing the first existed, unless the first left something written in a place that survives.

Without that place, here's what happens:

09:12  Execution #1  →  ORD-2041  →  creates charge  →  finishes (and forgets everything)
09:19  Execution #2  →  ORD-2041  →  creates charge  →  finishes
                                        ▲
                        second charge for the same order

What to expect. Two successful executions, both green in n8n's history, not a single error. And two charges in the CRM for an order the customer placed once. It's the worst kind of failure: silent. Nothing broke. The system did exactly what you asked it to twice, because you never gave it a way to remember it had already done it once.

The question this module solves is direct: where does the sentence "ORD-2041 has already been processed" have to live so the second execution can read it? Not in the first execution's memory, which no longer exists. It has to live in an external, persistent, shared place: a database. The courier's notebook.

What you're going to build in this module

This module's answer has two pieces, and you're going to build both on top of local Postgres.

A run ledger (execution log). A table noting what ran, with what key, in what state, and with what result. It's the complete notebook: it doesn't just say "this already happened," it says "this happened, at this time, and ended like this." Lesson 3 designs it.

A deduplication store (dedup store). A smaller, sharper table, built for a single question: "have I already seen this key?" Its trick is a uniqueness constraint in the database that turns the question "does it already exist?" and the action "mark it as seen" into a single indivisible step. Lessons 4 and 5 build it, and there you're going to see why that "single step" solves a trap module 2 left open.

The two tables complement each other, and it's worth being clear from the start about how they differ:

	run_ledger (lesson 3)	processed_orders (lesson 5)
Question it answers	"What ran, when, and how did it end?"	"Have I seen this key already, yes or no?"
How much it stores per execution	A lot: key, state, result, timing	The minimum: the key and little else
What you query it for	Auditing, understanding what happened, retrieving a result	Deciding in an instant whether to act or discard
Analogy	The complete accounting ledger	The "already served" list at the door

They don't compete: you're often going to use them together. The dedup store gives you the fast yes/no before acting; the ledger gives you the full history for when someone asks what happened to ORD-2041. This module teaches you to build and combine both.

With those two pieces, order-triage changes shape. Before creating the charge, it checks the deduplication store:

Webhook  ──►  Is ORD-2041 already in the dedup store?
                   │
                   ├─ No  →  mark it  →  AI Agent  →  create the charge
                   │
                   └─ Yes →  discard it, do nothing

The second execution arrives, asks the same question, and this time the answer is "yes." It doesn't create the second charge. The truth —"ORD-2041 has already been processed"— lived outside both executions, in a place both could query. That's this whole module, in one picture.

Execution state and system state: the distinction that orders everything

There's a distinction that, once you see it, orders not just this module but your entire way of designing automations. It's worth making it explicit, because it's what separates someone who assembles flows from someone who owns a system.

In any automation there are two classes of data coexisting, and confusing them is the origin of almost every duplicate bug.

Execution state is everything that exists while an execution runs and only makes sense within it. The order that came in through the webhook, the total you calculated, the AI Agent's response, the body you built for the CRM call. It's born when the execution starts and dies when it ends. It's correct that it dies: if the same order gets processed again, all of that recalculates with no problem. This state naturally lives in n8n's items and in your nodes' variables.

System state is what has to be true between executions, what defines what the system has done over time. "I already charged ORD-2041." "I already embedded this document." "The last order I synced with the CRM was ORD-2040." This state can't die with the execution, because its only reason for existing is so the next execution can query it. If it lives in one execution's memory, it's like writing the warehouse inventory on the palm of the hand of whoever's leaving their shift: nobody on the next shift can read it.

Think of it with a store image. Execution state is the conversation with a customer at the register: the products going through, the subtotal accumulating, the change you give them. When the customer leaves, that conversation ends, and it's fine that it ends. System state is the inventory and the sales ledger: they don't belong to any particular conversation, they survive every customer of the day, and they're what gives the business continuity. If the inventory got erased when the register closed, the store couldn't open tomorrow.

The design mistake this module prevents can be said in one sentence: treating system state as if it were execution state. Storing "I already charged ORD-2041" in a Code node's variable —which is execution state— is exactly that mistake. It works in the demo, where you test everything in a single run, and fails in production, where every trigger is a new execution that doesn't inherit the previous one's memory.

And there's a trap lesson 2 takes apart in detail. n8n seems to offer a place for system state without leaving the workflow: $getWorkflowStaticData(). Its name suggests it —"static workflow data," something that persists—. Lesson 2 shows why that promise doesn't hold up for a serious idempotent system, starting with a fact that surprises almost everyone: Static Data isn't even saved when you test the workflow from the editor. For now, hold on to the distinction, which is what matters: there's state that dies with the execution and state that has to survive it, and the second needs a real home.

The local stack: where that truth is going to live, at zero cost

You might think "a real database" means hiring a service, paying a monthly fee, and configuring a server. For learning, and for most small teams, no. You already have a database to spare, and you probably didn't even know you had it running.

The Self-Hosted AI Starter Kit v2 is an official n8n template that spins up, with a single command, a complete AI automation environment on your own machine. Per its documentation, it includes four pieces:

Service	What it is	What we use it for in this module
n8n	The workflow platform you already know	Where order-triage and this module's flows run
PostgreSQL	A robust, mature relational database	The home for the run ledger and the dedup store
Qdrant	A vector store for semantic search	Lesson 7's idempotent RAG ingestion
Ollama	An engine for running local language models	Generates lesson 7's embeddings, with no paid API

Notice something important: n8n, when you spin it up with the Starter Kit, already stores its own workflows and executions in that Postgres. The database isn't an extra you have to set up; it's there, running, from the first minute. The only new thing you're going to do is create a couple of tables of your own —run_ledger and processed_orders— in that same database, and connect a few Postgres nodes to them. Lesson 6 does that wiring step by step.

And the cost of all this is zero. It runs on your computer, with Docker, no cloud accounts or credit cards. As of this guide's writing —July 2026— the Starter Kit is brought up with docker compose --profile cpu up and n8n becomes available at http://localhost:5678. Like with any versioned fact, the details will probably change over time; the pattern —a local stack with Postgres included— is what transfers. Lesson 6 honestly flags what to confirm on your own installation.

Why a relational database, and not a spreadsheet or a file

A legitimate question before continuing: if all you need is to note "I've seen this key," why Postgres and not something simpler, like a Google sheet or a file? The answer previews what makes this module special, and it's worth having from the start.

A relational database like Postgres has three capabilities a spreadsheet doesn't give you, and all three are exactly what idempotency needs.

The uniqueness constraint. You can tell a table "this column never allows repeated values," and the database enforces it for you. It isn't a rule you program and hope to remember to apply; it's a property of the table the database guarantees on every write. When in lesson 5 you mark the idempotency_key column as unique, Postgres becomes your ally: it rejects the second ORD-2041 with you not having to check anything. A spreadsheet lets you put the same row in a thousand times with no complaint.

Atomicity. Postgres can do "check if it exists and, if not, insert it" as a single indivisible operation, impossible to interrupt halfway. This sounds minor and it's the heart of this entire module. In a spreadsheet, "read whether it's already there" and "write that it's now there" are two separate steps, and between those two steps the webhook's second trigger can slip in —and duplicate—. That's exactly the "check then act" trap module 2 flagged and lesson 5 closes with ON CONFLICT.

Serious durability. When Postgres confirms it saved something, it truly saved it, with guarantees designed not to lose data in a power outage. A loose file or a shared sheet doesn't give you that level of certainty about what was written and what wasn't.

It isn't that a spreadsheet is "bad" —for many things it's perfect, and in fact n8n has excellent nodes for talking to Google Sheets—. It's that idempotency rests exactly on the three things a relational database does well and a sheet doesn't: guaranteed uniqueness, atomic operations, and serious durability. That's why the home for the system's truth is Postgres, and that's why the Starter Kit, which brings it included, is so convenient for learning this. If you already use another relational database at work —MySQL, SQL Server— the concepts transfer as-is; the syntax details change, not the pattern.

This module's map

Lesson	What it solves
2	Why Static Data, $vars, and the workflow's memory aren't enough for the system's truth, with the concrete proof that Static Data isn't saved when testing from the editor
3	Designing the run ledger: what columns it has, what states it records (pending / done / failed), and why it's the single source of truth for what already happened
4	The three deduplication strategies —time window, seen-key, and the Remove Duplicates node— and where each falls short
5	The dedup store across executions: INSERT ... ON CONFLICT DO NOTHING as the atomic step that resolves the "check then act" trap
6	The local stack: connecting Postgres nodes to the Starter Kit's Postgres, with the credential set up step by step
7	Idempotent RAG ingestion: not re-embedding an already-processed document, using a content hash as the key
8	Project: the complete dedup ledger for a webhook that fires double, tested end to end

The order isn't accidental. First the argument (lesson 2): why you need a database and can't avoid it with a trick inside the workflow. Then the design of the two tables (lessons 3 through 5). Then the wiring to the real stack (lesson 6). Then an application of the same pattern in a different domain, RAG (lesson 7). And finally, the project that integrates it (lesson 8).

What you'll be able to do by the end of the module

The exit skill is concrete and verifiable. By the end of lesson 8 you're going to be able to:

• Explain, with an example, why one execution's memory doesn't work for deduplicating across separate executions, and why Static Data doesn't reliably close that gap.
• Design a run ledger in Postgres: its columns, its states, and its unique key.
• Build a deduplication store that, with a uniqueness constraint and ON CONFLICT DO NOTHING, decides in a single atomic step whether a key is new or repeated.
• Connect a Postgres node to the Starter Kit's local Postgres and use it to query and record keys.
• Apply the same pattern to a RAG ingestion to avoid re-embedding a document you already processed.

What you're not going to do in this module, and that's fine: administer Postgres in production —backups, replicas, tuning—, or write advanced SQL. You're going to use exactly the SQL needed to sustain idempotency, and every query comes explained. If by the end you feel you understand why each table exists and what each query does, even if you don't feel like a database expert, that's exactly the expected result.

Common mistakes

Believing idempotency gets solved "inside the code" (conceptual). What happens: someone finishes module 2, understands the idea of "check before acting," and implements it with a variable or a list inside a Code node, convinced that's enough. Why it happens: within a single execution, that logic works —if the same order shows up twice in the same batch, you can filter it—. The trick is the real case isn't that one. How to spot it: ask yourself whether your deduplication survives the execution ending. If the answer lives in a Code node's variable, it doesn't survive. How to fix it: accept from now that the system's truth has to leave the workflow and live in a database. It's lesson 2's entire argument, and it's this whole module's reason for existing.

Confusing "the system already knows it" with "I can query it" (conceptual). What happens: Cumbre's CRM, internally, surely has order ORD-2041 registered; someone concludes the workflow therefore "already has" the information and no ledger of its own is needed. Why it happens: it's true the data exists somewhere. The problem is the cost and reliability of querying it in time, before acting, on every trigger. How to spot it: if your plan for not duplicating is "the destination system is going to reject it," you're delegating your correctness to another system that may not guarantee it. How to fix it: in lessons 4 and 5 you're going to see why it's worth your workflow having its own dedup store, cheap and under your control, instead of depending on a remote query on every call.

Thinking "database" implies money and servers (conceptual). What happens: someone postpones the whole module because they think they need to hire a cloud database. Why it happens: the word "database" sounds like expensive infrastructure. How to spot it: if your obstacle to starting is "I don't have a database," you almost certainly do. How to fix it: the Starter Kit v2 ships with Postgres running on your machine, at zero cost. It's already there. Lesson 6 just shows you how to point a node at it.

Exercises

Exercise 1 — Trace a truth's life. For order-triage's double-trigger case, write in one sentence each: (a) what information the second execution needs to know to avoid duplicating the charge, (b) at exactly what moment the first execution should write that information, and (c) why that information can't live inside the first execution.

See solution

(a) The second execution needs to know that order_id ORD-2041 has already been processed (or at least, that its processing has already started). It's a single fact: "this key has already been seen."

(b) The first execution should write that fact before creating the charge, not after. You'll see the fine-grained reason in lesson 5: if you write it after the effect and the second trigger arrives in between, you can still duplicate. The record goes first, the effect after.

(c) It can't live inside the first execution because that execution has already ended by the time the second arrives, and with it its whole memory was erased. Two separate executions don't share memory; they can only communicate through something external that outlives both. That "something" is the database.

Why this works: notice all three answers point to the same place. The what (a small fact), the when (before the effect), and the where (outside the execution) are the three decisions defining a deduplication store. You already have them laid out; the module just makes them concrete.

Exercise 2 — Classify where each piece of data lives. For each of these order-triage data points, decide whether it makes sense for it to live in the execution's memory (cleared when it ends) or in a database (survives). Justify in one sentence.

(a) The order's total, which you calculate by summing the lines to build the charge. (b) The fact that ORD-2041 has already been processed. (c) The priority classification the AI Agent returned for this order. (d) The count of how many orders you've processed in total since the workflow has existed.

See solution

(a) Execution memory. The total gets calculated, used to build the charge within this same execution, and doesn't need to survive: if the order gets processed again, it recalculates. It's transit data.

(b) Database. It's exactly the truth that has to survive between executions so the second one doesn't duplicate. It's the module's central case.

(c) Depends on what for. If you only use it within this execution to decide the route, memory. If you want to be able to later answer "what priority did we give ORD-2041?", then it's worth storing —and in fact lesson 3's run ledger has a result column precisely for that—. This is a good example that the boundary isn't always obvious.

(d) Database. A running "since forever" count is, by definition, something that has to survive every execution. If it lives in memory, every execution would start counting from zero. Lesson 2 shows why trying to keep this count with Static Data is fragile.

Why this works: the question you're practicing is a single one —does this data need to be remembered after the execution ends?—. If the answer is yes, it's a database. If no, it's memory. Almost all system state design comes down to getting that classification right.

Exercise 3 — Anticipate the module. Without looking back at the map table, write from memory what you think each lesson from 2 to 8 solves, one sentence each. Then compare and mark the ones you missed.

See solution

A reference version: (2) why the workflow's memory and Static Data aren't enough; (3) the run ledger's design; (4) deduplication strategies and their limits; (5) the dedup store with ON CONFLICT; (6) connecting everything to the Starter Kit's local Postgres; (7) applying the pattern to RAG ingestion; (8) the integrating project with the double-firing webhook.

Why this works: if you rebuilt at least five of the seven, you've already internalized the progression —from the argument (why a database?) to the design (the two tables) to the wiring (the real stack) to the application (RAG) to the project—. The ones people usually miss are 4 and 7, because they seem "in passing"; really, 4 gives you the criterion for choosing a strategy and 7 shows you the pattern is general, not a trick for a single case.

Summary and next step

In this lesson you laid out the problem organizing the whole module. The idempotency and deduplication you worked on before need a place for the system's truth to live, and that place can't be the workflow's memory, because every execution starts from zero and forgets everything when it ends. You saw it with order-triage: two triggers for the same order ORD-2041 start two executions that share no memory, and with no external record, the second creates a second charge with nothing visibly failing.

The solution has two pieces you're going to build: a run ledger, the complete notebook noting what ran and how it ended; and a deduplication store, the sharp table answering "have I seen this key?" in a single atomic step. Both live in the Postgres the Self-Hosted AI Starter Kit v2 already ships with, running on your machine at zero cost, alongside Qdrant and Ollama you're going to use in lesson 7.

Before moving on you should be able to: explain why two separate executions don't share memory; name the two tables you're going to build and what each is for; and say which four services the Starter Kit brings.

Lesson 2 is the argument that makes everything else inevitable. There you're going to see head-on why $getWorkflowStaticData() —which looks like n8n's natural answer to "I need to save something between executions"— isn't a reliable place for a serious idempotent system's truth. You're going to see the concrete proof: that it isn't even saved when you test the workflow from the editor. And with that clear, the database stops being one option among several and becomes the only serious answer.

Resources

• Deploy with the AI starter kit — n8n Docs — the official Self-Hosted AI Starter Kit guide: what services it brings (n8n, PostgreSQL, Qdrant, Ollama) and how to spin it up with Docker on your machine.
• Self-hosted AI Starter Kit — n8n Blog — the kit's presentation, useful for understanding what each component solves and why they come together.
• self-hosted-ai-starter-kit — GitHub — the repository with the real docker-compose.yml; confirm there the services and the current commands for your version.
• Postgres node — n8n Docs — the node you're going to use to talk to the database: its operations and how it passes parameters safely. You use it from lesson 3 onward.
• getWorkflowStaticData — n8n Docs — the Static Data mechanism lesson 2 examines and rules out as a home for the system's truth.