4. Deduplication strategies

Description

By the end of this lesson you'll be able to choose, with judgment, among the three deduplication strategies that exist in practice: the time window, the seen-key, and n8n's built-in Remove Duplicates node. You're going to know what each one does, when it's the right one, and —most importantly— where each falls short. You're going to understand why the Remove Duplicates node, which looks like it solves the whole problem with one click, has two distinct modes with very different capabilities, and why neither is the right foundation for an idempotent system's truth, even though both are useful for other things.

This matters because deduplication isn't one technique, it's a family, and using the wrong one for the case is a classic source of subtle bugs. A time window that lets through a duplicate that arrived late. A node that deduplicates within an execution but not across executions. An internal store that "forgets" old keys right when you needed them. Every one of those failures comes from applying a strategy outside its territory. This lesson gives you the map of territories so the dedup store you build in lesson 5 is an informed choice, not just the first one you found.

Connection to the module: lesson 3 gave you the ledger, which records. This lesson is the bridge toward lesson 5's dedup store, which decides. Before building that store with a table and ON CONFLICT, it's worth understanding what other options existed and why this is the one sustaining idempotency across executions. You're going to see that the Remove Duplicates node solves similar cases —and for many flows it's perfect— but that, as the foundation of a system's correctness, it carries the same problems lesson 2 flagged in Static Data: it's a store managed by n8n, coupled to the workflow, bounded in size, and one you can't query or audit. Lesson 5 chooses its own table precisely to avoid inheriting those problems.

Three ways to keep the list at the door

Think of the doorman at an event with a list. Their job is to let each person in only once. They have three ways to organize themselves, and each makes sense for a different type of event.

The first: the doorman remembers faces from the last few minutes. If someone tries to sneak in twice in a row, they recognize them and stop them. But if that person comes back three hours later, they don't remember them anymore —they only retain what's recent—. This is the time window: you deduplicate what shows up within a short span, and forget the old stuff.

The second: the doorman marks with invisible ink the hand of every person who comes in, and at the door there's a lamp that reveals the mark. It doesn't matter if you come back in five minutes or the next day: if your hand already has the mark, you don't get in. The list of marked people never gets forgotten. This is the seen-key: you persistently store every key you processed and check it forever (or for a long time). It's the robust strategy, and the one demanding a serious store.

The third: the doorman uses an automatic counter the event company lent them. It comes in a box, does its job, and the doorman can't see inside how it works or change it. It's useful, but it's tied to decisions whoever built the box made: how much memory it has, what it considers "the same person," when it forgets. This is the Remove Duplicates node: a packaged n8n solution, convenient, with the trade-off that its behavior was defined by n8n, not you.

All three deduplicate. The question isn't "which is the good one?", but "which fits this event?" A doorman who only remembers recent faces would be a disaster at a multi-day event, and one who marks the hand forever would be overkill for a line rotating every ten minutes. The right strategy depends on the event, not on which one sounds fanciest. Let's go one by one.

Strategy 1: the time window

The idea is direct: you consider an event a duplicate only if it arrived within a recent span relative to an identical one. "If I've already seen this order_id in the last ten minutes, it's a duplicate; if I see it for the first time in ten minutes, I treat it as new."

Anatomy. You need to store, for every key, when you last saw it. Before acting, you compare: was the last time I saw this key less than N minutes ago? If yes, discard. If no (or if you never saw it), process and update the timestamp. A detail worth noting from the start: even the time window needs to store something between executions —at least the key and its last timestamp—, so it doesn't escape needing a persistent store either. It isn't "the option with no database"; it's "the option that also forgets the old stuff." The difference with the seen-key isn't having a store or not, but what you do with old entries: the window lets them expire, the seen-key keeps them.

When it's the right one. The time window shines against the most common duplicate case: the near-immediate retry. The system calling your webhook didn't get your response in time, so it retries within a few seconds or minutes. The two triggers arrive close together. A window of, say, fifteen minutes catches them with no effort, and it has a real advantage: you don't need to remember the key forever. You can clean up the old marks, so the store doesn't grow forever. For very high-volume flows where remembering every key in history would be expensive, the window is a sensible balance.

Where it falls short. The problem is the duplicate arriving outside the window. Imagine order ORD-2041 got processed today, and because of a manual reprocess, a resent backup, or an upstream error, the same order comes back in three days later. If your window is fifteen minutes, that late duplicate passes as new, and you charge again. The window assumes duplicates arrive close together, and that assumption is true for automatic retries and false for almost everything else.

There's also an uncomfortable decision: how big do you make the window? Too short, and you miss duplicates that arrived late. Too long, and the store grows so much you lose the advantage of being able to forget —you get close to the seen-key, but with the extra fragility of a time comparison—. There's no universal correct size; it depends on how far apart your duplicates arrive, which you often don't know beforehand.

The rule: the time window is good when you know your duplicates arrive close together and you need to not accumulate keys forever. It's insufficient when a duplicate can arrive late and the effect is costly —like a charge—.

Strategy 2: the seen-key

This is the robust strategy, and the one lesson 5 turns into a table. The idea: you store every key you processed, with no expiration (or with a very long, deliberate one), and before acting you ask "is this key already in my seen list?" If it is, it's a duplicate, no matter how much time passed. It's the hand marked with invisible ink: the mark doesn't fade on its own.

Anatomy. A persistent store —a table— with a column for the key, marked as unique. Before acting, you query or try to insert. If the key already existed, discard; if it's new, insert it and proceed. Lesson 5 shows how to do that query-and-insert as a single atomic step with ON CONFLICT; for now, hold on to the shape: remember every key, forever, and check before acting.

When it's the right one. Whenever the effect is costly and a late duplicate is possible. Which is, exactly, order-triage's case: creating a charge is costly, and an order could come back in days later for a thousand reasons. Against that scenario, the seen-key is the only one of the three that doesn't fail, because it makes no assumption about when the duplicate arrives. The mark is there or it isn't; time doesn't factor into the decision.

Where it falls short. It has two honest costs. One: the store grows. If you remember every key forever, the table accumulates a row for every unique piece of work you've ever done in the system's life. For most systems this isn't a problem —a table with an indexed column handles millions of rows without breaking a sweat, and querying it by key is instant—, but it's a real cost that at huge volumes needs managing (for example, archiving very old keys that can no longer come back). Two: you need a real store, with uniqueness and atomicity, which is exactly what Static Data didn't give you. In other words, the seen-key is more robust but forces you to have a database. That "cost" is precisely what this module assumes from the start.

The rule: the seen-key is the default strategy for costly, irreversible effects. It assumes nothing about time, so it catches the duplicate from five minutes ago and the one from five days ago equally. Its price is a table that grows, and that price is almost always worth it.

Strategy 3: the Remove Duplicates node, with its two faces

n8n ships a node called Remove Duplicates that looks like it solves all of this with one click. It's worth understanding it well, because it's useful, but it's easy to believe it does more —or less— than it does. The key is that it has several modes, and they do very different things. As of this guide's writing, the documentation describes these operations; worth confirming your version's on the node itself:

Mode A — "Remove Items Repeated Within Current Input." Removes duplicates within the item batch of a single execution. If in one run fifty items arrive and three share the same order_id, this mode keeps one and discards the other two.

This mode has a sharp limit, and it's the most misunderstood one about the node: it only looks at the current execution's items. It knows nothing about previous executions. For the module's central case —the webhook firing twice in two separate executions— this mode doesn't work, because each trigger is a separate execution with its own batch, and mode A never sees both together. It's a classic mistake: someone puts a Remove Duplicates node expecting it to catch the double trigger, and it catches nothing, because the two triggers never coincide in the same batch.

Mode B — "Remove Items Processed in Previous Executions." This one does compare against previous executions. n8n internally keeps its own deduplication store that remembers keys already seen, and this mode queries it. It has a History Size parameter (defaulting to 10,000, per the documentation as of this guide's writing) limiting how many keys it remembers, and a scope that can be node- or workflow-level. It's, at bottom, a packaged implementation of the seen-key.

Worked example: the two modes against the double trigger

Let's put both modes up against our case to see the difference.

With mode A, the workflow would look something like:

Webhook  ──►  Remove Duplicates (Within Current Input, field: order_id)  ──►  create charge

What to expect. The first trigger comes in with ORD-2041 (one item), passes the node —there's nothing to deduplicate within a single item—, and creates the charge. The second trigger, minutes later, comes in with ORD-2041 in another execution, passes the node the same way —again, a single item in its batch, nothing to deduplicate—, and creates a second charge. The node didn't fail; it simply never saw the two triggers together. Mode A is blind across executions.

With mode B, the node would remember ORD-2041 from the first trigger and discard the second. It would work. So why don't we end the module here and use mode B?

Because mode B, though it solves the mechanics, is the same kind of store lesson 2 taught us to distrust as a system's source of truth. Notice the parallels:

• It's a black box you can't query or audit. n8n's dedup store isn't a table you can open with a SELECT to answer "which keys does it remember right now?" or "when did it see ORD-2041?" When something looks strange, you have nowhere to look. Lesson 3's ledger exists precisely so you can look.
• It's bounded in size. History Size defaults to 10,000: when key number 10,001 comes in, the oldest one falls off. If a late duplicate uses a key that's already fallen out of the history, it passes as new. You don't control that forgetting with the precision a charge demands.
• It's coupled to the node or the workflow. Just like Static Data, that store lives glued to the n8n instance, and its scope is the node or the workflow. Another flow needing to know "has this order already been charged?" can't query it. And when the workflow gets moved or reimported, that state's continuity isn't under your control.
• It deduplicates by discarding items, not by branching. The node removes duplicates from the flow. That's fine for "cleaning up a list," but for an idempotent system you often want to know something was a duplicate and do something with that information —log it in the ledger, respond to the webhook with "already processed," alert if it repeats too much—. A table of your own with ON CONFLICT gives you that explicit "first time / duplicate" branching; the node, by design, just makes the duplicate disappear.

None of this means Remove Duplicates is bad. For deduplicating a feed, an imported list, or low-risk events where remembering 10,000 recent keys is enough and you don't need to audit, mode B is convenient and correct. It's the same logic as lesson 2 with Static Data: the packaged tool has its place, and that place isn't the truth you depend on not to charge twice.

The rule: mode A (within input) is for cleaning up duplicates from a single batch, and it's blind across executions. Mode B (previous executions) does cross executions, but as a bounded, coupled black box; use it for low-risk dedup, not as the system's source of truth. For a costly effect's idempotency, the seen-key in your own table gives you control, auditing, and atomicity the node doesn't.

How to choose: the decision table

With all three on the table, the choice comes down to a few questions.

Your situation	Strategy
Duplicates within the same execution's batch (a list with repeats)	Remove Duplicates, mode A (within input)
Automatic retries arriving close together, and you can forget the old stuff	Time window
Costly or irreversible effect (a charge), duplicate can arrive late	Seen-key in your own table (lesson 5)
Low-risk dedup across executions, no need to audit, moderate volume	Remove Duplicates, mode B (previous executions)
You need to query, audit, or branch based on "first time / duplicate"	Your own table (Remove Duplicates doesn't give you this)

And the one-line rule summarizing the module:

To clean up a list, use the node. For the truth a costly effect depends on, use your own table.

Notice your own table wins exactly in the rows where the mistake costs money or where you need to see what happened. It isn't that it's "better" in the abstract; it's that it gives you control, auditing, and atomicity, and those three things are exactly what a serious idempotent system needs and a black box doesn't offer.

Two nuances that prevent bugs: combining strategies and the false duplicate

Two clarifications before building the table, because both are sources of real bugs.

The strategies aren't mutually exclusive; they combine. It can sound like you have to pick one and discard the rest, and that isn't so. In order-triage you're going to end up using two at once: lesson 3's run ledger, which records every execution's rich history for auditing and recovery, and lesson 5's dedup store, which decides the atomic yes/no before acting. The ledger doesn't deduplicate; the store doesn't keep the full history. Together they cover both needs. You could even add a time window on top of the seen-key for a different purpose —for example, allowing a legitimately repeated order (a real repurchase by the same customer) to be processed if it arrives much later, combining the key with a temporal component—. The question isn't "which of the three?", but "what do I need to answer, and which strategy answers each part?"

The false duplicate: over-deduplicating is also a bug. This whole lesson has worried about the duplicate that slips past you. There's a symmetric, less obvious mistake: discarding as a duplicate something that was actually new and legitimate. And its origin is almost always the same: you chose the wrong key. Think of it with a Cumbre case. If your idempotency_key were only the customer's name, then two different orders from the same customer —Luna Coffee bought on Monday and bought again on Thursday— would have the same key, and your system would discard the second purchase as if it were a duplicate. You didn't overcharge; you undercharged, and you silently lost a real sale. It's the exact mirror of the duplicate charge, and it's usually harder to detect because "nothing happened" looks like success.

The lesson here connects directly to module 2: the quality of your deduplication is the quality of your key. A good idempotency_key identifies the exact work —this order, with this content— so two triggers of the same work share a key and two genuinely different pieces of work have different keys. That's why in lesson 3 the synthetic key combined order_id with the total: so the same order always produces the same key, but a different order doesn't collide with it. When lesson 5's dedup store discards something, it's going to discard exactly what its key says is a duplicate —no more, no less—. Choosing that key carefully is what prevents both mistakes at once: the duplicate that slips through and the new one that gets discarded.

Common mistakes

Putting a Remove Duplicates node expecting it to catch the double trigger (practical). What happens: someone adds the node in "Within Current Input" mode convinced it's going to stop the webhook that fires twice, and duplicate charges keep showing up. Why it happens: that mode only sees the current execution's batch, and the two triggers are two separate executions that never share a batch. How to spot it: if your Remove Duplicates is in "within input" mode and you expect it to cross executions, that's the mistake. How to fix it: to cross executions you need mode B (with its limits) or, for a costly effect, lesson 5's own table.

Choosing a time window for a costly effect (conceptual). What happens: you deduplicate with a fifteen-minute window, it works in every test, and months later a manual reprocess brings back an old order and charges it again. Why it happens: the window assumes duplicates arrive close together, and this re-entry arrived days late. How to spot it: if your deduplication "forgets" keys after a while and the effect is a charge, you have this bomb set. How to fix it: for costly effects use the seen-key, which assumes nothing about time. The window is for low-risk automatic retries, not for money.

Trusting History Size with no thought for its edge (practical). What happens: mode B of Remove Duplicates gets used and everything's fine until, with volume, old keys start falling out of the 10,000-key store, and a late duplicate whose key already dropped out passes as new. Why it happens: the store is bounded and forgets the oldest entries when it fills up. How to spot it: if your volume of unique keys comfortably exceeds History Size and you still depend on it not to duplicate, this is it. How to fix it: for high volume and costly effects, your own table with no artificial limit (or with deliberate archiving) is safer, and on top of that you can query it.

Treating deduplication as a single technique (conceptual). What happens: someone learns one strategy —whichever it is— and applies it to everything. Why it happens: "deduplicating" sounds like a single thing. How to spot it: if you use the same technique to clean up an imported list and to avoid duplicating a charge, one of the two is probably poorly served. How to fix it: recognize they're a family; the right question is "what territory is this?" (single batch, close-together retry, costly effect with a possible late duplicate) and choose accordingly.

Deduplicating by discarding when you needed to branch (conceptual). What happens: Remove Duplicates gets used, the duplicate disappears, and afterward there's no way to know it existed or to respond to the system that sent it. Why it happens: the node, by design, makes the duplicate disappear. How to spot it: if you need to log the duplicate, respond "already processed," or alert when something repeats a lot, and your tool just deletes it, this is it. How to fix it: a table with ON CONFLICT gives you the explicit "first time / duplicate" branch and lets you do something different on each one.

Exercises

Exercise 1 — Assign the strategy. For each case, say which strategy you'd use and why, in one sentence.

(a) You import a 2,000-row CSV of orders and some are repeated within the file. (b) A payments webhook automatically retries after 30 seconds if you don't respond, and creating the charge is costly. (c) An order can come back in through a manual reprocess weeks later, and shouldn't be charged twice. (d) A low-risk internal flow that refreshes a contact list and doesn't want to reprocess ones already seen in recent executions, with no need to audit anything.

See solution

(a) Remove Duplicates, mode A (within input). The repeats are within a single execution's batch; it's exactly its case, and you don't need to cross executions.

(b) Seen-key in your own table (or, at minimum defensibly, a time window). The retry arrives close together, so a window would catch the case; but since the effect is costly, the seen-key is worth it, and it also covers an unexpected late duplicate. With money on the line, go with the robust option.

(c) Seen-key in your own table, no question. The duplicate arrives weeks later: any time window would let it through, and the node's History Size could have forgotten it. Only remembering the key with no expiration guarantees you don't charge twice.

(d) Remove Duplicates, mode B (previous executions). It crosses executions, the risk is low, you don't need to audit, and the volume fits within History Size. It's its ideal case: convenient and enough.

Why this works: notice the deciding factor isn't "how often it repeats" but when the duplicate arrives and how much a mistake costs. Single batch → mode A. Close together and cheap → window or mode B. Late or expensive → your own table. That's the whole criterion.

Exercise 2 — Break the window. A coworker implemented dedup with a ten-minute window for order-triage and says "it's gone a month with no duplicates, it's solved." Describe a concrete, realistic scenario where their solution would charge twice, and explain why the seen-key wouldn't have that problem.

See solution

A realistic scenario: Cumbre's CRM went down yesterday, and several orders didn't process correctly. Today, someone on the team resends yesterday's batch to reprocess the ones left pending. Among them is ORD-2041, which had been charged yesterday before the outage. That resend arrives more than ten minutes —actually, more than a day— after the original trigger. The ten-minute window already forgot ORD-2041, so it treats it as new and creates a second charge.

Why the seen-key doesn't fail there: the key ORD-2041 was stored in the table since yesterday, with no expiration. When the resend arrives today, the query finds the key and discards the order, no matter that 24 hours passed. The seen-key assumes nothing about time; the window assumes duplicates arrive close together, and this one didn't.

Why this works: the exercise shows "a month with no duplicates" doesn't prove the solution is correct, only that the late duplicate hasn't arrived yet. Time-window bugs are silent until the right scenario —a reprocess, a resent backup, an upstream outage— wakes them up.

Exercise 3 — Justify your own table. Your team asks: "the Remove Duplicates node in mode B already crosses executions, why set up a table in Postgres?" Write the answer you'd give, covering at least three reasons why, for a costly effect, your own table is preferable.

See solution

A reference version:

Mode B works for many cases, and if this were a low-risk flow I'd use it without hesitation. For a charge, I prefer my own table for three concrete reasons.

First, I can query and audit it. If a customer disputes a duplicate charge, with a table I run SELECT * FROM processed_orders WHERE order_id = 'ORD-2041' and see exactly when it got recorded. The node's internal store is a black box: I have nowhere to look when something looks off.

Second, I control the size and the expiration. Mode B has a bounded History Size (10,000 by default), and when it fills up it forgets the oldest keys. A late duplicate whose key already fell off would pass as new. In my table, I decide what I remember and for how long; there's no automatic forgetting I don't control.

Third, I can branch instead of just discarding. With ON CONFLICT I know whether it was the first time or a duplicate, and I can do something different in each case: log the duplicate in the ledger, respond to the webhook with "already processed," or alert if an order repeats too much. The node just makes the duplicate disappear; I lose that information.

And at bottom: the table is mine, it lives separate from the workflow, and every flow that needs it can see it. For the truth not charging twice depends on, I want that control, not a borrowed box.

Why this works: the answer doesn't rule out the node —it recognizes where it's the right choice— and raises the argument to what truly matters for a costly effect: auditing, control over forgetting, and the ability to branch. Those are the three things separating a convenient tool from a source of truth.

Summary and next step

In this lesson you saw deduplication is a family of three strategies, not a single technique. The time window deduplicates what arrives close together and forgets the old stuff: perfect against automatic retries, insufficient against a late duplicate. The seen-key remembers every key with no expiration and assumes nothing about time: it's the robust strategy for costly effects, and its price is a table that grows. And the Remove Duplicates node has two faces: mode A cleans up duplicates within a single batch but is blind across executions, and mode B does cross executions but as a bounded, coupled black box, useful for low-risk dedup and not as a source of truth.

The conclusion opening lesson 5: for the idempotency of a costly, irreversible effect like a charge, the right strategy is the seen-key in your own table, because it gives you control over forgetting, the ability to audit, and the possibility of branching between "first time" and "duplicate" —three things none of the packaged alternatives offer all at once—.

Before moving on you should be able to: name the three strategies and their territory; explain why the node's mode A doesn't catch the double trigger; and give a reason why your own table beats mode B for a charge.

Lesson 5 builds that table of your own and gives it its superpower: INSERT ... ON CONFLICT DO NOTHING, the pattern turning "check if it exists" and "record that it now exists" into a single atomic, indivisible step. That step is what finally closes the "check then act" trap module 2 left open and that neither Static Data nor the time window could resolve. There, the seen-key stops being an idea and becomes a mechanism proof against the double-trigger race.

Resources

• Remove Duplicates node — n8n Docs — the node's operations: "Remove Items Repeated Within Current Input" (mode A), "Remove Items Processed in Previous Executions" (mode B), and "Clear Deduplication History," plus the History Size parameter and node- or workflow-level scope. Confirm your version's options there.
• Postgres node — n8n Docs — the node lesson 5 uses to implement the seen-key in your own table, with the operation that queries and inserts.
• PostgreSQL — INSERT ... ON CONFLICT — the atomic mechanism lesson 5 uses for the seen-key; get a head start here if you want to see the official syntax.
• Understand n8n's data structure — n8n Docs — how items flow within an execution, useful for understanding why the node's mode A only sees the current batch.