3. Idempotency keys: natural vs. synthetic

Description

By the end of this lesson you'll be able to choose the correct idempotency key for an event —the tag that says "this order and that one are the same, not two different ones"— and you're going to know when to use a key that already comes in the data (natural) and when to manufacture one yourself (synthetic). You're going to compute a stable synthetic key with a hash inside a Code node, using the crypto module, and you're going to immediately recognize the mistake that ruins the whole mechanism: choosing a key that changes on every execution.

This matters because the key is the raw material for all idempotency. Lesson 4's upsert needs a key to know which column to avoid duplicating on. Lesson 5's header needs a key to send the gateway. Module 4's deduplication needs a key to remember what it already processed. If the key is chosen badly, none of those tools work —and the worst part is they seem to work in testing and fail in production, which is the most expensive way to fail—. Choosing the key well is, without exaggeration, half of this module.

Connection to the module: lesson 2 taught you to recognize an effect that isn't idempotent. This one gives you the first thing any repair needs: a stable name for the event. Everything that follows consumes it. Lesson 4 uses the key as the upsert's column; lesson 5 sends it in the Idempotency-Key header; lesson 6 shows why the key has to live inside the atomic operation and not in a separate node; lesson 7 derives it from an agent's decision; and lesson 8's project starts, precisely, by choosing ORD-2041's key.

What an idempotency key is

Let's start by defining it, because the name sounds more technical than it is.

An idempotency key is a value that uniquely identifies an event, such that two arrivals of the same event share the same key and two different events have different keys.

Think of it like a theater's coat check ticket. You arrive with your coat, they take it, and give you a ticket numbered 47. That number identifies your coat. When you return, you hand over the 47 and get back exactly your coat, not someone else's. If for some reason you hand in the 47 twice —say you got in the wrong line and got back in it— the coat check looks at the 47, sees that coat's already been handed back to you, and doesn't give you a second coat that doesn't exist. The ticket number is the key: same coat, same number; different coats, different numbers.

At Cumbre, order ORD-2041's idempotency key is what's going to let the payment gateway say "I already made this charge" when the webhook arrives a second time. It's the coat check's number 47. All of idempotency's machinery hangs off that number.

For a key to be useful, it has to satisfy three properties. It's worth naming them, because every mistake in this lesson is violating one of the three:

Deterministic. The same input always produces the same key. If you compute the key from the order, computing it a thousand times on the same order gives the same result a thousand times. Without this, nothing works.

Stable across retries. The key for the same event is identical on the first arrival, the second, and the fifth. This is the property that breaks most often, and almost always for the same reason: including something in the key that changes between arrivals, like the time it arrived.

Specific. Different events produce different keys. If two genuinely different orders end up with the same key, the system is going to believe the second is a duplicate of the first and discard it. That's a lost order, which is just as bad as a duplicate order.

A key that satisfies all three is a good coat check number. One that fails at any of the three is a source of bugs.

The natural key: the one that already came in the data

The simplest, most robust way to get a key is not manufacturing it: using one that already comes in the event. That's a natural key.

In Cumbre's order, the candidate is obvious:

{
  "order_id": "ORD-2041",
  "customer_id": "CUST-118",
  "amount": 1780,
  ...
}

The order_id —ORD-2041— is an identifier the online store assigned to the order when it was created. And here's the property that makes it gold: when the store retries the webhook, it sends the same order_id. It doesn't generate a new one; it resends the order as-is, with its ORD-2041 intact. That's why order_id satisfies all three properties without you doing anything: it's deterministic (it's a fixed value), stable across retries (the store sends the same one), and specific (every real order has its own).

When a good natural key exists, use it. Don't manufacture a synthetic one for fun. The natural key is simpler, more readable, and easier to debug: when you see a duplicate in production, being able to search for ORD-2041 in your logs is worth a lot more than searching for a sixty-character hash.

What makes a natural key "good"? That the source system assigns it once and keeps it across retries. A serious store's order_id does that. An invoice number, a bank's transaction id, an email's message-id, the event id many platforms include precisely for this —many webhook APIs send an event_id or a delivery_id that's the same on every retry, and exists precisely so you use it as an idempotency key—: all of these are excellent natural keys.

The question that decides whether a natural key is usable is a single one: is this identifier the same when the event arrives twice? If the answer is yes, you already have your key. If it's no, you need a synthetic one.

The synthetic key: the one you manufacture

Sometimes there's no good natural key. The event arrives with no stable identifier, or the identifier it carries changes between arrivals. That's when you manufacture a synthetic key: you compute it from the event's content.

Let's go back to Cumbre, but through another channel. Orders coming in through WhatsApp don't always carry an order_id —the customer filled out a simple form and the system didn't assign a folio—. An order like that can look like this:

{
  "channel": "whatsapp",
  "customer_id": "CUST-204",
  "amount": 940,
  "line_items": [
    { "sku": "TE-CHM-100", "quantity": 4 }
  ]
}

There's no order_id. If the WhatsApp webhook arrives twice, how do you know it's the same order and not two orders that happen to look alike? You need a key, and the only raw material you have is the content. You manufacture it with a hash.

A hash is a function that takes text of any size and returns a fixed-length string that acts as its "fingerprint." The property that serves us: the same input text always produces the same fingerprint, and different texts produce different fingerprints. If we build a text from the fields that define the order —the customer, the amount, the products— and run it through a hash, we get a key that's identical if the content is identical. Deterministic and stable: exactly what we asked for.

Worked example: computing a synthetic key with crypto

Let's compute it in a Code node. Remember the restriction you already know: inside n8n 2.0's Code node you can't make HTTP requests or touch the file system, and on n8n Cloud you only have two modules available: crypto and moment. It turns out crypto is exactly what we need for hashing —computing a fingerprint is a local calculation, not a call anywhere—, so this can indeed be done here.

Put a Code node after the Webhook, in Run Once for Each Item mode, with this code:

// Node: Code — "Compute idempotency_key"
// Mode: Run Once for Each Item
// Goal: give each order a stable key derived from its content.

const crypto = require('crypto');   // the hashing module; allowed inside the n8n Code node

const order = $input.item.json;

// I build a text with ONLY the fields that define "the same order".
// Careful: no arrival timestamps or random ids. Only stable content.
const fingerprint = [
  order.customer_id,
  order.amount,
  order.line_items.map((line) => `${line.sku}x${line.quantity}`).join(','),
].join('|');

// sha256 returns the fingerprint; 'hex' formats it as readable 64-character text.
const idempotencyKey = crypto
  .createHash('sha256')
  .update(fingerprint)
  .digest('hex');

return {
  json: {
    ...order,                       // keep everything that already came in
    idempotency_key: idempotencyKey, // and add the computed key
  },
};

Let's break down the pieces, because every line has a reason.

require('crypto') brings in the hashing module. It's one of the two exceptions the Code node allows; crypto doesn't go out to the internet, it only does math on what you give it.

fingerprint is the text that summarizes the order. And here's the most important decision in the whole piece of code: which fields I include. I put customer_id, amount, and a representation of the lines (TE-CHM-100x4). I didn't put the arrival time, the channel, or any value that could differ between the first and second arrival of the same order. The rule is: include what makes the order identical, exclude what can vary between retries.

createHash('sha256').update(fingerprint).digest('hex') is the computation. sha256 is a standard, trusted hash algorithm; update feeds it the text; digest('hex') asks for the result in hexadecimal format, a 64-character string like a3f1c.... The same fingerprint always produces the same hash.

What to expect when you run it. The OUTPUT panel shows the order with a new field, idempotency_key, with a value like "a3f1c9e2...". (64 characters). If you re-run the node with the same input order, the key is exactly the same. If you change any of the fields that go into the fingerprint —the amount, a line's quantity— the key changes completely. That's the proof the key is deterministic and specific: same content, same key; different content, different key.

That idempotency_key is what you're going to pass to the upsert (lesson 4) and to the API header (lesson 5). You just manufactured the coat check's number.

The fine-grained decision: which fields go into the hash

There's a trap here worth paying attention to, because it's where even experienced people get it wrong.

The hash defines what "the same order" means. If you include too much, you break stability. If you include too little, you break specificity. It's a balance.

Imagine you'd included created_at (the arrival time) in the fingerprint. The first arrival carries 09:12:00; the retry, three seconds later, carries 09:12:03. Different texts, different hashes, different keys for the same order. The gateway would see two new events and you'd be back to two charges. Including the arrival time is the classic mistake, and the next section covers it in depth.

Now the opposite problem. Imagine Cumbre has a customer who legitimately places the same order twice on the same day —CUST-204 orders four chamomile teas in the morning and another four in the afternoon, two real, distinct orders—. If your fingerprint only has customer_id, amount, and lines, the two orders produce the same key, and your system would discard the second, believing it's a duplicate. You just cost Cumbre a sale. Here you included too little: you're missing something that distinguishes two real orders with the same content.

How do you solve it? It depends on what information you have. If the WhatsApp form includes a session or message identifier that's unique per real submission but stable on retry, that's your best friend: include it. If there's genuinely nothing that distinguishes two identical orders, you have to negotiate with business reality —maybe a coarse time window ("same customer, same content, within the same hour = same order") is acceptable, accepting the small risk of merging two genuine, closely-spaced orders—. There's no universal answer; there's a design decision you make with knowledge of the business. What this lesson gives you is awareness that the decision exists.

The honest conclusion: a good natural key saves you all of this agony. That's why, when the event carries a stable order_id, it's preferred. The synthetic key is for when there's no other option, and then the quality of your key is only as good as your choice of fields.

Where the key lives in the workflow

A practical detail worth pinning down before moving on: at what point in order-triage does the key get computed, and how does it reach the nodes that use it?

The key is computed once, as early as possible, and then travels with the item toward the nodes that need it. In order-triage it looks like this:

Webhook  ──►  Code               ──►  AI Agent  ──►  HTTP Request
              (computes                (classifies)   (uses idempotency_key
               idempotency_key                          in the upsert and
               and adds it to the item)                 the header)

Notice three decisions in this design.

It's computed early, right after the Webhook. That way, everything that comes after —the agent, the HTTP Request— already has the key available in the item. Computing it at the end, right next to the effect, works, but computing it early leaves the data ready for any node that needs it, including ones you'll add later.

It's computed once, not in every node. If two different nodes computed the key on their own, you'd risk one including a field the other doesn't, and you'd end up with two different keys for the same order —the silent disaster—. A single source of truth: one node computes it, the rest read it.

It travels as one more field of the item. By doing { ...order, idempotency_key: idempotencyKey }, the key becomes part of the order flowing through the workflow. Any later node reads it with a normal expression, for example {{ $json.idempotency_key }} in an HTTP Request field. There's no magic; it's a field like order_id or amount.

There's a subtlety lesson 6 is going to develop, so I'll plant it here: computing the key in one node and using it in a different node is fine for deriving the key, but it doesn't, by itself, make the operation idempotent. Real idempotency happens when that key gets applied inside an atomic operation —the database upsert, the header the API processes atomically—. Computing the key is step 1; applying it in the right place is what truly protects. For now, hold on to this: the key gets computed early and travels with the item.

The mistake that ruins everything: the key that changes every time

If you take away a single warning from this lesson, let it be this one, because it's the most common and the most expensive mistake.

Never use, as an idempotency key, a value that changes on every execution. The two usual suspects:

The timestamp. new Date(), Date.now(), the current time, created_at if it's generated at processing time instead of at order creation time. Every execution happens at a different instant, so the key is different every time. The retry three seconds later has a new key, the system sees it as new, and you duplicate. A timestamp is the opposite of a stable key: it's a value literally designed to be different every time.

A new random identifier per execution. crypto.randomUUID(), a random number, an id you generate on the fly. A freshly generated UUID is unique by definition —that's what it's for— so two executions of the same event produce two different UUIDs and two different keys. It's the same mistake in a different costume.

The symptom is cruel: it works in testing and fails in production. When you test, you fire the event once, you see a charge, everything's fine. The "changing" key never gets put to the test because there was never a second trigger with "the same" key —in fact there's no "the same" key, every trigger has its own—. The day the provider retries in production, the new key doesn't match anything, and the duplicate you swore your test ruled out shows up.

The mental rule to avoid falling into it: ask yourself "if this same event arrived again in five seconds, would my key be identical?". If your key depends on the time or on something random, the answer is no, and you have a bug waiting. If it depends only on the event's stable content, the answer is yes, and you're safe. This question is so cheap to ask and so expensive to skip that it's worth turning into a reflex: every time you define a key, say it out loud before moving on.

Notice the useful irony: crypto.randomUUID() is for generating unique identifiers, and crypto.createHash() is for generating stable identifiers. The same module, two functions, opposite purposes. For an idempotency key you want stability, so you want createHash over the content, never randomUUID.

Common mistakes

Putting the timestamp into the key (practical). What happens: someone builds the key including created_at, Date.now(), or the processing time, tests it with a single trigger, it works, and it ships to production. Weeks later, duplicate charges show up that "shouldn't exist." Why it happens: the timestamp is different on every arrival, so the retry generates a new key the system doesn't recognize as a repeat. And since manual testing almost never fires the same event twice seconds apart, the bug doesn't show up until the provider retries in production. How to spot it: read your fingerprint or your key and look for anything that depends on the clock; if it's there, you have the bug. It also helps to ask "would it be identical if the event arrived again in five seconds?". How to fix it: remove everything time-related from the computation. The key derives only from the content that identifies the event —the natural order_id, or a hash of the stable fields—, never from when it arrived.

Generating a new UUID per execution (practical). What happens: in the Code node, someone writes const key = crypto.randomUUID() thinking "I need a unique identifier for the charge." Every execution produces a new UUID, so idempotency never kicks in. Why it happens: "unique" gets confused with "stable." A charge does need an identifier that's unique in the world, but the idempotency key needs to be the same across retries of the same event. They're opposite requirements, and randomUUID satisfies the first, not the second. How to spot it: if randomUUID(), Math.random(), or any source of randomness shows up in the key computation, it's wrong. How to fix it: replace the random value with a deterministic hash of the content (createHash('sha256')') or, better yet, with the order's natural key if it exists. Save randomUUID` for when you truly need a new identifier —which isn't this case—.

Using a natural key the source doesn't keep stable (practical). What happens: someone picks, as a natural key, a field that looks stable but that the source system regenerates on every send —for example, a request_id that the sending platform creates fresh for every delivery attempt, not per event—. It sounds like a natural key, but it changes between retries. Why it happens: not every identifier that comes in the payload is stable; some identify the delivery, not the event, and those change exactly when it retries. How to spot it: don't assume; check the provider's documentation for whether the field is "the same across retries." Many platforms say so explicitly and even separate an event_id (stable) from a delivery_id (changes per attempt). How to fix it: use the identifier the provider guarantees stable across retries —often called event_id or similar—; if none is guaranteed, fall back to a synthetic content-based key.

Exercises

Exercise 1 — Natural or synthetic? For each event, decide whether you'd use a natural key (and which field) or a synthetic one (and which fields you'd hash), and justify in one sentence:

(a) A Cumbre web order carrying order_id: "ORD-2041", and you know the store resends the same order_id on retries. (b) A WhatsApp form with no order_id, with customer_id, amount, and line_items. (c) A payment gateway webhook carrying an event_id field that, per its documentation, is identical across all retries of the same notification. (d) An event carrying request_id, whose documentation says that id is different on every delivery attempt.

Exercise 2 — Find the poison in the hash. A coworker wrote this key computation for WhatsApp orders. It has a problem that will make idempotency never work. Find it and fix it:

const crypto = require('crypto');
const order = $input.item.json;

const fingerprint = [
  order.customer_id,
  order.amount,
  new Date().toISOString(),   // "so the key is unique"
].join('|');

const idempotencyKey = crypto.createHash('sha256').update(fingerprint).digest('hex');

const fingerprint = [
  order.customer_id,
  order.amount,
  order.line_items.map((line) => `${line.sku}x${line.quantity}`).join(','),
].join('|');

Exercise 3 — Design the key for the three channels. Cumbre receives orders through three channels with data of different quality. For each one, propose a key strategy and explain the main risk you have to watch for:

• web: carries a stable order_id.
• whatsapp: no order_id, carries customer_id, amount, line_items, and a session_id the app guarantees stable per submission.
• rep_csv: a file salespeople upload, with rows that sometimes repeat within the same file by human error, with no stable id at all.

Summary and next step

In this lesson you learned that all of idempotency hangs off a key: a value that makes two arrivals of the same event share an identity and two different events not share one. You pictured it as the coat check's number —same coat, same number— and you gave it three properties: deterministic, stable across retries, and specific. You saw the two ways of getting it: the natural one, an identifier that already comes in the data and that the source keeps across retries (Cumbre's order_id, a webhook's event_id), which is always the first choice when it exists; and the synthetic one, a hash of the content you manufacture with crypto.createHash('sha256') in a Code node when there's no good natural key —taking care with the fine-grained decision of which fields go into the hash, so as not to break stability or specificity—. And you burned in this module's most expensive mistake: using a timestamp or a new UUID per execution, which produces a different key every time, works in testing, and duplicates in production.

Before moving on to lesson 4 you should be able to: choose between a natural and a synthetic key based on whether the event carries a stable id; compute a synthetic key with a hash while including nothing time-related; and explain why randomUUID() is exactly what you don't want for an idempotency key.

You now have the coat check's number. Lesson 4 uses it for the first thing: not duplicating in your own database. You're going to learn about the upsert —"insert if it doesn't exist, update if it does, but never duplicate"— which turns a table into a set by key, and you're going to see how the database node and the spreadsheet node express it in n8n. It's the tool that fixes Cumbre's blind CRM INSERT, and the one you're going to use the most in your real life with workflows.

Resources

• Crypto — Node.js documentation — the reference for the crypto module you use in the Code node: createHash, update, digest, and their options. It's standard Node.js, available inside n8n's Code node.
• Code node — allowed modules — n8n Docs — which built-in and external modules are available inside the Code node; confirms that crypto and moment are the ones you have on n8n Cloud.
• Idempotency — Stripe API reference — how a real API describes what a good idempotency key should be (stable, unique per operation) and how long it remembers it. Always verify the current conditions.
• Webhooks best practices — n8n Docs — the node that receives the event you pull the natural key from; useful for understanding which fields arrive and which are worth inspecting before choosing the key.