6. The outbox pattern: deciding and executing separately

Description

By the end of this lesson you will be able to explain and apply the outbox pattern, the technique that makes this module's entire coordination robust. The idea, in one sentence: instead of deciding an effect and executing it in the same step, you separate the two — you record the effect's intent in an outbox table, in the same atomic operation where you register your decision, and a separate flow reads that table and executes the effect idempotently. You will understand the exact problem this solves — the dual-write problem, which no retry on its own can fix — you will see the anatomy of the outbox table and the two workflows that use it (the one that decides and the one that executes, called the relay), and you will be able to reason through what happens if there's a crash at each point in the chain and why the effect is neither lost nor duplicated at any of them.

This matters because it's the missing piece. Up to now you know how to make an effect idempotent (Module 2), you have a ledger (Module 4), you know how to split and join (lesson 4) and control the pace (lesson 5). But there's still a crack none of those pieces covers alone: the instant between "I decided an effect needs to happen" and "the effect happened." If the system crashes in that instant, did the effect happen or not? Will it happen on retry, or did it get lost? Or will it happen twice? The outbox is the answer to that question, and it's what turns coordination that "almost always works" into coordination that's correct by construction.

Connection to the module: this lesson pulls everything before it together. The receive/process decoupling that peeked through in lesson 5 is, formalized, the outbox. Module 2's idempotency is what makes the relay's execution safe. Module 4's ledger is, many times, the very same table as the outbox. Lesson 3's graph gains a new arrow shape — "record in the outbox" instead of "call and wait" — that cuts cascades. And lesson 7 is going to apply exactly this pattern to the effects an agent triggers. If there's one lesson in this module to master, it's this one.

The problem no retry fixes on its own

Let's name the crack precisely, because it's subtle and it's the pattern's whole reason for being.

When order-triage decides to issue a refund, deep down it has to do two things: (1) register in its own system that the order is now refunded — change a state, in the ledger or the CRM — and (2) actually execute the effect — call the payment gateway to return the money. Two writes: one in your own house (your database) and one in someone else's house (the gateway). And here's the problem: there is no way to make those two writes a single atomic operation. Your database and the payment gateway are two separate worlds; you can't wrap them in a single transaction that does both or neither.

This problem is called the dual-write problem, and it has only two possible orderings, both broken:

Order A — change the state first, then execute the effect.

1. UPDATE: mark the order as "refunded" in my database   ✓
   ─── the system crashes here ───
2. HTTP: call the gateway to issue the refund               ✗ never happens

If I crash between step 1 and step 2, my database says "refunded" but the money was never returned. The effect got lost. And the worst part: since my state says "refunded," no retry is going to try it again — the system believes it's already done. The customer never gets their money and the system swears it's complete.

Order B — execute the effect first, then change the state.

1. HTTP: call the gateway to issue the refund               ✓ the money was returned
   ─── the system crashes here ───
2. UPDATE: mark the order as "refunded" in my database    ✗ never happens

If I crash between step 1 and step 2, the money was returned but my database doesn't know it. Since my state doesn't say "refunded," the retry is going to issue the refund again: the effect gets duplicated. The customer receives the money twice.

Read that slowly, because it's the heart of the lesson: there is no good order. Either one of the two, given a crash at the exact right moment, breaks something — A loses the effect, B duplicates it. And no retry fixes this on its own, because the retry doesn't know at what point you crashed. This is the gap the previous lessons left open, and the outbox is what closes it.

The outbox idea: separating deciding from executing

The solution is clever and, once you see it, obvious. The problem was that "change my state" and "execute the other side's effect" can't be made atomic together. So: don't do them together. Split the work into two moments:

1. Decide (atomic, all in my own house). When order-triage decides to issue the refund, it makes two writes that are both in my database and therefore do fit into a single atomic transaction: it changes the order's state and records, in a table called outbox, a row that says "a refund needs to be issued for ORD-2041." Both writes are in Postgres, my house, so either both happen or neither does. I don't call the gateway yet. I only register the intent.

2. Execute (separate, idempotent, retryable). A separate flow — the relay, or dispatcher — reads the pending rows from the outbox, and for each one executes the real effect — calls the gateway — idempotently, and marks the row as done.

Notice what got gained. The decision — "this order gets refunded" — is now recorded atomically, together with the effect's intent, all in my own house. There's no instant where "I decided to refund" exists without "the refund needs to be issued" also existing: they're born together or not at all. And the effect's execution is now separate, in a flow that can retry as many times as needed with no danger, because it's idempotent. The dual-write problem dissolved: there are no longer two writes in different worlds that both need to be atomic; there's one atomic write in my own house (deciding) and one idempotent execution off to the side (executing).

The analogy: the restaurant order rail

Imagine a busy restaurant kitchen. The waiter takes your order. What do they do? They don't run to the kitchen to cook your dish themselves. They write the order on a ticket — a slip of paper — and pin it to a rail hanging in front of the kitchen. That gesture, write-and-pin, is a single one: the order is recorded and hung in the same motion. Then the waiter goes to attend another table.

The kitchen works the rail at its own pace. It takes the oldest ticket, prepares the dish, and when it goes out, it takes the ticket off the rail or marks it as done. If the kitchen gets swamped, tickets pile up on the rail — but none of them get lost, they're all hanging there. If the cook loses track and isn't sure whether they already prepared a dish, they check the rail: if the ticket is still pinned, it isn't done; if it's gone, it already went out. And every ticket has a number, so even if two cooks look at the rail, they don't prepare the same dish twice: the number says which one is which.

That kitchen is the outbox pattern in full:

• The waiter is the workflow that decides (order-triage). It doesn't execute the effect; it writes the ticket (writes to the outbox) in the same gesture as recording the order.
• The rail is the outbox table. Effect intentions hang there, pending, without getting lost even if the kitchen falls behind.
• The kitchen is the relay: it takes the pending tickets and executes them at its own pace, marking them done as it finishes.
• The ticket number is the idempotency key: it guarantees no dish gets cooked twice, even if the relay retries.

And the key point of the analogy, the one that resolves the dual write: writing the ticket and executing it are two separate moments. The waiter never gets stuck between "I took the order" and "the dish is ready," because their job ends the moment they pin the paper. The kitchen, for its part, never duplicates a dish, because it works off a rail where every ticket has its number. Separating the one who takes the order from the one who cooks is what makes the whole restaurant robust.

Anatomy of the outbox table

The outbox table is the rail. Its typical shape, in Postgres:

CREATE TABLE outbox (
  id               BIGSERIAL PRIMARY KEY,       -- the ticket number
  aggregate_id     TEXT NOT NULL,               -- what it refers to: the order_id
  effect_type      TEXT NOT NULL,               -- which effect: 'issue_refund', 'inventory_sync'
  payload          JSONB NOT NULL,              -- the data the effect needs
  idempotency_key  TEXT NOT NULL UNIQUE,        -- the key that prevents duplicating the effect
  status           TEXT NOT NULL DEFAULT 'pending',  -- pending | processing | done | failed
  attempts         INT  NOT NULL DEFAULT 0,     -- how many times execution was attempted
  last_error       TEXT,                        -- the last error, if it failed
  created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
  processed_at     TIMESTAMPTZ                  -- when it was marked done
);

Let's go through the columns that matter:

• aggregate_id is what the effect refers to: the order_id. It lets you query "all the pending intentions for order ORD-2041."
• effect_type is which effect needs to run. The same outbox table can carry intentions of several types — refunds, inventory syncs, emails — and the relay decides what to do based on this field.
• payload is the data the effect needs: the amount to refund, the units to discount. It gets saved at the moment of deciding, so the relay doesn't have to go fetch it.
• idempotency_key is the ticket number: Module 2's key that guarantees the effect doesn't run twice. Marking it UNIQUE is an extra defense: the database itself rejects two identical intentions.
• status is where each ticket stands: pending (just written, not executed), processing (the relay is working on it), done (finished), or failed (failed and needs attention). The relay moves through these states.
• attempts and last_error are for operations: how many times it was attempted and what went wrong, useful for Module 6's lesson on retries and alerting.

A continuity note that saves work: many times the outbox and the ledger are the same table, or sibling tables. Recording "I decided to refund order ORD-2041" in the ledger is writing the ticket. You don't always need a separate table; sometimes the ledger, with a status column, acts as the outbox. Start simple: if your ledger already registers decisions, add a status to it and you already have the rail.

The two workflows: the one that decides and the relay

The pattern lives in two workflows. Let's build them for Cumbre's refund.

Workflow 1 — The one that decides (order-triage)

When order-triage determines an order needs a refund, it does not call issue-refund. Instead, it makes an atomic write: it registers its decision and writes the ticket to the outbox, all in a single Postgres operation.

Atomicity is the delicate point in n8n, so let's be concrete. The Postgres node executes SQL queries; for the two writes to be atomic, they go in a single statement or transaction. If the ledger and the outbox are the same table, it's one insert — trivially atomic. If they're separate tables, they get wrapped in a transaction. A clean form with a single statement, using the ledger as the source and the outbox as the destination:

-- Postgres node in order-triage: decide the refund (atomic)
-- Registers the decision in the ledger AND inserts the ticket into the outbox,
-- in a single transaction. Either both happen or neither does.

WITH decision AS (
  INSERT INTO ledger (order_id, effect, status, decided_at)
  VALUES ('ORD-2041', 'refund', 'decided', now())
  ON CONFLICT (order_id, effect) DO NOTHING     -- idempotent: if already decided, does not repeat
  RETURNING order_id
)
INSERT INTO outbox (aggregate_id, effect_type, payload, idempotency_key, status)
SELECT 'ORD-2041', 'issue_refund',
       '{"amount": 2154.00, "currency": "MXN"}'::jsonb,
       'refund:ORD-2041',                        -- the effect's idempotency key
       'pending'
FROM decision;                                   -- only inserts the ticket if there was a new decision

Read what that query does, because it holds all the robustness inside it:

• The first part registers the decision in the ledger with ON CONFLICT DO NOTHING: if the order had already been decided to refund, it does nothing — idempotency in the decision, so two triggers of order-triage don't write two tickets.
• The second part inserts the ticket into the outbox only if there was a new decision (the FROM decision). If the decision already existed, no ticket gets inserted.
• The two parts are in a single statement, so they're atomic: either the decision gets registered and the ticket gets written together, or nothing happens. One never ends up without the other.

And crucially: order-triage stops here. It didn't wait for the gateway, it didn't execute the effect, it can't crash "between deciding and executing" because for it there is no "executing." It wrote the ticket and moved on. If it crashes right after, the ticket is already pinned to the rail, safe.

What to expect. When you run order-triage with an order that requires a refund, you'll see the Postgres node insert a row into outbox with status = 'pending'. Query the table: there's the ticket, waiting. issue-refund hasn't run yet, and the money hasn't moved yet. That's correct: the decision is durably recorded, the execution will come later.

Workflow 2 — The relay (the one that executes)

The relay is a separate workflow that runs on its own, with a Schedule Trigger — every few seconds, or whatever pace fits. Its cycle is: take the pending tickets, execute each effect, mark done. Its structure:

# Workflow: outbox-relay (Schedule Trigger, every N seconds)

1. Postgres: take the pending tickets
     SELECT * FROM outbox
     WHERE status = 'pending'
     ORDER BY created_at            -- oldest first: preserves order
     LIMIT 10
     FOR UPDATE SKIP LOCKED;        -- so two relays don't take the same ticket

2. (for each ticket) mark it 'processing'
     UPDATE outbox SET status = 'processing', attempts = attempts + 1
     WHERE id = {{ $json.id }};

3. HTTP Request: execute the real effect
     POST to the payment gateway, with Idempotency-Key = {{ $json.idempotency_key }}
     body = {{ $json.payload }}

4. Postgres: mark the ticket 'done'
     UPDATE outbox SET status = 'done', processed_at = now()
     WHERE id = {{ $json.id }};

Three details make this correct and not just "a loop that calls the gateway":

FOR UPDATE SKIP LOCKED in step 1. If you run more than one relay (for capacity), this guarantees two relays don't take the same ticket: the first one locks it, the second one skips it. It's the database version of "two cooks don't take the same ticket off the rail."

The Idempotency-Key in step 3. The effect gets executed with the key you saved when writing the ticket (refund:ORD-2041). This is what you learned in Module 2: the gateway, receiving the same Idempotency-Key twice, executes the effect only once. It's the safety net for the case coming up next.

The order of steps 3 and 4. Notice: first the effect gets executed (3), then it's marked done (4). Isn't that the broken "Order B" we saw at the start — effect, then state? Yes, it is. But now it's protected by the effect's idempotency, and that changes everything. Let's see it.

Why the relay doesn't duplicate: the crash at the worst moment

The dangerous moment for the relay is between step 3 (executed the effect) and step 4 (marked done). If the relay crashes there, the ticket is left in processing, the refund was already issued, but the table doesn't say done. On the next cycle, what happens?

It depends on how you pick up tickets. If the relay also recovers ones that have been in processing too long (assuming they crashed), it's going to re-execute step 3's effect — it's going to call the gateway again. And here the safety net kicks in: since it uses the same Idempotency-Key (refund:ORD-2041), the gateway recognizes it's the same refund it already issued and does not issue it again. It returns the first one's result. The relay receives that response, and now it marks done. Result: the refund got issued exactly once, even though the relay executed the effect twice.

That's the magic of the pattern, and it's worth spelling out fully: the relay delivers the effect "at least once," and the effect's idempotency turns that into "exactly once." The relay can crash, retry, execute the same effect several times — and since the effect is idempotent by its key, the real-world outcome happens exactly once. The outbox guarantees the effect isn't lost (the ticket stays on the rail until marked done); idempotency guarantees it isn't duplicated (the key deduplicates it). Both together close the crack.

Compare this to the problem at the start. Before, we had two writes in different worlds that couldn't be atomic, and any order broke. Now: the decision is atomic (one write in my own house), and the execution is idempotent (it can repeat with no damage). Neither part has the dual-write problem, because we split them right where it hurt.

What crashes, and why nothing breaks at any point

It's worth walking through the whole chain and seeing what happens if the system crashes at each point. This table is the proof that the pattern is correct by construction, not by luck:

Walk through it and notice that at no row is the effect lost or duplicated. It can get delayed — the ticket waits on the rail — but it doesn't get lost, because it was durably recorded at the moment of deciding. And it can get retried — the relay might execute the effect more than once — but it doesn't get duplicated, because the idempotency key deduplicates it. That's the definition of robust coordination: not that nothing fails, but that no failure breaks correctness.

How the outbox resolves the three disasters

This closes the module's circle. The three disasters from lesson 1, all attacked by this one pattern:

• The cascade: order-triage no longer waits on the gateway — it writes the ticket and moves on. A slow payment service no longer blocks order intake; it just makes tickets pile up on the rail, where they wait without causing damage. The relay drains them at its own pace. The outbox is a buffer, just like lesson 5's Redis queue.
• Lost ordering: the relay processes tickets ORDER BY created_at — oldest first — so it can apply effects in the order they were decided, even under load. And if you combine it with lesson 5's version, order is fully shielded.
• The duplicated effect: resolved by the relay's execution idempotency, as we just saw. Two triggers of order-triage don't write two tickets (the decision is idempotent); two executions of the relay don't issue two refunds (the key is idempotent).

One pattern, all three disasters. That's why this is the hinge lesson: it isn't one more technique, it's the one that integrates everything before it into a coordination that holds together.

Common mistakes

Executing the effect in the deciding workflow, "to keep it simple" (conceptual). What happens: someone builds order-triage so that, upon deciding on the refund, it calls the gateway directly in the same flow — no outbox — because "one flow is simpler than two." It works in the demo. In production, the first crash between the state change and the call breaks something: it either loses the refund or duplicates it, depending on the order. Why it happens: two workflows seem more complex than one, and the dual-write problem doesn't feel real until a crash makes it materialize. How to detect it: if your flow has a state change (an UPDATE, a ledger entry) and a call to an external effect in the same execution, you have the latent dual-write problem. How to fix it: separate them — decide and write the ticket atomically in one flow; execute the effect in the relay; the apparent simplicity of a single flow is debt that gets paid off on the first crash.

Writing the ticket and changing the state in two separate operations, with no atomicity (practical). What happens: someone places one Postgres node that changes the state, and another Postgres node that inserts the ticket, as two steps of the flow. If the flow crashes between the two nodes, you're left with a changed state and no ticket (lost effect) or a ticket with no state (phantom effect). The outbox did nothing, because its premise — the atomic write — got broken. Why it happens: in n8n it's natural to put one operation per node, and "two writes" reflexively translates to "two nodes." How to detect it: if the state change and the ticket insertion are in different nodes, they aren't atomic. How to fix it: the two writes go in a single statement or transaction — one single Postgres node with a query that does both, like this lesson's WITH ... INSERT, or a single insert if the ledger and the outbox are the same table; the atomicity of the decision is the pattern's premise, not a detail.

Doing the relay's effect without an Idempotency-Key (practical). What happens: the relay gets built with the outbox and all, but the HTTP Request to the gateway sends no idempotency key. Everything's fine until the relay crashes between executing and marking done, retries, and issues the refund twice, because the gateway had no way of knowing it was the same one. The outbox guaranteed it wouldn't get lost, but without the key it didn't guarantee it wouldn't get duplicated. Why it happens: the outbox feels like "the complete solution," and it's easy to forget half its robustness depends on the execution being idempotent. How to detect it: if the relay's effect can execute twice (and it can, by design) and carries no idempotency key, it's going to duplicate. How to fix it: the key you saved on the ticket (idempotency_key) travels with every call of the effect — as Idempotency-Key in the header for an API, or as the key checked against the ledger for an internal effect; the outbox and idempotency are a team, neither works alone.

Leaving tickets stuck in processing with no recovery (practical). What happens: the relay marks a ticket processing, crashes before finishing, and that ticket stays in processing forever — neither pending for a retry, nor done. The effect got left halfway and no one picks it back up. Why it happens: the processing state protects against two relays taking the same ticket, but if the relay dies right there, nothing returns it to pending. How to detect it: query tickets in processing with a high attempts count or a long time since they were taken; those are stuck. How to fix it: the relay, when picking up tickets, should also recover ones that have been in processing too long (assuming they crashed and retrying them) — safe, because the effect is idempotent — and ones that fail repeatedly go to failed for an alert, a Module 6 subject. A ticket should never be able to stay silent forever.

Exercises

Exercise 1 — Explain why there's no good order. Without the outbox, order-triage has to change its state and call the gateway. Write, for each of the two possible orders, what breaks if the system crashes at the exact point between the two operations, and why the retry doesn't fix it.

Exercise 2 — Trace the relay's crashes. The relay executes: (3) calls the gateway with Idempotency-Key, (4) marks done. For each of these two crash moments, say the state of the world and what happens on recovery, and confirm the refund happens exactly once:

(a) The relay crashes right before step 3 (ticket in processing, effect not executed). (b) The relay crashes right after step 3 and before step 4 (effect executed, ticket still in processing).

Exercise 3 — Design the inventory ticket. Cumbre wants to apply the outbox to inventory-sync too: when order-triage decides to discount inventory for a three-line order, it must write the corresponding tickets atomically along with the decision. Design what tickets get written (how many, and with what effect_type, aggregate_id, idempotency_key, and payload), and explain how the relay executes them without duplicating if it crashes halfway.

outbox:
  { aggregate_id: 'ORD-2041', effect_type: 'inventory_sync',
    idempotency_key: 'inventory:ORD-2041:CF-ARA-500',
    payload: {sku:'CF-ARA-500', quantity:12}, status:'pending' }
  { aggregate_id: 'ORD-2041', effect_type: 'inventory_sync',
    idempotency_key: 'inventory:ORD-2041:TE-CHM-100',
    payload: {sku:'TE-CHM-100', quantity:6}, status:'pending' }
  { aggregate_id: 'ORD-2041', effect_type: 'inventory_sync',
    idempotency_key: 'inventory:ORD-2041:CF-DEC-250',
    payload: {sku:'CF-DEC-250', quantity:4}, status:'pending' }

Summary and next step

In this lesson you learned the outbox pattern, the module's hinge. It's born from a problem no retry fixes on its own: the dual-write problem — changing your state and executing an external effect are two writes in different worlds that can't be made atomic together, and either of the two orders, given a crash at the exact right moment, either loses the effect or duplicates it. The solution is to separate deciding from executing: the workflow that decides registers its decision and writes the effect's intent to an outbox table, in a single atomic operation in its own database — the waiter pins the ticket to the rail; a separate flow, the relay, reads the pending tickets and executes each effect idempotently, marking them done — the kitchen works the rail. That way the effect doesn't get lost (it stays durable on the rail) and doesn't get duplicated (the idempotency key deduplicates it even if the relay retries): "at least once" in delivery plus idempotency in execution gives "exactly once" in the real world. And you saw that a single pattern attacks all three disasters: it cuts the cascade (the decider doesn't wait), it preserves order (the relay processes by age), and it eliminates the duplicate (idempotent execution).

Before moving on to lesson 7 you should be able to: explain the dual-write problem and why no order resolves it; describe the outbox's two parts (atomic decision, idempotent execution) and what each one guarantees; and trace what happens if the system crashes at each point in the chain, confirming the effect happens exactly once.

Lesson 7 takes all of this into the territory of agents. When an AI Agent delegates work to another — as you saw in the chatbots guide, with one agent connected as a tool of another — every action an agent triggers is still an effect, and everything in this module applies: two agents can fire the same effect without knowing it, a delegation loop can repeat work, and the solution is once again the usual one — idempotent effects, the ledger as shared memory, and dangerous effects routed through an outbox. You'll see how to put those brakes on without breaking the autonomy that makes an agent useful.

Resources

• Postgres node — n8n Docs — the node you use to write the decision and the ticket atomically (one statement or transaction), and the one the relay uses to pick up tickets with FOR UPDATE SKIP LOCKED.
• Schedule Trigger — n8n Docs — the relay's trigger, which checks the outbox every so often and drains the pending tickets.
• HTTP Request node — n8n Docs — the node the relay uses to execute the real effect, sending the Idempotency-Key that guarantees the gateway won't duplicate it. Remember the Code node has no HTTP capability; the effect goes through here.
• Execute Sub-workflow node — n8n Docs — an alternative to HTTP Request when the effect is executed by another one of your workflows (issue-refund, inventory-sync) instead of an external API; the relay calls it the same way, with the idempotency key in the assignment.
• SQL INSERT ... ON CONFLICT — PostgreSQL Docs — the conditional write that makes the decision idempotent (ON CONFLICT DO NOTHING), so two triggers of the decider don't write two tickets.