4. Fan-out and fan-in without losing items

Description

By the end of this lesson you will be able to split a piece of work into several branches or sub-executions — what's called fan-out — and join their results back together in one place — the fan-in — without losing or duplicating items along the way. You will know the two ways to do fan-out in n8n (branching inside one execution, or firing a sub-execution per item), you will know how the Merge node does the fan-in and why joining by key is safer than joining by position, and you will understand the danger that gives the lesson its name: the partial retry, when out of N branches some finished and others didn't, and retrying re-runs the ones that were already done. And you will see how Module 4's ledger — the shared whiteboard — is exactly what covers that gap.

This matters because fan-out is one of those things that look best in a demo and behave worst in production. Splitting a ten-line order into ten inventory discounts works beautifully when all ten succeed. The problem shows up the day line seven fails: what happened to the other nine? Did they go through? Are they going to run again if I retry? Did line seven get lost? Without a clear answer to those questions, a fan-out is a factory for duplicated effects and lost work. This lesson gives you the answer.

Connection to the module: lesson 3 taught you to draw the graph; fan-out is a concrete arrow shape — one box that opens into many — and fan-in is where those arrows converge back together. In Cumbre's graph, when order-triage discounts inventory line by line, that's a fan-out. The partial retry we'll see here is a specific case of lesson 1's duplicated effect, and its cure is the same one that will reappear in lesson 6 (outbox) and in lesson 7 (agents): an idempotent effect checked against the ledger. Here you see it in the context of splitting and joining; in the next lessons, in other contexts. It's the same hammer, different nails.

Splitting work and joining it back together

Think of a purchasing manager who has to stock a list of ten products, and each product is at a different store. There are two ways to do it.

The first: go themselves, store by store, in a row. Buy at the first, come back, buy at the second, come back. It takes the sum of the ten trips. Simple and orderly, but slow.

The second: send ten assistants at the same time, one to each store. That's a fan-out: one job splits into many that happen in parallel. It takes as long as the slowest assistant, not the sum. But now a new problem shows up that didn't exist with a single manager: when the assistants start coming back, you have to wait for all of them to return and gather the ten bags before you can say "the shopping is complete." That moment of waiting-for-everyone-and-gathering is the fan-in.

And here's the detail that makes this lesson interesting: fan-in is harder than fan-out. Splitting is easy — you send the ten and that's it. Joining well is the hard part: what do you do if an assistant doesn't come back? Do you wait forever? Do you call the shopping complete with nine bags? And if you send someone for the tenth, how do you keep the other nine from going back and buying what they already brought? All of those questions are this lesson's, translated to workflows.

Fan-out in n8n: two forms

n8n gives you two ways to split work, and it's worth telling them apart because they behave differently.

Form A — branching inside one execution

The simplest one: a node has several output connections, and each one starts a different branch of the same workflow. The branches run inside one single execution.

                     ┌──▶ branch 1: check-credit
   AI Agent  ────────┤
                     └──▶ branch 2: inventory-sync

Here order-triage splits into two branches that do different things with the same order. It's a fan-out "by task": each branch is a different job. An honest nuance about n8n: even though the drawing suggests parallelism, inside a single execution n8n processes the branches one after another, not literally at the same time. The "parallel" part is conceptual — the two branches are independent — but the execution walks through them in sequence. For correctness that changes nothing; for speed, it does: don't expect branching inside one execution to give you the speed of ten real assistants. Real parallelism comes from queue mode, lesson 5's subject.

Form B — one sub-execution per item

The second form splits by item: you have N items — say, an order's ten lines — and you want to run the same job once per item. This is done with the Execute Sub-workflow node in Run once for each item mode: it fires one sub-workflow execution per incoming item.

   Split order lines (10 items)
        │
        ▼
   Execute Sub-workflow: inventory-sync   (Mode: Run once for each item)
        └─▶ runs 10 times, once per line

Each order line triggers its own inventory-sync sub-execution. By default, n8n runs them in sequence — one finishes, the next starts — not all at once. Again: real parallelism is queue mode's business. What you do gain from this form is isolation: each item is its own execution, with its own result and its own possible failure, which — as you'll see — is key to handling the partial retry.

There's a third piece worth naming: the Loop Over Items (Split in Batches) node, which walks through the items in batches of a size you define (Batch Size), delivering each batch through its loop output and, once all are done, combining the result through its done output. It's used when you want to process a little at a time — for instance, to respect an API's rate limit — instead of releasing the N at once. Its done output is, in itself, a form of fan-in: it gathers what was processed once the loop finishes.

Fan-in in n8n: the Merge node

Fan-in — joining back together what got split — has a dedicated node: Merge. Its job is to combine data from several inputs into one output, and its most important characteristic for us is this: it waits for all of its connected inputs to have data before producing its output. It is, literally, "waiting for all the assistants to come back."

Merge has several modes, and picking the right one is half of doing a fan-in well:

Out of that whole table, hold on to one comparison that prevents the most mistakes: Matching Fields vs. Position.

Position pairs by order: the first with the first, the second with the second. It's tempting because it's simple, and it's a trap. It works only if you are absolutely certain the two inputs arrived in the same order and with the same number of items. In a fan-out where the branches ran at different paces, or where a branch filtered out or failed an item, the order shifts, and Position pairs line 3's result with line 5 without any warning. The damage is silent: it doesn't throw an error, it just joins wrong.

Matching Fields pairs by identity: it reunites items that share the same value for a key — in Cumbre, order_id or sku. It doesn't care about order or whether some are missing; it joins each result with its item by who it is, not by where it landed in line. It's more robust, and it's the correct default option for almost every coordination fan-in. Memorize the rule: join by key, not by position. Position gets scrambled; identity doesn't.

The real challenge: the partial retry

Now the problem that gives the lesson its name, and where almost every fan-out breaks in production.

Imagine order-triage splits a five-line order into five inventory-sync sub-executions, one per line. Lines 1, 2, and 3 discount their inventory without a problem. Line 4 fails — the warehouse system hiccupped right at that moment. Line 5 never even got to run, because line 4's failure stopped the chain.

State of the world at that instant:

line 1: discounted ✓
line 2: discounted ✓
line 3: discounted ✓
line 4: FAILED ✗
line 5: never ran —

Now retry the whole execution — because that's what n8n does, or what you do when you see the error. What happens? The retry starts over from the beginning: it discounts line 1 again, line 2 again, line 3 again, retries line 4 (which now maybe works) and finally runs line 5. Result: lines 1, 2, and 3 got discounted twice. The warehouse ended up with lower stock than reality. The retry, meant to recover line 4, ended up duplicating three effects that were already fine.

That's the partial retry: when a split-up job fails halfway, and retrying the whole thing re-executes the parts that had already finished. It's lesson 1's duplicated effect, but born from splitting. And it's endemic to fan-out: the more branches, the more likely one fails, and the more already-done effects get duplicated on retry.

Notice why fan-out makes this worse compared to a linear flow. In a single-effect flow, one retry repeats one effect. In a five-effect fan-out where the fourth fails, one retry repeats three good effects to recover one bad one. The arithmetic of the damage is against you.

The cure: the ledger covers every branch

The solution isn't avoiding retries — you need them to recover line 4. The solution is making sure retrying a branch that already finished does nothing. That is: making every effect idempotent, checked against the shared whiteboard.

This is exactly what you built in Module 4. Every inventory discount, before running, checks the ledger: "did I already discount line 3 of this order?" If the ledger says yes, it doesn't do it again. If it says no, it does it and records it. The idempotency key here isn't just the order_id — that would identify the whole order — but something finer: the order plus the line, for example ORD-2041:CF-ARA-500. Each fan-out branch has its own ledger entry.

With that, let's retry the execution that failed on line 4:

Retry, with the ledger in the loop:

line 1: ledger says "ORD-2041:CF-ARA-500 already discounted" → does nothing ✓
line 2: ledger says "already discounted"                     → does nothing ✓
line 3: ledger says "already discounted"                     → does nothing ✓
line 4: ledger says "not registered"  → discounts it and records it ✓
line 5: ledger says "not registered"  → discounts it and records it ✓

The retry did exactly what it should: it recovered lines 4 and 5, and left the three that were already done alone. The first three "re-ran" in the sense that the flow passed through them, but since each one checked the ledger and saw its work was already done, nothing got duplicated. That's the kitchen whiteboard at work: every cook checks whether the dish already went out before preparing it.

The structure of each idempotent branch, inside inventory-sync, is Module 4's:

# Inside inventory-sync, for each line:

1. Build the key: order_id + ":" + sku   (e.g. "ORD-2041:CF-ARA-500")
2. Postgres: does this key exist in the ledger?
3. IF it already exists → finish without doing anything (idempotent)
   IF it doesn't exist  → HTTP Request to the warehouse system (the effect)
                        → Postgres: record the key in the ledger (done)

A reminder of the n8n 2.0 rules that apply here: the effect — discounting at the warehouse — is done by an HTTP Request node, not a Code node, because you can't make HTTP requests from a Code node. Querying and writing the ledger is done by a Postgres node, not a Code node, because the Code node also can't access the database directly. The Code node, if you use it, is only for building the key — concatenating order_id and sku, or computing a hash with crypto, which is one of the few allowed modules. The underlying rule: the Code node computes and decides; effects and calls are done by the dedicated nodes.

Worked example: order-triage splits inventory by line

Let's build the complete fan-out and test the partial retry, so you see the whole mechanism.

Step 1 — Split the lines. Cumbre's order carries line_items, a list. To split by line, you first break them into individual items with a Split Out node (the node that takes a list inside one item and turns it into one item per element). From an order with two lines you get two items:

[
  { "order_id": "ORD-2041", "sku": "CF-ARA-500", "quantity": 12 },
  { "order_id": "ORD-2041", "sku": "TE-CHM-100", "quantity": 6 }
]

Step 2 — Fan-out by item. You connect an Execute Sub-workflow in Run once for each item mode pointing to inventory-sync. Each line triggers its own sub-execution.

# Node: Execute Sub-workflow — inventory-sync (one per line)
Source: Database
Workflow: inventory-sync
Mode: Run once for each item
Wait for Sub-Workflow Completion: on
Workflow Inputs:
  order_id = {{ $json.order_id }}
  sku      = {{ $json.sku }}
  quantity = {{ $json.quantity }}

What to expect. When you run this with the two-line order, you'll see inventory-sync run twice — two linked executions, one per line. In order-triage's execution panel, the Execute Sub-workflow node shows two output items, one for each sub-execution, each carrying what inventory-sync returned for its line. If a line fails, you'll see its execution in red while the others stay green: that per-item isolation is exactly what's going to help you.

Step 3 — The fan-in with Merge. To gather the N lines' results into a single summary item — "order ORD-2041: 2 of 2 lines discounted" — you use a Merge. Since each result carries its order_id and its sku, and you want to reunite them by identity, you use Combine → Matching Fields on the appropriate key, not Position. If you just want to stack them into a list to count them, Append is enough.

Step 4 — Trigger the partial retry. Now the part that matters. With a five-line order, you make the warehouse fail on the fourth (in testing, an item with a sku the warehouse system rejects). The execution fails on line 4. You check the ledger: lines 1, 2, and 3 are recorded; 4 and 5 aren't.

What to expect when retrying. You retry the execution. Since inventory-sync checks the ledger before discounting, lines 1, 2, and 3 see their key already registered and don't discount again — they finish immediately, without touching the warehouse. Line 4, now that the warehouse works, gets discounted and recorded. So does line 5. In the end, the ledger has all five lines recorded exactly once, and the warehouse discounted each sku exactly once. Verify it by checking the ledger: five entries for ORD-2041, one per sku, no duplicates. That's fan-out proofed against the partial retry.

When a branch never comes back: the incomplete fan-in

There's still the most uncomfortable fan-in case: what happens if a branch never comes back? The assistant who went to the tenth store and never returned. Merge waits for all of its inputs; if one never arrives, Merge never produces its output, and your fan-in hangs.

In n8n this shows up depending on how you built the fan-out. With per-item sub-executions, if one sub-execution fails, that branch doesn't feed its result into Merge, and depending on the configuration, Merge can be left waiting or the failure can propagate. Honesty matters here: a perfect fan-in — "wait for everyone, and if one doesn't come back within X time, move on with the ones that did" — doesn't come out of a single n8n node. It has to be designed.

The robust way to design it avoids relying on Merge "waiting properly," and instead uses the ledger as the fan-in's source of truth. Instead of asking Merge "did everyone come back yet?", you ask the ledger: "how many of ORD-2041's five lines are recorded as done?" If all five are there, the order is complete. If some are missing, you know exactly which ones and can retry only those. The ledger doesn't just cover the partial retry; it's also your fan-in's completeness gauge: the truth of "how much of the split work is already done" doesn't live in a node that can hang, it lives in a table you can query. This pattern — the director queries the ledger to know whether the fan-out finished — is the one the lesson 8 project builds out fully.

Common mistakes

Joining by position in a fan-in where order isn't guaranteed (practical). What happens: someone uses Merge → Combine → Position to reunite a fan-out's results with the original items, and it works in tests — where everything arrived in order — but in production, when a branch filtered out an item or arrived out of turn, it pairs one order's result with another's data. The damage is silent: no error, the data is simply crossed. Why it happens: Position is the simplest option to set up and in the demo the order is always preserved. How to detect it: check whether your two Merge inputs can have different item counts or different order in any scenario; if the answer is "maybe," Position is unsafe. How to fix it: join by Matching Fields on a stable key (order_id, sku); pairing by identity doesn't get scrambled no matter what order the branches arrive in.

Fanning out effects with no idempotency, trusting that "a branch almost never fails" (conceptual). What happens: an effect gets split into N branches with no ledger protection, on the reasoning that failures are rare. In the demo, zero problems. In production, the first time a branch fails and someone retries, every branch that had already finished gets duplicated. Why it happens: with few branches and fast services, the partial retry seems unlikely, until volume and time make it inevitable. How to detect it: for each branch of your fan-out that's an effect, ask yourself "if I retry the whole execution, does this branch re-run its effect?" If the answer is yes, it isn't protected. How to fix it: every branch that's an effect goes through the Module 4 pattern — a per-branch idempotency key, a ledger check, effect only if not already done; never fan out effects without that.

Using an idempotency key that's too coarse for the fan-out (practical). What happens: someone protects the inventory fan-out with the order_id key, and discovers only the first line ever gets discounted: since all the lines share the same order_id, after the first one the ledger says "ORD-2041 already done" and the rest get skipped. Why it happens: the correct key for a single-effect flow (order_id) is too coarse when the same order has N effects that each need to happen once. How to detect it: if protecting a fan-out results in only one of the N branches ever running, the key is too coarse. How to fix it: the key has to identify the branch, not just the order: order_id plus whatever distinguishes each branch (sku, line number, effect type). One ledger entry per unit of work that must happen exactly once.

Trusting that Merge "waits properly" when a branch can fail (conceptual). What happens: a fan-in gets built with Merge assuming that if a branch fails, Merge will move on with the rest, and instead the execution hangs waiting for the branch that never arrived, or fails entirely. Why it happens: "waits for all inputs to have data" sounds like "waits for what it can and moves on," and that's not what it is: it's "waits for all of them." How to detect it: test your fan-in by deliberately killing a branch and watching what Merge does; if it hangs or fails, your design depends on no branch ever failing. How to fix it: don't make Merge the judge of completeness; use the ledger — check how many branches are recorded as done — to know whether the fan-out finished, and retry only the ones missing. Merge joins data; the ledger measures completeness.

Exercises

Exercise 1 — Choose the Merge mode. For each case, say which Merge mode you'd use and why:

(a) You split an order into three read queries — credit, history, and verification — and want to reunite the same customer's three responses into a single item, knowing each response carries the customer_id. (b) You ran two branches that produce lists of items and just want to stack them all into one list to count them. (c) You have two branches that return exactly the same number of items, in the same guaranteed order, and want to pair them 1 to 1.

Exercise 2 — Diagnose the vanished stock. A fan-out discounts inventory by line, with no ledger protection. An eight-line order fails on line 6; someone retries the whole execution and the order ends up "successful." The next day, the warehouse reports that five products are missing double the units they should be. Explain exactly what happened, how many times each line got discounted, and what the fix is.

lines 1-5: discounted TWICE (once per attempt) ✗
line 6:    discounted once (failed on the first attempt) ✓
lines 7-8: discounted once ✓

Exercise 3 — Design the completeness gauge. order-triage splits an N-line order into N inventory discounts. You want the director to reliably know when the order is "completely discounted," even if some sub-execution failed or never came back. Design, with boxes and steps, how you'd use the ledger — not the Merge node — to measure the fan-out's completeness and retry only what's missing.

# In the director, after the fan-out:

1. Postgres: count how many lines of this order_id are recorded
   as "done" in the ledger.
      SELECT count(*) FROM ledger
      WHERE order_id = 'ORD-2041' AND effect = 'inventory' AND status = 'done';

2. Compare against the expected total (N lines of the order).

3. IF count = N  → the order is completely discounted. Finish.
   IF count < N  → (N - count) lines are missing.
        → Postgres: get which sku's from the order are NOT in the ledger.
        → Fan-out ONLY over those missing lines (targeted retry).
        → go back to step 1.

Summary and next step

In this lesson you split work up and joined it back together. Fan-out in n8n has two forms: branching inside one execution (several outputs from a node, which n8n walks through in sequence) and firing one sub-execution per item (Execute Sub-workflow in Run once for each item), plus Loop Over Items (Split in Batches) for batch processing. Fan-in is done by the Merge node, which waits for all of its inputs; and its key decision is joining by Matching Fields (identity, robust) rather than by Position (order, fragile). The central danger is the partial retry: when a fan-out fails halfway, retrying the whole thing re-executes the branches that had already finished and duplicates their effects — and the more branches, the worse the arithmetic. The cure is Module 4's: every branch idempotent, with a key at the branch's granularity (order_id:sku), checked against the ledger. And you saw that the ledger does double duty: it covers the partial retry, and it's also the fan-in's completeness gauge, more reliable than a Merge that can hang if a branch never comes back.

Before moving on to lesson 5 you should be able to: name the two forms of fan-out in n8n; explain why Matching Fields is safer than Position; describe the partial retry and why fan-out makes it worse; and say at what granularity a fan-out's idempotency key goes, and why.

Lesson 5 gets into a pacing problem that fan-out makes sharper: what happens when events arrive faster than they get processed. You'll see backpressure — the buildup that forms when the producer goes faster than the consumer — how order gets lost along the way, and how n8n's queue mode provides capacity and pace control. And you'll see an uncomfortable consequence that connects directly to this lesson: queue mode, by running things truly in parallel, does not guarantee order on its own, and that has to be solved at the data layer — again, with the ledger.

Resources

• Merge node — n8n Docs — the fan-in node: its Append, Combine (with Matching Fields, Position, All Possible Combinations), SQL Query, and Choose Branch modes, and its behavior of waiting for all inputs. Check the exact mode names in your version.
• Execute Sub-workflow node — n8n Docs — per-item fan-out with the Run once for each item mode, which fires one sub-execution per incoming item.
• Loop Over Items (Split in Batches) — n8n Docs — the node for batch processing of a Batch Size you set, with its loop output for each batch and its done output that gathers the result once finished.
• Split Out node — n8n Docs — the node that turns a list inside one item (like line_items) into one item per element, the first step of a per-line fan-out.
• Postgres node — n8n Docs — the node each branch uses to query and write the ledger; remember the Code node has no database access or HTTP capability, so the ledger check and the effect go through dedicated nodes.