Step 6: synthesize the signals, update confidence, decide

Overview

After lessons 4, 5, and 6, you have three sources of evidence about recommendations: the notes from the eight behavioral interviews, the fake door's signal, and that same evidence already audited for bias. This lesson brings those sources together in the step that gives this guide's module 7 its name —synthesize and decide— and, for the first time in this entire module, moves the bet's confidence from the 0.3 ceiling it's carried since product-thinking-for-engineers-guide, all the way to a decision: persevere, pivot, or kill.

How this connects to the module. This guide's module 7 —which gets completed alongside this one— formalized three models for this exact task: synthesize(), which groups findings by problem and counts real signals among distinct users; updateConfidence(), which moves the bet's confidence in proportion to the evidence's strength and direction; and decide(), which translates that number, against two thresholds agreed on ahead of time, into one of three actions. This lesson reuses all three functions exactly as they stood in module 7, with no changes at all —same signature, same formula, same vocabulary— over recommendations's complete case, the same one you've been building since this module's lesson 2.

An analogy: the jury that already heard every testimony

A jury doesn't deliberate after hearing a single witness — it waits for all of them to pass, each with their own credibility and their own weight, and only then retires to deliberate. The eight behavioral interviews are a collective testimony —what several buyers, without coordinating with each other, said about their own problems—; the fake door is a different, more direct witness —not what people say, but what people did with a real click at checkout—. The jury doesn't average the two testimonies or blend them into one number: it first listens to the group of buyers to know whether the underlying problem is real (synthesize()), then weighs, separately, what the fake door's observed behavior contributes to confidence in the specific solution (updateConfidence()) — and only with both pieces on the table does it deliberate (decide()). This lesson does exactly that with recommendations's evidence: not an instant verdict on seeing the first positive result, but a deliberation that processes each source for what it actually is, arriving at a conclusion that can be defended piece by piece.

Worked example: synthesize(), updateConfidence(), and decide() over the complete case

Part 1 — Count real signals in the interviews

We pick back up the ten notes from the eight behavioral interviews module 7 (lessons 2 through 4) already synthesized about this same bet —not a new round, the same real evidence— and run synthesize() exactly as it stood in that module's lesson 4: it groups by problem, counts how many distinct users, without being prompted, mentioned each one, and classifies the group as signal if at least 2 distinct users confirmed it on their own, or noise if it doesn't reach that minimum:

// L7 (M8), Part 1: synthesize() -- CANONICAL function from module 7 (L2-L4),
// reused with no changes at all, over the real notes from the eight
// behavioral interviews module 7 already synthesized for this same bet --
// not a new round of interviews specific to this module.
function synthesize(findings, options = {}) {
  const signalMinUsers = options.signalMinUsers || 2;
  const byProblem = {};
  findings.forEach((f) => {
    if (!byProblem[f.problem]) byProblem[f.problem] = [];
    byProblem[f.problem].push(f);
  });
  return Object.keys(byProblem)
    .map((problem) => {
      const items = byProblem[problem];
      const distinctUsersUnprompted = new Set(
        items.filter((f) => f.unprompted).map((f) => f.user)
      ).size;
      return {
        problem,
        mentions: items.length,
        distinctUsersUnprompted,
        classification: distinctUsersUnprompted >= signalMinUsers ? 'signal' : 'noise',
      };
    })
    .sort((a, b) => b.distinctUsersUnprompted - a.distinctUsersUnprompted);
}

console.log("=== synthesize() over the 10 notes from Mercado's 8 interviews ===\n");
const findings = [
  { user: 'buyer_01', problem: "I don't discover products I'd like without searching for the exact name", unprompted: true },
  { user: 'buyer_01', problem: "I don't discover products I'd like without searching for the exact name", unprompted: true },
  { user: 'buyer_02', problem: "I don't discover products I'd like without searching for the exact name", unprompted: true },
  { user: 'buyer_02', problem: 'I forget about my cart and never go back to finish it', unprompted: true },
  { user: 'buyer_03', problem: "I don't trust new sellers without reviews", unprompted: true },
  { user: 'buyer_04', problem: "I don't discover products I'd like without searching for the exact name", unprompted: false },
  { user: 'buyer_05', problem: 'I forget about my cart and never go back to finish it', unprompted: true },
  { user: 'buyer_06', problem: "I don't discover products I'd like without searching for the exact name", unprompted: true },
  { user: 'buyer_07', problem: 'I forget about my cart and never go back to finish it', unprompted: true },
  { user: 'buyer_08', problem: 'the app crashes on its own when the phone has low memory', unprompted: true },
];
const synthesized = synthesize(findings);
synthesized.forEach((s) => {
  console.log('[' + s.classification.toUpperCase().padEnd(6) + '] "' + s.problem + '"');
  console.log('         mentions=' + s.mentions + '  distinct-unprompted-users=' + s.distinctUsersUnprompted + '\n');
});
const signals = synthesized.filter((s) => s.classification === 'signal');
const noise = synthesized.filter((s) => s.classification === 'noise');
console.log('Summary: ' + signals.length + ' signal(s), ' + noise.length + ' noise, out of ' + synthesized.length + ' distinct problems mentioned.');

What to expect.

=== synthesize() over the 10 notes from Mercado's 8 interviews ===

[SIGNAL] "I don't discover products I'd like without searching for the exact name"
         mentions=5  distinct-unprompted-users=3

[SIGNAL] "I forget about my cart and never go back to finish it"
         mentions=3  distinct-unprompted-users=3

[NOISE ] "I don't trust new sellers without reviews"
         mentions=1  distinct-unprompted-users=1

[NOISE ] "the app crashes on its own when the phone has low memory"
         mentions=1  distinct-unprompted-users=1

Summary: 2 signal(s), 2 noise, out of 4 distinct problems mentioned.

With signalMinUsers: 2 —the threshold module 7 fixed ahead of time—, two problems clear the bar, not just one. The first, "I don't discover products I'd like...," is the same highest-impact opportunity (impact: 5) from module 3's tree, the one that's held up recommendations from the start, confirmed by 3 distinct buyers with nobody suggesting it. The second, "I forget about my cart...," ties it with the same strength —also 3 distinct users— even though it has fewer total mentions (3 against 5): the synthesis rewards distinct users, not note volume, exactly the correction module 7's lesson 3 introduced. The other two problems —trust in new sellers, the isolated report of the app crashing on its own— stay on the noise side, with a single buyer each.

Part 2 — Update confidence with the fake door's result

With the fake door's signal (lesson 5) validating the hypothesis, we move the bet's confidence from the 0.3 ceiling RICE left it at. updateConfidence(prior, evidence), exactly as it stood in module 7's lesson 7, receives a single formal piece of evidence: the fake door's aggregate result, with strength: 0.7 —the same strength that module's lesson 7 declared for this exact test—.

// L7 (M8), Part 2: updateConfidence() -- CANONICAL function from module 7
// (L7), reused with no changes at all. Receives ONE single formal piece of
// evidence -- the fake door's aggregate result -- so as not to double-count
// the same signal that already summarizes, in a single number, hundreds of
// real clicks.
function updateConfidence(prior, evidence) {
  const posteriorRaw = evidence.validates
    ? prior + (1 - prior) * evidence.strength
    : prior - prior * evidence.strength;
  const posterior = Math.round(posteriorRaw * 100) / 100;
  return {
    prior,
    posterior,
    delta: Math.round((posterior - prior) * 100) / 100,
    direction: evidence.validates ? 'up' : 'down',
  };
}

console.log("\n=== updateConfidence() -- lesson 5's fake door VALIDATED the hypothesis ===\n");
console.log('reminder: fakeDoorSignal() gave 6.5% click-through against a 5% threshold -- it passed.');
console.log('strength=0.7: real behavior observed at checkout, but a single week, a single location.\n');
const PRIOR = 0.3;
const confidenceUpdate = updateConfidence(PRIOR, { validates: true, strength: 0.7 });
console.log('prior=' + confidenceUpdate.prior + ' (capped by RICE since product-thinking-for-engineers)');
console.log('posterior=' + confidenceUpdate.posterior + '  (' + confidenceUpdate.direction + ', delta=' + confidenceUpdate.delta + ')');

What to expect.

=== updateConfidence() -- lesson 5's fake door VALIDATED the hypothesis ===

reminder: fakeDoorSignal() gave 6.5% click-through against a 5% threshold -- it passed.
strength=0.7: real behavior observed at checkout, but a single week, a single location.

prior=0.3 (capped by RICE since product-thinking-for-engineers)
posterior=0.79  (up, delta=0.49)

It's worth explaining why updateConfidence() receives one piece of evidence here, and not the five lesson 6 left ordered by strength. The fake door's aggregate result —260 clicks out of 4,000 impressions, 6.5%— is already the sum of exactly that kind of individual behavior observations, multiplied by thousands of real checkout visits, not just the few cases the team could write down by hand. Adding the aggregate and, on top of it, the individual pieces that make it up —the click, the "ignored the section," the product added to cart— would be counting the same signal twice, exactly the mistake updateConfidence()'s canonical formula avoids by design: one piece of evidence, a single jump from prior to posterior. That doesn't make lesson 6's audit useless: the five pieces ordered by rankByStrength() still serve their real purpose, which is qualitative, not arithmetic —they confirm the evidence behind the aggregate isn't unanimous (someone ignored the section across two visits), a concrete reason not to read the 6.5% as an overwhelming validation, even though that nuance doesn't change the confidence calculation.

Part 3 — Decide: persevere, pivot, or kill

With confidence at 0.79, and the two thresholds agreed on ahead of time in module 7's lesson 6 —persevere: 0.6, pivot: 0.3—, the decision arrives:

// L7 (M8), Part 3: decide() -- CANONICAL function from module 7 (L6). The
// thresholds get agreed on BEFORE calculating confidence, not adjusted
// afterward to make the result "look good."
function decide(signalStrength, threshold) {
  if (signalStrength >= threshold.persevere) {
    return { signalStrength, action: 'persevere', reason: 'the accumulated evidence clears the threshold to keep investing as-is' };
  }
  if (signalStrength >= threshold.pivot) {
    return { signalStrength, action: 'pivot', reason: 'there is real signal but not enough to continue unchanged' };
  }
  return { signalStrength, action: 'kill', reason: 'the accumulated evidence is not even enough to pivot' };
}

console.log('\n=== decide() ===\n');
const THRESHOLD = { persevere: 0.6, pivot: 0.3 };
const decision = decide(confidenceUpdate.posterior, THRESHOLD);
console.log('thresholds agreed on ahead of time: persevere >= ' + THRESHOLD.persevere + '  |  pivot >= ' + THRESHOLD.pivot);
console.log('signalStrength=' + decision.signalStrength + '  ->  ' + decision.action.toUpperCase());
console.log(decision.reason);

What to expect.

=== decide() ===

thresholds agreed on ahead of time: persevere >= 0.6  |  pivot >= 0.3
signalStrength=0.79  ->  PERSEVERE
the accumulated evidence clears the threshold to keep investing as-is

0.79 clears the persevere threshold (0.6) with a real margin —almost two-tenths, not a technical tie—. decide() doesn't even need to reach the second if: the decision is PERSEVERE, the same one module 7 already reached, with the same evidence, in its own closing project. This isn't coincidence or redundancy — it's exactly what's expected of a canonical model applied twice to the same data: if synthesize(), updateConfidence(), and decide() produced different results between module 7 and this module over the same bet, something would be wrong with at least one of the two runs, not with the evidence itself.

The three models, side by side

Model                 Input                       Output                          Question it answers
────────────────────  ─────────────────────────  ────────────────────────────  ──────────────────────────
synthesize()           interview notes,            grouped problems, with        how many distinct users
                        each with user +            mentions and                  mention the same thing,
                        problem + unprompted        distinctUsersUnprompted,       without being suggested it?
                                                     and classification (signal/
                                                     noise)

updateConfidence()     prior + evidence            prior, posterior, delta,      how much does this piece of
                        {validates, strength}       direction                     evidence move the belief?

decide()               signalStrength + threshold  signalStrength, action        with EVERYTHING above, what
                                                     (persevere/pivot/kill)        do we do now?
────────────────────  ─────────────────────────  ────────────────────────────  ──────────────────────────

Common mistakes

Treating a comfortable confidence as if it were a perfect 1.0. What happens: seeing that 0.79 comfortably clears the persevere threshold (0.6), someone communicates the result as "recommendations is 100% validated already," without mentioning it's still a pedagogical model based on a single formal piece of evidence —the fake door—, with its own declared limits (one week, one location, measured in clicks, not real purchases). Why it happens: a wide margin over the threshold feels like absolute certainty, and it's easy to forget the number is still an estimate with limited evidence, not an exhaustive measurement. How to spot it: if you ask the team "what would happen if we ran the fake door a second week and it gave a different result?," and the answer is "it shouldn't change anything, it's already 100% validated," confidence got communicated without its limits. How to fix it: always report the exact number —0.79, not "validated" or "100%"— and remember that, just as module 7's lesson 7 stated, it's the best estimate available today, not a permanent guarantee.

Treating PERSEVERE as "it's already built," instead of "let's keep investing in the bet as it's currently framed". What happens: when communicating the decision, someone summarizes "recommendations is already validated, let's ship it" —skipping over the fact that persevere is a decision to keep investing in discovering and building the bet, not an announcement that the recommendations engine already exists in production—. Why it happens: after seven lessons and a favorable decision, it's tempting to treat the module's close as if it were the whole project's close. How to spot it: if the result's communication doesn't distinguish between "we decided to keep investing in this bet" and "the engine is already built and measuring real GMV," the guide's central boundary got lost. How to fix it: PERSEVERE in this pipeline means the collected evidence —cheap, early, qualitative, and from a single fake door— justifies continuing to invest in recommendations without changing solutions; it doesn't mean the work of building and rigorously measuring is already done. Lesson 8 makes explicit, with the guide's full map, what remains after this decision.

Exercises

Exercise 1 — Recalculate decide() with a stricter threshold. If the team had set { persevere: 0.85, pivot: 0.3 } instead of { persevere: 0.6, pivot: 0.3 } —a higher persevere threshold, decided before running any test—, would the final decision change with the same confidence of 0.79?

See solution

Yes, it would change: with persevere: 0.85, 0.79 >= 0.85 is false, so decide() moves to the second check; since 0.79 >= 0.3 (the pivot threshold) is true, the decision would be PIVOT instead of PERSEVERE. This exercise isn't permission to "choose the threshold that gives you the result you want" —that's exactly the mistake module 7's lesson 6 warned about—, but a demonstration of why the threshold has to be fixed beforehand, with an explicit business criterion for how much confidence justifies continuing to invest in recommendations as it's currently framed, and not adjusted afterward to fit whatever number the team expected to find.

Exercise 2 — Calculate the opposite case: if the fake door had refuted the hypothesis. Without running Node, if the same fake door had given a result below the 5% threshold —validates: false, with the same strength: 0.7—, what would the resulting posterior be, starting from the same prior: 0.3? What would decide() decide about that number?

See solution

posterior = 0.3 - 0.3 * 0.7 = 0.3 - 0.21 = 0.09. With decide(0.09, { persevere: 0.6, pivot: 0.3 }), the result would be KILL: 0.09 doesn't even reach the pivot threshold (0.3), so the function returns the third branch. This exercise confirms the asymmetry of updateConfidence() you already saw in module 7: starting from the same low prior (0.3), a strong validation rises a lot (+0.49, up to 0.79) because there's a lot of room to grow, while a refutation with the same strength falls less in absolute terms (-0.21, down to 0.09) because the room to fall, from an already-low prior, is smaller — but in this concrete case, that drop is still enough to cross both of decide()'s thresholds in one jump and land straight on kill.

Exercise 3 — Add a ninth interview and recalculate the synthesis. Without running Node, if a ninth interview added { user: 'buyer_09', problem: "I don't trust new sellers without reviews", unprompted: true } to findings, would the classification of "I don't trust new sellers without reviews" change? Would it change decide()'s final decision about recommendations?

See solution

That problem's classification would change: distinctUsersUnprompted would rise from 1 (buyer_03) to 2 (buyer_03, buyer_09), crossing the signalMinUsers: 2 threshold and moving from noise to signal. But the final decision about recommendations wouldn't change: the signalStrength that enters decide() comes specifically from updateConfidence() over the fake door's result (0.79), not from the interview synthesis — a new cluster crossing into signal is real evidence about a different problem, one that would deserve its own discovery cycle, but it doesn't move the number that already decided recommendations's bet. This exercise confirms something important: the interview synthesis confirms (or doesn't) the underlying opportunity, and updateConfidence() moves confidence in the specific solution — two different questions, with their own numbers, that decide() combines only through the latter.

Summary and next step

In this lesson you ran module 7's three canonical models over recommendations's complete case: synthesize() confirmed the product-discovery opportunity has real signal (3 of 8 buyers mention it unprompted, tied with "forgotten cart"), updateConfidence() moved the bet's confidence from the 0.3 ceiling up to 0.79 with the single formal piece of evidence —the fake door—, and decide() translated that number, against thresholds fixed ahead of time, into this entire module's first complete decision: PERSEVERE.

Before moving on you should be able to: explain why decide()'s thresholds get fixed before calculating confidence, not after; and distinguish, in your own words, what PERSEVERE means versus PIVOT and KILL in this model.

Lesson 8 —this guide's final project— doesn't repeat this pipeline in isolation: it runs it once more, end to end, chained together with the previous six steps in a single file, and turns today's PERSEVERE decision into a concrete launch and measurement plan, closing out everything you've learned since module 1's lesson 1.

Resources

• Teresa Torres, Continuous Discovery Habits — producttalk.org/continuous-discovery-habits. On how to synthesize discovery evidence continuously, not as a one-off event at the end of a sprint. In English.
• Eric Ries, The Lean Startup — theleanstartup.com/book. The original source of the "pivot" vocabulary — worth rereading now that this project's decision was persevere, not pivot, to know clearly, ahead of time, what the alternative would have meant. In English.
• Marty Cagan (Silicon Valley Product Group), "The Four Big Risks" — svpg.com/four-big-risks. A final reminder of why a decision of this magnitude —persevere, pivot, or kill— deserves the full pipeline, not a meeting-room hunch. In English.