Module 7: Synthesizing And Deciding

Counting signals: `countUnpromptedUsers`

Next →

Overview

Lesson 2 left the findings sorted into four clusters, but with a deliberately declared gap: mentions counts notes, not people. A cluster with five mentions could be five different buyers independently confirming the same thing, or a single very talkative buyer repeating the same idea five times — and those two situations aren't worth the same as evidence. This lesson closes that gap: it counts how many different users touched each problem, and of those, how many did it without anyone suggesting it to them.

How this connects to the module. countUnpromptedUsers() is synthesize()'s second piece, the complete function you're going to finish in lesson 4. It takes a cluster already built by clusterByProblem() (lesson 2) as input and adds the count that truly matters for deciding: not how many times something got said, but how many different people, on their own, said it.

An everyday analogy: the pothole in the street

If one neighbor complains about a pothole in the street, that's a complaint — worth writing down, but on its own it doesn't prove much: they could be exaggerating, they could have an old car that complains about every bump, they could be wrong about where the pothole is. But if five different neighbors, who don't know each other and hadn't talked about it before, complain about the same pothole, on the same block, without anyone specifically asking them about potholes — that's no longer an isolated opinion. It's a pattern. The pothole exists, it's real, and it deserves the city's attention, regardless of whether each neighbor only mentioned it once or whether some mentioned it two or three times in the same conversation.

The key isn't how many times the pothole got mentioned — it's how many different people, without coordinating, arrived at the same complaint. That's exactly what countUnpromptedUsers() measures over Mercado's interview notes: not how many notes are in a cluster, but how many different buyers, on their own, pointed at the same problem.

Worked example: countUnpromptedUsers() over Mercado's four clusters

We reuse lesson 2's clusterByProblem(), with no changes at all, and add this lesson's new function: for each cluster, it builds a Set with the users who mentioned the problem without being suggested it (unprompted: true) — a Set in JavaScript never has repeated values, so if the same buyer shows up twice in a cluster's notes, it counts only once.

// L3: countUnpromptedUsers() -- from each of lesson 2's clusters, counts
// how many DIFFERENT USERS (not how many mentions) touched the problem
// without the team suggesting it. The same buyer repeating the same idea
// twice in the same interview is still ONE single vote.
function clusterByProblem(findings) {
  const clusters = {};
  findings.forEach((f) => {
    if (!clusters[f.problem]) clusters[f.problem] = [];
    clusters[f.problem].push(f);
  });
  return Object.keys(clusters).map((problem) => ({ problem, items: clusters[problem] }));
}

function countUnpromptedUsers(cluster) {
  const distinctUsers = new Set(
    cluster.items.filter((f) => f.unprompted).map((f) => f.user)
  );
  return {
    problem: cluster.problem,
    mentions: cluster.items.length,
    distinctUsersUnprompted: distinctUsers.size,
    users: [...distinctUsers],
  };
}

const findings = [
  { user: 'buyer_01', problem: "I don't discover products I'd like without searching for the exact name", unprompted: true },
  { user: 'buyer_01', problem: "I don't discover products I'd like without searching for the exact name", unprompted: true },
  { user: 'buyer_02', problem: "I don't discover products I'd like without searching for the exact name", unprompted: true },
  { user: 'buyer_02', problem: 'I forget about my cart and never go back to finish it', unprompted: true },
  { user: 'buyer_03', problem: "I don't trust new sellers without reviews", unprompted: true },
  { user: 'buyer_04', problem: "I don't discover products I'd like without searching for the exact name", unprompted: false },
  { user: 'buyer_05', problem: 'I forget about my cart and never go back to finish it', unprompted: true },
  { user: 'buyer_06', problem: "I don't discover products I'd like without searching for the exact name", unprompted: true },
  { user: 'buyer_07', problem: 'I forget about my cart and never go back to finish it', unprompted: true },
  { user: 'buyer_08', problem: 'the app crashes on its own when the phone has low memory', unprompted: true },
];

console.log('=== countUnpromptedUsers() over each cluster ===\n');
const clusters = clusterByProblem(findings);
clusters.forEach((c) => {
  const r = countUnpromptedUsers(c);
  console.log('"' + r.problem + '"');
  console.log('  total mentions: ' + r.mentions + '  |  distinct unprompted users: ' + r.distinctUsersUnprompted + '  (' + r.users.join(', ') + ')\n');
});

// The case that shows deduplication: buyer_01 shows up TWICE in the
// discovery-problem notes (mentioned it, then repeated it when the
// interviewer asked to dig deeper) -- but counts as ONE single user.
const discoveryCluster = clusters.find((c) => c.problem.startsWith("I don't discover"));
console.log('=== Deduplication check ===');
console.log('mentions of "I don\'t discover..." in the raw notes: ' + discoveryCluster.items.length);
console.log('DISTINCT unprompted users counted by countUnpromptedUsers(): ' + countUnpromptedUsers(discoveryCluster).distinctUsersUnprompted);

What to expect. When you run the file with Node, the output is exactly this:

=== countUnpromptedUsers() over each cluster ===

"I don't discover products I'd like without searching for the exact name"
  total mentions: 5  |  distinct unprompted users: 3  (buyer_01, buyer_02, buyer_06)

"I forget about my cart and never go back to finish it"
  total mentions: 3  |  distinct unprompted users: 3  (buyer_02, buyer_05, buyer_07)

"I don't trust new sellers without reviews"
  total mentions: 1  |  distinct unprompted users: 1  (buyer_03)

"the app crashes on its own when the phone has low memory"
  total mentions: 1  |  distinct unprompted users: 1  (buyer_08)

=== Deduplication check ===
mentions of "I don't discover..." in the raw notes: 5
DISTINCT unprompted users counted by countUnpromptedUsers(): 3

Notice two things this result reveals that lesson 2's mentions count completely hid. First, "I don't discover products..." had 5 mentions, the highest of the four clusters — but of those five, only 3 different users said them without being suggested it (buyer_01, buyer_02, buyer_06); the other two mentions are buyer_01's repetition (which already counts once) and buyer_04's mention, which got left out of the count because it was unprompted: false — the interviewer had already mentioned "recommendations" before buyer_04 said it. Second, and more surprising: "I forget about my cart..." had only 3 mentions, fewer than the previous cluster — but all three come from three completely different buyers (buyer_02, buyer_05, buyer_07), so its distinctUsersUnprompted is also 3, tying with the cluster that seemed stronger by mentions.

Why use a Set instead of just counting items.length

A Set in JavaScript guarantees, by construction, that no value repeats — adding the same value twice doesn't change the set's size. That's exactly the property needed here: if buyer_01 mentions the same problem twice in the same interview —something that happens all the time, when the interviewer asks to dig deeper into something already mentioned—, that repetition shouldn't add up as if it were a second buyer confirming the idea. Without the Set, simply counting items.length of the unprompted notes, the product-discovery cluster would show up as 4 (four unprompted notes, counting buyer_01's repetition), instead of the 3 real users who actually said it.

Common mistakes

Counting mentions instead of distinct users, and getting impressed by the highest number. What happens: someone on the team looks at lesson 2's result —mentions: 5 for "I don't discover products..."— and presents it as "the strongest signal of all," without going through the distinct-user count that reveals that cluster is actually tied in real strength with "I forget about my cart...," which had less than half the mentions (3 distinct users each). Why it happens: a higher number feels, almost automatically, like "more important" — and mentions is the first number available, before doing the work of deduplicating by person. How to spot it: if someone cites a cluster's mention count without being able to say how many different people are behind that number, this lesson's work hasn't been done yet. How to fix it: never report mentions without distinctUsersUnprompted next to it — the first measures note volume, the second measures the evidence's real breadth, and they're the same question only in appearance.

Counting as "unprompted signal" a mention where the interviewer themselves already mentioned the solution beforehand. What happens: while writing up the notes, someone marks buyer_04's mention of "I don't discover products..." as unprompted: true, because the buyer did end up describing the problem in their own words — without noticing that, a few minutes earlier in the same interview, the interviewer had already mentioned "personalized recommendations" as an example. Why it happens: when transcribing a long interview, it's easy to lose track of the exact order things were said in, and end up keeping only the answer's final content. How to spot it: review the full transcript, not just the summarized note — if the interviewer named the problem, the solution, or something very similar, at any point before the buyer mentioned it, that note should be marked unprompted: false, no matter how genuine the buyer's answer sounded at the time. How to fix it: define the unprompted rule strictly and in writing, before starting to code the notes —"did the interviewer say this idea, with these words or obvious synonyms, at any earlier point in the conversation?"— and apply it with the same criterion across all eight interviews, not from memory, interview by interview.

Exercises

Exercise 1 — Manually calculate a case with more repetition. Without running Node, if buyer_05 had mentioned "I forget about my cart..." three times in their interview (instead of once), and no other note changed, what would that cluster's new mentions and new distinctUsersUnprompted be?

See solution

mentions would go from 3 to 5 (two more notes from buyer_05), but distinctUsersUnprompted would stay exactly at 3 (buyer_02, buyer_05, buyer_07) — because the Set gives no extra weight to buyer_05 having repeated the idea several times; it's still one person. This exercise confirms, with a concrete number, why distinctUsersUnprompted is a much more manipulation-resistant measure (intentional or not) than simply counting mentions: a very talkative buyer can't, on their own, inflate the evidence's strength.

Exercise 2 — Find the cluster with the biggest gap between mentions and distinctUsersUnprompted. Without running Node, of the worked example's four clusters, which has the biggest difference between its mention count and its distinct-unprompted-user count? What explains that difference in this specific case?

See solution

"I don't discover products I'd like...", with mentions: 5 and distinctUsersUnprompted: 3 — a difference of 2. The explanation, looking at the notes: buyer_01's repetition (which already counts once, so it "costs" a mention without adding a new user) and buyer_04's mention, which was unprompted: false and so gets left out of the signal count entirely, even though it does count as one of the five total mentions. The other three clusters have mentions equal to (or nearly equal to) distinctUsersUnprompted, because each note comes from a different buyer with no repetitions or interviewer suggestions involved.

Exercise 3 — Explain why unprompted gets filtered before creating the Set, not after. Looking at countUnpromptedUsers()'s code, .filter((f) => f.unprompted) happens before .map((f) => f.user) and building the Set. Without running Node, explain in 2-3 sentences what would happen to the result if someone accidentally reversed the order and counted all distinct users first, without filtering by unprompted.

See solution

If all distinct users got counted without first filtering by unprompted, the "I don't discover products..." cluster would include buyer_04 in the count, giving distinctUsersUnprompted: 4 instead of the correct 3 — artificially inflating the strength of a signal that actually includes a mention suggested by the team itself. The order of operations isn't a stylistic detail: filtering by unprompted before deduplicating is what guarantees the function measures specifically what matters to this module —how many people reached the same conclusion on their own—, not how many people mentioned the problem in any way at all, suggested or not.

Summary and next step

In this lesson you built countUnpromptedUsers(cluster): it takes an already-built cluster and counts, with a Set, how many distinct users mentioned that problem without the team suggesting it. Over Mercado's four clusters, the result corrected lesson 2's first impression: two clusters —product discovery and forgotten cart— ended up tied with 3 distinct users each, despite having very different mentions numbers (5 and 3), while the other two stayed at 1 user each.

Before moving on you should be able to: explain why a Set gets used instead of just counting filtered elements; and manually calculate, for a new cluster, how many distinct unprompted users it would contain.

Lesson 4 brings this piece together with lesson 2's into a single complete function —synthesize()— and adds the one thing still missing: with the distinct-user count already calculated, deciding, with an explicit threshold, whether each cluster counts as signal or as noise.

Resources

  • Teresa Torres, "The Interview Snapshot: How to Synthesize and Share What You Learned from a Single Customer Interview" — producttalk.org/interview-snapshot. The artifact each interview produces, with its opportunities already identified per buyer — the source of the data countUnpromptedUsers() aggregates across different interviews. In English.
  • Rob Fitzpatrick, The Mom Test — momtestbook.com. The book that already underpinned this guide's module 2; its central warning —people lie without meaning to, especially when you suggest an idea to them— is exactly why unprompted matters so much in this count. In English.
  • Nielsen Norman Group, "Data Is More than Numbers: Why Qualitative Data Isn't Just Opinions" — nngroup.com/articles/qualitative-rigor. On why counting how many different people —not how many quotes— backs a qualitative finding is what gives real rigor to user research. In English.
← PreviousClustering findings by problem: `clusterByProblem`Next →Signal vs. noise: complete `synthesize`