Step 5: audit the sample and weigh the evidence without bias

Overview

After lesson 5, you have a positive result but with a tight margin. Before letting that number —alone, unaccompanied— decide recommendations's fate, this step puts it through the same check module 6 designed: is the sample that generated this result biased? And how does it compare, in strength, against the rest of the evidence the team collected during the week? This lesson reuses sampleBias({ segments }) and rankByStrength(evidences), exactly as they stood in module 6, over this discovery round's real sample and evidence.

How this connects to the module. There's no new model here: both models are exactly module 6's. What this step brings to the full pipeline is the already-audited, already-ordered evidence lesson 7 is going to use, piece by piece, to move the bet's confidence — without that explicit ordering by strength, updateConfidence() wouldn't know how much weight to give each piece.

An analogy: the audit before the board meeting

Before a finance team presents its numbers at a board meeting, someone audits them: checks whether the sources are reliable, whether any number got inflated without backing. The audit doesn't change the facts of the business — it changes whether those numbers deserve the trust the following decision is going to place on them. This lesson is that audit, applied to the evidence collected during recommendations's week, before it reaches lesson 7 to become a decision.

Worked example: sampleBias() over the sample and rankByStrength() over the evidence collected

The team, learning from the mistake module 6's project found in the first round —a sample dominated 70% by power_user, with new_user completely absent—, recruited this time with an explicit criterion of balance across segments:

// L6 (M8): sampleBias() and rankByStrength(), exactly as they stood in
// module 6, over this "recommendations" round's real sample and evidence.
function sampleBias({ segments }) {
  const total = segments.reduce((sum, s) => sum + s.count, 0);
  const withShare = segments.map((s) => ({ ...s, share: +((s.count / total) * 100).toFixed(1) }));
  const dominant = withShare.reduce((max, s) => (s.share > max.share ? s : max));
  const missing = withShare.filter((s) => s.count === 0);
  const biased = dominant.share >= 70 || missing.length > 0;
  return { total, segments: withShare, dominant, missing, biased };
}

const STRENGTH_BY_TYPE = { observed_behavior: 4, reported_behavior: 3, stated_opinion: 2, hypothetical: 1 };
function rankByStrength(evidences) {
  return evidences.map((e) => ({ ...e, strength: STRENGTH_BY_TYPE[e.type] })).sort((a, b) => b.strength - a.strength);
}

console.log('=== sampleBias() over the sample recruited for this round ===\n');
const sample = { segments: [
  { segment: 'power_user', count: 3 },
  { segment: 'occasional_buyer', count: 4 },
  { segment: 'new_user', count: 3 },
] };
const sampleCheck = sampleBias(sample);
console.log('Sample: ' + sampleCheck.total + ' people.');
console.log('Dominant segment: ' + sampleCheck.dominant.segment + ' (' + sampleCheck.dominant.share + '%)');
console.log('Missing segments: ' + (sampleCheck.missing.length === 0 ? 'none' : sampleCheck.missing.map((s) => s.segment).join(', ')));
console.log('sampleBias.biased: ' + sampleCheck.biased);

console.log('\n=== rankByStrength() over the five pieces of evidence collected ===\n');
const evidence = [
  { claim: 'Clicked "recommended for you" on the checkout fake door', type: 'observed_behavior' },
  { claim: 'Ignored the recommendations section, zero clicks across two visits', type: 'observed_behavior' },
  { claim: 'Added a recommended product to the cart on the fake door', type: 'observed_behavior' },
  { claim: 'Reported that the last time they bought from another store, they followed a recommendation', type: 'reported_behavior' },
  { claim: 'Said they would love to see personalized recommendations', type: 'hypothetical' },
];
const ranked = rankByStrength(evidence);
ranked.forEach((e) => console.log('  strength=' + e.strength + ' [' + e.type.padEnd(18) + '] ' + e.claim));

const observedFavorable = ranked.filter((e) => e.type === 'observed_behavior' && !e.claim.toLowerCase().includes('ignored')).length;
const observedUnfavorable = ranked.filter((e) => e.type === 'observed_behavior' && e.claim.toLowerCase().includes('ignored')).length;
console.log('\nOf the observed_behavior evidence (strength=4): ' + observedFavorable + ' favorable, ' + observedUnfavorable + ' unfavorable.');

What to expect. When you run the file with Node, the output is exactly this:

=== sampleBias() over the sample recruited for this round ===

Sample: 10 people.
Dominant segment: occasional_buyer (40%)
Missing segments: none
sampleBias.biased: false

=== rankByStrength() over the five pieces of evidence collected ===

  strength=4 [observed_behavior ] Clicked "recommended for you" on the checkout fake door
  strength=4 [observed_behavior ] Ignored the recommendations section, zero clicks across two visits
  strength=4 [observed_behavior ] Added a recommended product to the cart on the fake door
  strength=3 [reported_behavior ] Reported that the last time they bought from another store, they followed a recommendation
  strength=1 [hypothetical      ] Said they would love to see personalized recommendations

Of the observed_behavior evidence (strength=4): 2 favorable, 1 unfavorable.

The sample passes the check: 10 people spread across three segments, none above 70%, none absent — sampleBias.biased: false. This result isn't a coincidence: the team recruited, this time, with the exact correction module 6's project left ready, instead of repeating the pattern of recruiting "whoever's easiest to reach."

The evidence, ordered by strength, reveals something important: three pieces of observed_behavior (the maximum strength), but not all three point in the same direction. Two are favorable —a real click, a product added to cart—, and one is unfavorable —someone completely ignored the section, across two separate visits—. rankByStrength() has no opinion at all on whether a piece of evidence is good or bad news: it only measures how much it weighs, by its type. A team tempted to keep only the two favorable ones, ignoring the third, would be committing exactly the confirmation bias module 6 named in detail — and this lesson, by running rankByStrength() over all five pieces together, doesn't allow that silent discard: the unfavorable evidence stays at the same strength level as the two favorable ones, not relegated to a footnote.

Why the negative signal doesn't cancel out the positive signal

It's worth resisting two equally mistaken readings of this result. The first: ignoring the unfavorable evidence because "two against one, the majority wins" — that confuses volume with strength, exactly the mistake module 6 flagged when warning that counting votes isn't the same as weighing evidence. The second: treating the single unfavorable signal as if it completely canceled out the two favorable ones, leaving the evidence tied — that ignores that all three are real observations, each legitimate, of different behaviors in front of the same feature. Neither reading is correct. What's called for is carrying all three, with their exact weight, into lesson 7 — where updateConfidence() is going to process them one by one, letting each one move the needle in its own direction, with none canceling another out by decree.

EVIDENCE TYPE            STRENGTH   COUNTS FOR       COUNTS AGAINST
────────────────────────  ────────  ───────────────  ─────────────────
observed_behavior             4      2 (click, cart)    1 (ignored)
reported_behavior             3      1 (reported...)    0
hypothetical                   1      1 (would love)     0
────────────────────────  ────────  ───────────────  ─────────────────

Common mistakes

Reporting only the favorable evidence when presenting the result. What happens: when summarizing the week for the rest of the team, someone mentions the click and the product added to cart, but leaves out entirely the evidence of the person who ignored the section across two visits — not out of bad faith, but because the favorable result feels more relevant to tell. Why it happens: a story with a happy ending is easier to communicate, and leaving out an uncomfortable data point doesn't always feel like hiding information, but like "simplifying the report." How to spot it: ask directly whether there was any unfavorable evidence during the week — if the answer takes a while to arrive or gets quickly minimized, it probably got left out of the initial summary. How to fix it: always report all five pieces of evidence together, ordered by rankByStrength(), without filtering by whether they confirm or refute — this lesson's audit exists exactly to make visible what a selective summary would hide.

Confusing sampleBias.biased: false with "the evidence is sufficient". What happens: seeing that the sample passed the bias check, someone concludes the collected evidence is already solid and sufficient to decide, without considering it's only 10 people total —a small sample, even if well balanced—. Why it happens: passing an explicit check ("it isn't biased") feels like a general quality approval, beyond what the check actually measures. How to spot it: if the conclusion about the evidence's size rests solely on sampleBias.biased: false, with no mention at all of how many people participated in total. How to fix it: sampleBias() answers a single question —does any segment dominate disproportionately, or is one completely missing?—, not whether the sample is large enough for statistical rigor. That second question, about sample size and significance, is product-metrics-and-experimentation-guide's territory — this lesson only confirms that, within the 10 people who did get interviewed, no segment took an unfair share of the conversation.

Exercises

Exercise 1 — Add a sixth piece of evidence and recalculate. To this example's evidence set, add { claim: 'Bought a recommended product two weeks later, with no reminder from anyone', type: 'observed_behavior' }. How many observed_behavior pieces would there be in total, and does the reported maximum strength change?

See solution

There would be 4 observed_behavior pieces instead of 3 (the three original ones plus this new, favorable one). The maximum strength would stay 4 — it was already the maximum possible with the three original observations, so a fourth doesn't change it, though it does improve the overall proportion of real-behavior evidence (now 4 of 6 pieces, instead of 3 of 5). rankByStrength() reports each individual piece's strength, not a running total — adding more evidence at the same strongest level doesn't change the maximum strength, but it does change how many pieces at that level reach lesson 7's synthesis.

Exercise 2 — Recalculate sampleBias() with a different sample. If the sample had been { power_user: 6, occasional_buyer: 3, new_user: 1 } (10 people total), would it pass sampleBias()'s check? Calculate each segment's share by hand before answering.

See solution

power_user: 60%, occasional_buyer: 30%, new_user: 10%. No segment reaches 70% dominance, and none is at zero — so sampleBias.biased: false, the sample would pass the check, even though power_user is clearly the majority segment. This exercise shows a real limitation of sampleBias(): the 70% threshold detects extreme dominance, not any imbalance — a 60/30/10 sample is still less representative than this example's 30/40/30, even though the model doesn't flag it as biased. The check is a necessary minimum, not a guarantee of perfect balance.

Exercise 3 — Connect it to the bet's state. After this lesson, what exactly changed about the recommendationsBet object (risk: 0.6, impact: 5, tested: false, confidenceCeiling: 0.3)? Be precise about which fields changed and which stay exactly the same.

See solution

No field changed. risk, impact, tested, and confidenceCeiling all stay exactly the same — tested is still false. What exists now, and didn't exist before this lesson, is a verified audit of the available evidence: a sample confirmed to be unbiased and five pieces of evidence ordered by real strength, with an unfavorable signal included honestly. The bet object —and its tested: false state— only changes when that audited evidence gets synthesized and confidence gets updated, exactly lesson 7's job.

Summary and next step

In this lesson you audited, without changing any model from module 6, the sample and evidence collected during recommendations's week: 10 people spread without bias across three segments, and five pieces of evidence ordered by real strength —three of observed behavior (two favorable, one unfavorable), one of reported behavior, one hypothetical. You saw, with the mixed evidence in front of you, why neither ignoring the negative signal nor letting it cancel out the positive ones is the right reading — all three are legitimate observations that deserve their own weight.

Before moving on you should be able to: run sampleBias() and rankByStrength() over any new set of evidence; and explain why an unbiased sample doesn't, by itself, guarantee the evidence is sufficient to decide.

Lesson 7 takes this already-audited evidence —together with lesson 4's interview notes— and truly synthesizes it: it counts the real signals, moves recommendations's confidence piece by piece, and reaches, for the first time in the entire guide, a decision.

Resources

• Teresa Torres, "Assumption Testing: Everything You Need to Know to Get Started" — producttalk.org/assumption-testing. On why auditing evidence before synthesizing it is a discipline, not an optional step. In English.
• Rob Fitzpatrick, The Mom Test — momtestbook.com. The reminder of why real commitment —a click, a purchase— weighs more than a compliment, whatever project you're validating. In English.
• Daniel Kahneman, Thinking, Fast and Slow — us.macmillan.com/books/9780374533557/thinkingfastandslow. On why the human brain tends to discard, without noticing, evidence that contradicts what it already wanted to believe. In English.