The say-do gap

Overview

Up to now, this module assumed that if you ask the right questions —about past behavior, not leading, not selling your solution— you're going to get the truth. This lesson introduces an uncomfortable complication: even with the perfect interview, what people say still isn't what people do. Not out of bad faith — memory fails, people overestimate themselves, and the circumstances of the actual moment of deciding are different from the circumstances of remembering or imagining a decision. This distance between word and action is called the say-do gap, and it's the reason discovery never stops at the interview: observed behavior always outweighs stated opinion.

How this connects to the module. This lesson builds sayDoGap(), the module's second central model (alongside classifyQuestion), which compares what a group of users said they would do with what they actually did when facing a real situation. It doesn't solve the gap —that requires methods beyond the interview, like module 5's fake door— but it gives you the tool to measure it and to never confuse a successful interview with complete validation.

An everyday analogy: everyone swears they'll hit the gym in January

Every January first, gyms fill up with people swearing, with total sincerity, that this year they're really going to go three times a week. They aren't lying when they say it — in that moment, with New Year's motivation fresh, they genuinely believe it. By March, most of them have stopped going. The gap isn't between what they said and a conscious lie; it's between the genuine intent of the moment they say something and actual behavior when it's time to act, weeks later, with the tiredness of the day, cold weather, and a hundred other priorities competing for the same hour.

A Mercado buyer who says in an interview "yes, I would definitely buy more if I saw recommendations" is in the same position as the person who swears they'll go to the gym in January: the intent is real at the moment they say it, but future behavior depends on factors —that day's mood, how much of a hurry they're in, whether they trust the specific recommendation they see— that the interview, on its own, can't capture.

Worked example: sayDoGap() over eight Mercado users

Imagine that, besides the interviews, Mercado's team showed a group of users an early recommendations prototype (the type of test you're going to design in depth in module 5) and logged two data points per person: what they said in the earlier interview (said, whether they claimed they'd buy more seeing recommendations) and what they did facing the real prototype (did, whether they actually interacted with it/bought something).

// L6: the say-do model. Each user said whether they'd buy more seeing
// recommendations (said) and afterward, facing a real prototype (M5),
// whether they actually clicked/bought (did).
function sayDoGap(users) {
  const saidYes = users.filter((u) => u.said === 'yes');
  const saidYesButDidNot = saidYes.filter((u) => u.did === 'no');
  const gapPercent = saidYes.length === 0 ? 0 : +((saidYesButDidNot.length / saidYes.length) * 100).toFixed(1);
  const saidNoButDid = users.filter((u) => u.said === 'no' && u.did === 'yes');

  return {
    total: users.length,
    saidYes: saidYes.length,
    saidYesButDidNot: saidYesButDidNot.length,
    gapPercent,
    saidNoButDid: saidNoButDid.length,
  };
}

const mercadoUsers = [
  { id: 'u1', said: 'yes', did: 'yes' },
  { id: 'u2', said: 'yes', did: 'no' },
  { id: 'u3', said: 'yes', did: 'no' },
  { id: 'u4', said: 'no', did: 'no' },
  { id: 'u5', said: 'yes', did: 'yes' },
  { id: 'u6', said: 'yes', did: 'no' },
  { id: 'u7', said: 'no', did: 'yes' },
  { id: 'u8', said: 'yes', did: 'no' },
];

console.log('=== sayDoGap() over 8 Mercado users ===\n');
mercadoUsers.forEach((u) => {
  console.log('  ' + u.id + ': said=' + u.said.padEnd(3) + ' | did=' + u.did);
});

const result = sayDoGap(mercadoUsers);
console.log('\nTotal users: ' + result.total);
console.log('Said they would buy more: ' + result.saidYes);
console.log('Of those, did NOT do it when it came time: ' + result.saidYesButDidNot);
console.log('say-do gap: ' + result.gapPercent + '%');
console.log('(separate data point) said no and still did it: ' + result.saidNoButDid);

What to expect. When you run the file with Node, the output is exactly this:

=== sayDoGap() over 8 Mercado users ===

  u1: said=yes | did=yes
  u2: said=yes | did=no
  u3: said=yes | did=no
  u4: said=no  | did=no
  u5: said=yes | did=yes
  u6: said=yes | did=no
  u7: said=no  | did=yes
  u8: said=yes | did=no

Total users: 8
Said they would buy more: 6
Of those, did NOT do it when it came time: 4
say-do gap: 66.7%
(separate data point) said no and still did it: 1

Of the eight users, six said in the interview they'd buy more seeing recommendations — a result that, if you stopped at the words alone, would sound like strong validation: 75% "yeses". But facing the real prototype, four of those six bought nothing — a say-do gap of 66.7%, two thirds of those who said yes didn't follow through. That's the number that actually matters for the business decision, not the interview's 75% "yeses".

Notice the separate data point too: u7 said they would not buy more, and yet they did facing the prototype. This case is less common but just as instructive — it confirms that word and action are two genuinely different signals, not just that people "exaggerate to look good". Sometimes someone underestimates their own future behavior as much as others overestimate it. sayDoGap() doesn't count this case inside gapPercent (which specifically measures how many "yeses" didn't hold up, because that's the pattern that generates the most false hope in a product team), but it's worth reporting separately — it's exactly the kind of detail module 7's synthesis is going to want to capture.

Why behavior outweighs opinion

It isn't that interviews are useless — they're extremely useful for understanding why someone would do something, what problem it solves, what language they use to describe it. But to know whether they're really going to do it, observed behavior is a stronger evidence signal than stated opinion, no matter how well-written the question that produced it was:

STRENGTH OF EVIDENCE (weakest to strongest)
────────────────────────────────────────────────────────────────
opinion about a hypothetical    →   "would you like...?" -- the weakest
opinion about the past          →   "what did you think of...?" -- better,
                                     but still a verbal memory
reported past behavior          →   "tell me about the last time..."
                                     -- a fact, though told secondhand
observed behavior                →   what they did facing a real prototype
                                     (fake door, wizard-of-oz) -- the strongest
────────────────────────────────────────────────────────────────

This strength ordering —which you'll see formalized with more rigor in module 6— explains why the rest of this guide doesn't stop at the interview: module 5 designs prototypes specifically to observe real, not reported, behavior, and module 4 teaches you to choose the cheapest test that can refute an assumption, not just the one that confirms what you already believe. This module's interview is indispensable for understanding the problem — but, on its own, it's never enough to decide whether to build something.

Common mistakes

A sample of friends who tell you what you want to hear. What happens: to get interviews quickly, someone on the team interviews colleagues, friends, or family who know the project and want it to succeed — and everyone "validates" the idea enthusiastically. Why it happens: recruiting real strangers takes time and effort; recruiting people you already know is free and fast, and that closeness is precisely what breaks the data's validity — the same "your mom loves you and doesn't want to hurt you" dynamic that gives the Mom Test its name (lesson 3), multiplied by every person in the sample. How to spot it: check who was interviewed — if more than one or two know the team, work at the same company, or have some personal stake in the project succeeding, the sample is biased before a single question gets asked. How to fix it: recruit real users, with no prior relationship to the team — and when possible, complement the interview with observation of real behavior (sayDoGap is exactly the tool for making that honest comparison), not just more interviews from the same close circle.

Reporting the "say" and stopping there. What happens: the team interviews eight users, six say yes, and the conclusion that reaches the planning meeting is "75% validation" — with no attempt at all to observe whether those six would actually act that way facing something real. Why it happens: the "say" is available immediately, after a single round of interviews; the "do" requires an extra step —a prototype, a fake door— that takes more time and effort to design. How to spot it: if the only evidence behind a high confidence in RICE is interviews where people said they would do something, with no observation of real behavior at all, the number is inflated. How to fix it: treat the interview results as the first half of the work, not the end — the second half, designing something that measures real behavior (fakeDoorSignal, module 5), is what can truly move confidence from 0.3 to something higher, as you saw at the close of product-thinking-for-engineers-guide's module 6.

Exercises

Exercise 1 — Calculate sayDoGap without running Node. For this group of five users, manually calculate gapPercent:

const users = [
  { id: 'a', said: 'yes', did: 'yes' },
  { id: 'b', said: 'yes', did: 'yes' },
  { id: 'c', said: 'yes', did: 'no' },
  { id: 'd', said: 'no', did: 'no' },
  { id: 'e', said: 'yes', did: 'no' },
];

See solution

saidYes = 4 (a, b, c, e). saidYesButDidNot = 2 (c, e). gapPercent = (2 / 4) * 100 = 50.0. Half of those who said yes, in this group, did keep their word — a say-do gap of 50% is high, but less severe than the lesson's worked example 66.7%. It's still evidence that words alone aren't enough to decide.

Exercise 2 — Interpret a gapPercent of 0%. If sayDoGap() returns gapPercent: 0 for a group of users, does that mean recommendations's risky assumption is fully validated? Explain what that result does confirm and what it still doesn't.

See solution

A gapPercent: 0 confirms that all of those who said yes actually did it facing the prototype — a strong, encouraging signal, much better than 66.7%. But it doesn't automatically confirm the whole risky assumption for two reasons: first, sample size matters — a 0% over 3 users carries much less weight than a 0% over 50 (the statistical rigor for that is product-metrics-and-experimentation-guide's subject, not this module's); second, sayDoGap() only measures the gap between those who said yes and what they did — it says nothing about whether the interviewed group represents Mercado's typical buyer well, or about whether there was bias in how the sample was recruited (this same lesson's "friends who tell you what you want to hear" mistake could produce a low gapPercent for the wrong reasons).

Exercise 3 — Design your own high- and low-gap cases. Write two lists of 4 users each: one where sayDoGap() returns gapPercent greater than 50, and another where it returns gapPercent equal to 0. No need to run Node — verify manually by counting saidYes and saidYesButDidNot.

See solution

High gap (gapPercent > 50): [{said:'yes',did:'no'}, {said:'yes',did:'no'}, {said:'yes',did:'yes'}, {said:'no',did:'no'}]. Here saidYes = 3, saidYesButDidNot = 2, gapPercent = (2/3)*100 ≈ 66.7, greater than 50.

Zero gap (gapPercent = 0): [{said:'yes',did:'yes'}, {said:'yes',did:'yes'}, {said:'no',did:'no'}, {said:'no',did:'no'}]. Here saidYes = 2, saidYesButDidNot = 0, gapPercent = (0/2)*100 = 0. Everyone who said yes actually did it — the ideal case, although, as you saw in exercise 2, with a sample of only 4 people you'd still need to be cautious before declaring the assumption fully validated.

Summary and next step

This lesson gave you the missing piece for not confusing a good interview with complete validation: the say-do gap, the distance between what people say they would do and what they actually do facing a real situation. You saw sayDoGap() run live over eight Mercado users, with an uncomfortable but honest result: 66.7% of those who said yes didn't follow through when it came time — and you understood why observed behavior always outweighs stated opinion, no matter how well-written the question that produced it was.

Before moving on you should be able to: explain the say-do gap with your own example (beyond the gym); manually calculate gapPercent over a small group of users; and explain why a sample of close friends artificially inflates the "say" without moving the real "do".

Lesson 7 closes the module with the full structure of a good interview session, start to finish — and shows you how to audit your own script with classifyQuestion() before sitting down with a real user, not after.

Resources

• Steve Portigal, Interviewing Users (2nd edition) — rosenfeldmedia.com/books/interviewing-users-second-edition. Devotes a section to the difference between what participants say and what they actually do, and to research techniques that aim to observe behavior instead of just collecting opinions. In English.
• Nielsen Norman Group, "User Interviews 101" — nngroup.com/articles/user-interviews. Explicitly names imperfect memory and social desirability bias as the two main reasons what's said in an interview doesn't match real behavior. In English.
• Teresa Torres, "Why You Are Asking the Wrong Customer Interview Questions" — producttalk.org/customer-interview-questions. Explains why even well-crafted past-behavior questions still depend on a reconstructed memory, not a direct observation — the natural bridge to why module 5 designs tests that observe real behavior. In English.