How Confirmation Bias Makes Customer Research Systematically Lie to You

What it costs your business and what to do about it.

You ran a customer survey.

87% of respondents said they'd "definitely" or "probably" buy your new feature. Enthusiasm was through the roof. Your CEO was thrilled. Your board deck wrote itself.

You built the feature.

Three months later, adoption is 4%. The people who swore they'd buy it don't use it. The people who use it don't talk about it. And nobody, absolutely nobody, on your leadership team wants to talk about the survey anymore.

Here's what actually happened and it wasn't that your customers lied. Two invisible things went wrong at the same time. Your customers gave you softer, more agreeable answers than the truth. And your team, without noticing, heard exactly what it wanted to hear.

Neither side did anything wrong. Both are behaving the way humans normally behave. And together, they explain why so much customer research produces confident, well-presented, quantified answers that turn out to be wrong.

This piece is about the second half, how your own team's thinking makes the first half worse. And what to do about it.

The Bias That Feels Like Rigor

In 1960, a psychologist named Peter Wason ran a simple experiment. He gave people three numbers — 2, 4, 6 — and told them there was a hidden rule. Their job was to guess the rule by proposing new sequences and being told whether each one fit.

Almost everyone did the same thing. They proposed 8, 10, 12. Then 20, 22, 24. Then 100, 102, 104. Each one fit. Each one felt like progress. The rule, obviously, must be "even numbers going up by two."

The actual rule was much simpler: any three numbers in ascending order. Almost nobody found it, because almost nobody tried a sequence that would have proven them wrong. Everyone kept trying sequences that would prove them right.

Wason called this Confirmation Bias: the natural human habit of looking for evidence that fits what you already believe, and overlooking evidence that doesn't.

For anyone doing customer research, this should be uncomfortable. Because the whole way research usually gets done quietly makes this bias worse.

What Customers Actually Do

Before your team even opens the survey results, the answers are already softer than reality. Not because customers are dishonest, it is more about of how people behave when they're being asked questions.

People say yes when it's easy to say yes. If a question is friendly and requires no commitment, most people will nod along. Ask "would you like a feature that saves you time?" and most people will say yes the same way most people say yes to "would you like your fries hot?" Nobody's lying. They're just answering a question that costs them nothing.

People say what makes them look good. If a question makes them think about themselves, people shape the answer to feel better about their own image. Ask people how often they exercise, and the number comes back higher than it really is. Not deliberately, just automatically.

People pick up on what you want them to say. When someone interviews a customer about a product, the customer naturally reads the interviewer. Tone, facial expressions, follow-up questions, and even enthusiasm can signal which answers are welcomed and which aren't. Most customers don't consciously decide to please the interviewer, but these subtle cues can make them soften criticism or emphasize agreement, especially when the interviewer is the person who designed or built the product.

People overstate what they'll actually do. Marketing research has shown for decades: when someone says they'll "definitely" buy something, only a fraction of them actually do often around a third, though the exact ratio varies by category and study. What doesn't vary is the pattern: the gap between a hypothetical question and a real buying decision is huge, and no survey question closes it.

None of this requires customers to be lying. It just requires them to be human, in a specific situation answering questions from someone whose incentives they can partly see.

This is the raw material your team then interprets. Which is where the second problem shows up.

What Your Team Does With It

Confirmation Bias doesn't wait for the data. It shapes the research before the first question is even written.

It shapes how you ask. "Would you be interested in a feature that helps you save time on X?" is a question with a preferred answer built into it. The 87% approval you got wasn't evidence that people want the feature. It was the sound of your own hypothesis coming back to you.

The same mechanism shows up in many cancellation surveys. You try to cancel a subscription and get a modal: "Before you go, can you tell us why you're leaving?" followed by a dropdown with six or seven pre-written options: "Too expensive." "Not using it enough." "Missing feature X." The list looks helpful. But it also frames the answers. If the real reason doesn't fit neatly into one of the available categories, the product feels emotionally exhausting to use, a competitor's onboarding felt warmer, or they don't want to admit they signed up impulsively, respondents may choose the closest option or leave important details unexplained. The company's dashboard then reports the "top three cancellation reasons" as if they fully explain why customers leave. In reality, they reflect the reasons the survey was designed to capture.

It shapes who you ask. You call the customers who love you, because they pick up. The ones who churned don't answer. The ones who considered you but chose someone else weren't on your list. Your sample ends up being a mirror of people already agreeing with you.

It shapes what you remember. You did twenty interviews. Two of them said something positive about the feature. The other eighteen were lukewarm, confused, or off-topic. The two positive ones end up quoted in the board deck. The eighteen get summarized as "mixed feedback." This isn't dishonesty. It's the normal, unconscious pattern of remembering evidence that fits the story you already have and forgetting evidence that doesn't.

It shapes how you read the numbers. Engagement went up slightly after launch. You call it validation. It could easily have been launch-day novelty, a coincidental email campaign, a competitor's outage, but you already believed the feature would work, so the spike becomes proof.

Every one of these is the same bias at work. Not laziness. Just your brain steering you, invisibly, toward the answer you already had in mind.

Why This Compounds

The reason so much research goes wrong isn't that either of these two problems is dramatic on its own. It's that they reinforce each other.

Confirmation bias rarely operates in isolation. In customer research, it often amplifies other biases shaping not only what customers say, but also which answers organizations notice, trust, and ultimately act on.

The customer, sensing what you want to hear, gives an inflated answer. Your team, hoping to hear what you were hoping to hear, focuses on the inflated part. That inflated answer gets averaged into a percentage. The percentage gets put in a deck. The deck gets treated as evidence.

By the time it's clear the research was wrong, the feature has shipped. The campaign has run. The pricing has changed. And the usual response "we need better research", often just means running the same broken process with a bigger sample, a feedback loop that repeats the cycle. Which produces the same wrong answer, just with more confidence behind it.

What To Do This Week

Three small moves. Each one is simple. Together they change what your research is capable of finding.

1. Ask a question your idea could fail. Instead of "Would you use a feature that saved you time?" Ask "When was the last time you actually paid for a tool to save time on this?" The first question invites polite agreement. The second demands a real memory. If nobody has ever paid to solve this problem, all the enthusiasm in the world isn't going to change that.

2. Talk to the people who left. The most useful interviews aren't with your happy customers. They're with the people who tried you and stopped. Churned accounts. Canceled trials. Prospects who ghosted. These are the interviews everyone avoids because they're uncomfortable. They're also the only ones that surface what your happy customers can't tell you.

3. Don't let the person who built it or the person who sells it, run the interviews. If you designed the feature, you're the worst possible person to ask customers about it. Your presence changes what they'll say. The same applies to your sales team: even asking neutral questions, salespeople carry instincts to reassure, reframe objections, and steer toward a "yes" habits that are useful when closing deals and disastrous when trying to hear the truth. The most reliable interviewers are people with no stake in the outcome: another team member outside the project, a research contractor, or even a customer of a friendly company doing an informal call. The gap between "what a customer will tell the person who built it" and "what the same customer will tell a neutral outsider" is enormous and it's usually where the real answer lives.

Each of these is small. Together they change what the process is capable of producing. That's the shift that matters.

What This Actually Costs

The real cost of getting research systematically wrong isn't the wasted feature or the failed campaign those are visible failures you can at least count.

The invisible cost is the loyal, high-value customer you slowly lose. The one who's been with you for three years, refers new business, expands their account. The one who quietly notices when a certain change signals you don't understand them anymore, when a redesigned onboarding removes something they relied on, when the product starts optimizing for a persona they're not. They don't complain. They don't fill out cancellation surveys. They just, over months, use the product less. Then one day they're gone and the exit dashboard shows nothing.

In marketing terms, these are the customers driving your Customer Lifetime Value (CLV) the total revenue a single customer generates across the full life of the relationship. A small share of these high-CLV relationships typically generates a disproportionate share of total business value. A research done by Frederick Reichheld (Bain & Company), increasing customer retention by just 5% can increase profits by 25% to 95%, depending on the industry. Every research decision this article describes is happening upstream of them. And every failure mode this article describes ensures you won't see them leave until long after they already have.

According to research by Frederick Reichheld (Bain & Company), increasing customer retention by just 5% can increase profits by 25% to 95%, depending on the industry.

The Rule Behind the Rule

Confirmation Bias in customer research isn't a mistake one person makes. It's the natural state of any process where the same people design the question, choose who to ask, and interpret the answers, while quietly hoping for a specific result.

The fix isn't a better survey tool or a bigger sample. It's systemic and structural: build the process so that being wrong is possible. Ask questions your idea could fail. Talk to people who never bought. Hand the interviews to someone who doesn't need them to succeed.

Wason's participants weren't wrong because they were lazy. They were wrong because they never tried a sequence that could have shown them they were wrong.

Neither do most companies.

Behavioral basis: Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12(3), 129-140. Nickerson, R. S. (1998). Confirmation Bias: A Ubiquitous Phenomenon in Many Guises. Review of General Psychology, 2(2), 175-220. On response effects in surveys: Schuman, H. & Presser, S. (1981). Questions and Answers in Attitude Surveys. On demand characteristics: Orne, M. T. (1962). American Psychologist, 17(11), 776-783. On the gap between stated purchase intent and actual behavior: Morwitz, V. G., Steckel, J. H., & Gupta, A. (2007). When do purchase intentions predict sales? International Journal of Forecasting, 23(3), 347-364. On Customer Lifetime Value: Blattberg, R. C., & Deighton, J. (1996). Manage marketing by the customer equity test. Harvard Business Review, 74(4), 136-144. Kotler, P., & Keller, K. L. Marketing Management (any recent edition). Rigby, D. K., Reichheld, F. F., & Dawson, C. (2003). Winning Customer Loyalty Is the Key to a Winning CRM Strategy.Ivey Business Journal (reprint of Bain & Company research. Reichheld, F. F., & Sasser, W. E., Jr. (1990). Zero Defections: Quality Comes to Services. Harvard Business Review, 68(5), 105–111.h).

Read the science behind this article: Confirmation Bias →