If It Can’t Be Wrong, We Can’t Know That It’s Right

Recently we conducted a series of focus groups for a leading white tablecloth restaurant chain. Company management was considering a re-positioning of the restaurant to make it more “contemporary” in relation to its key competitors. So part of the focus group discussion included asking the restaurant’s current and potential customers what “contemporary” meant to them in the context of this type of high-end restaurant. The customers’ response was clear, the term contemporary was polarizing, conveying both positive elements of innovation and modernity, but also suggesting bright lighting, stark design, and a noisier, younger crowd – not the ideal for a restaurant that gets much of its business from couples celebrating romantic milestones and business customers dining with clients or discussing important deals over a lavish meal.

When we de-briefed these results with the client’s ad agency, one young agency staffer took issue with what the customers had said, suggesting “They don’t understand what contemporary means!” Of course, it isn’t unusual for clients (or their ad agencies) to take issue with research findings that don’t support their preferred course of action. But this incident illustrates two key points about research. The first is that research must be designed to actually test the preferred course of action, and the second is making sure we can hear and understand the results of the test.

Research designs that test specific hypotheses, question the flow of presumed processes or evaluate a course of action are fundamental to the scientific method. Rooted in a research history that stretches from Aristotle’s discussion of logic, through Ben Franklin’s key and kite electricity experiments, to Karl Popper’s more formal explication of the scientific method, the possibility of disproof is central to having a meaningful version of proof (http://en.wikipedia.org/wiki/Scientific_method). In market research, however, we often see research designs that do not really offer the chance for the idea in question to be disproven. For example, we see quantitative questionnaires with built-in positive bias toward ideas, concepts or ads – sometimes with no chance at all for consumers to indicate that they don’t really like the idea or prefer an alternative. In qualitative work, some discussion guides and focus group moderators “lead the witness,” producing a favorable halo of commentary around an idea that wouldn’t stand up in the real world of marketplace competition. Regardless of the data collection method, it is our job to build the possibility of disproof into our research designs. Remember, if the ideas you are testing cannot be disproved, your research results, no matter how positive, haven’t really shown anything.

Once we build in the possibility of disproof, it’s equally important to be able to hear, interpret and respond to less-than-welcome research findings. As researchers, we have to resist the temptation to dismiss negative results or interpret them away. As partners with our clients, we have to help them hear and understand the bad news and focus their energy on improving their ideas, products and marketing efforts, rather than wasting their time claiming that their customers are “wrong.”