If It Can’t Be Wrong, We Can’t Know That It’s Right

Recently we conducted a series of focus groups for a leading white tablecloth restaurant chain. Company management was considering a re-positioning of the restaurant to make it more “contemporary” in relation to its key competitors. So part of the focus group discussion included asking the restaurant’s current and potential customers what “contemporary” meant to them in the context of this type of high-end restaurant. The customers’ response was clear, the term contemporary was polarizing, conveying both positive elements of innovation and modernity, but also suggesting bright lighting, stark design, and a noisier, younger crowd – not the ideal for a restaurant that gets much of its business from couples celebrating romantic milestones and business customers dining with clients or discussing important deals over a lavish meal.

When we de-briefed these results with the client’s ad agency, one young agency staffer took issue with what the customers had said, suggesting “They don’t understand what contemporary means!” Of course, it isn’t unusual for clients (or their ad agencies) to take issue with research findings that don’t support their preferred course of action. But this incident illustrates two key points about research. The first is that research must be designed to actually test the preferred course of action, and the second is making sure we can hear and understand the results of the test.

Research designs that test specific hypotheses, question the flow of presumed processes or evaluate a course of action are fundamental to the scientific method. Rooted in a research history that stretches from Aristotle’s discussion of logic, through Ben Franklin’s key and kite electricity experiments, to Karl Popper’s more formal explication of the scientific method, the possibility of disproof is central to having a meaningful version of proof (http://en.wikipedia.org/wiki/Scientific_method). In market research, however, we often see research designs that do not really offer the chance for the idea in question to be disproven. For example, we see quantitative questionnaires with built-in positive bias toward ideas, concepts or ads – sometimes with no chance at all for consumers to indicate that they don’t really like the idea or prefer an alternative. In qualitative work, some discussion guides and focus group moderators “lead the witness,” producing a favorable halo of commentary around an idea that wouldn’t stand up in the real world of marketplace competition. Regardless of the data collection method, it is our job to build the possibility of disproof into our research designs. Remember, if the ideas you are testing cannot be disproved, your research results, no matter how positive, haven’t really shown anything.

Once we build in the possibility of disproof, it’s equally important to be able to hear, interpret and respond to less-than-welcome research findings. As researchers, we have to resist the temptation to dismiss negative results or interpret them away. As partners with our clients, we have to help them hear and understand the bad news and focus their energy on improving their ideas, products and marketing efforts, rather than wasting their time claiming that their customers are “wrong.”

What Questions Help Improve the Effectiveness of Qualitative Research?

All effective qualitative market research projects must start with a clear understanding of the background and objectives for each project. To help define a project and determine the appropriate methodology, we generally ask our clients the following questions:

• What are the research objectives? What are you hoping to learn? What background information can you share which led to the need for this research?
• Are there any other ways you might describe what you’re trying to explore in this research? (This question can help provide more richness to the definition of study objectives.)
• What team/internal clients is this research being conducted for? Does this team or these clients have specific preferences about how research results are summarized and/or presented?
• Are all team members in agreement about what this research should explore – and, if not, what are the differing perspectives?
• What have you/your team already done to explore these issues? (This can include previous qualitative research, quantitative research, internal data, secondary research, etc.)
• Have you ever done similar research in the past and, if so, are there any issues that were not addressed then, that you now wish you had explored? Do you have any frustrations concerning the last time you did similar research?
• What other initiatives/internal issues might affect this research? AND what other initiatives/internal issues might be affected by this research?
• What decisions will be impacted by the learning from this research? How might you act differently based on what you learn? Also, what are areas that can’t be changed, regardless of what the research might learn?
• Are there any hypotheses about the answers, among your team or your internal clients? If you were to imagine that the project is complete, what’s your ideal outcome?
• What do you expect to be the biggest challenges we encounter as we conduct this research?
• What specific constraints do we need to keep in mind?
• What stimulus material – if any – do you want to people react to and what format will it be in?
• What issues related to target audience might be relevant to know as we design the research? And are there any customer segments we should keep separate or perhaps combine for any reason?

Clear and thoughtful answers to these questions are essential to meeting both the stated and unstated objectives of any qualitative research project. These questions help to decide (1) if qualitative research is the right methodology for your objectives and, if so, (2) which approach would best meet your needs (such as deciding between focus groups and individual depth interviews), (3) what the necessary recruiting specifications are so that an accurate and effective screening questionnaire can be written, (4) what issues need to be covered in the moderator guide and, finally, (5) how your research analyst should prepare the project deliverables.

It’s impossible to overstate the importance of setting clear objectives BEFORE you undertake any research project if you want to have a successful outcome. In fact, this initial discussion can avoid those disastrous “Oh, by the ways” that have destroyed many research efforts!

How Can I Get the Most Out of Ideation or Brainstorming Research Sessions?

There are a number of established techniques for ideation and/or brainstorming (which are similar, but not exactly the same) that can be effectively used a part of a systematic search for targeted opportunities in the form of new features, new products, new markets, and/or new services within various categories of interest.

The fundamental premise of these techniques is to start with an issue or challenge and then generate a broad range of different possible ideas to address that challenge. Often, there are two important components to a brainstorming project: “Divergence” is the process of generating ideas followed by “Convergence,” which consists of selecting and developing the top ideas.

Although brainstorming sessions share some similar characteristics with focus groups, brainstorming research sessions are quite different from traditional focus groups ‒ within the field of new product development, brainstorming sessions are about exploring possibilities, generating new concepts and discovering new opportunities, whereas traditional focus groups are best used to validate ideas, weed out bad concepts and improve existing concepts.

The distinctions between brainstorming sessions and regular focus groups carry through to some critical differences in how the groups are conducted:

1. Brainstorming sessions last longer than most focus groups to ensure there is sufficient time for both training and ideation. Typically, each brainstorming session is scheduled to last between 2-1/2 and 3 hours whereas focus groups generally do not go beyond 2 hours.

2. Participants are recruited specifically to be natural “lateral thinkers” or “intuitors” because this thinking style has been shown to correlate positively with the ability to generate new ideas. However, this isn’t a common talent – most consumers are very good at reacting to ideas they are presented with but they’re not as good at coming up with new ideas on their own. In addition, the “creativity” recruiting specifications are over and above the need to invite participants who have experience with the topic under discussion.

3. Participants in brainstorming sessions are given a homework assignment to complete in advance of the session and are required to start generating ideas before attending the session. This helps to get them “primed” for the discussion and ensures that each session can start off with idea sharing from the start.

4. During the recruiting phase of the project, a member of the project team will contact each qualified participant by phone to encourage their idea generation and answer any questions about the process or expectations from the sessions.

5. Brainstorming sessions are not as much of a “discussion” as a focus group is – rather, the goal is to keep things moving and use ideas shared to spark additional ideas.

6. Ideally, the client team (often consisting of 4 to6 people) is encouraged to be fully engaged in the process and to use the ideas from the consumers to help spark their own thinking. In the end, it is often the client team members who end up generating the best, most workable ideas.

Getting the best value from brainstorming sessions also requires following a number of important steps to ensure that good quality ideas are generated. In our experience, the most effective brainstorming sessions consist of:

1. Introductions and training in the rules of brainstorming.

2. IDEATION GENERATION. Each participant shares one idea at a time, the facilitator probes for clarification if necessary, and other participants share any “builds” they have on the idea. A “build” is a new idea that is sparked by the original idea shared. The participants continue to generate and share ideas throughout the session, while the client team listens in the backroom and builds their own ideas.

3. Negative comments quickly shut down the idea-generating process; therefore, participants are taught to approach ideas with a specific mindset. If they hear a new idea they dislike, rather than share this negative reaction, they instead focus on generating a new idea that fixes what they don’t like or simply move on to sharing another idea they have generated.

4. The client team is brought in with the participants mid-way through the session and the client team members work in small teams with the consumers. Typically, each small team is asked to consider the ideas they heard throughout the session and then develop their own “ideal” solution to the project’s challenge. This co-creation process yields a range of different “ideal” solutions for the client team to consider after the session, as they choose and develop their final ideas.

How Do Qualitative and Quantitative Research Differ?

For market researchers, the distinctions between qualitative and quantitative research are very clear, even to the point where many researchers specialize in one or the other of these two types. If the difference is so clear in our minds, then why is it so difficult to write a clear, straightforward, practical and meaningful explanation of the difference?
Most explanations of the qualitative/quantitative distinction tend to focus on listing the typical methods used for each. But this puts the emphasis on the means – the way the data is collected. The more important issue in thinking about the qualitative/quantitative distinction is on the ends. It is the type of research questions you are trying to answer, and more importantly, what type of information would count as a meaningful answer, that determines whether your approach should be qualitative, quantitative or a mix of both.
For example, if your research question is about how some behavior or opinion is distributed across different groups of people, geographic areas or across time, you need numbers – usually percentages – to make these comparisons. This sort of numerical comparison implies a quantitative approach and all the methodological trappings that go with it, including:

• Closed-end survey-type questions
• Relatively large numbers of research participants or direct observations
• A sampling approach that ensures that you have the statistical validity to project the findings from your sample onto the population you are interested in

In contrast, if your research questions are less about comparisons and more about understanding chains of logic, decision processes, group norms, reactions to ideas or sensory stimuli, or cultural domains, you probably need a qualitative approach. Qualitative research focuses on understanding the human experience in context by either:

• Directly embedding the research in the scene or cultural setting (ethnography)
• Using group dynamics to explore norms, attitudes, reactions and beliefs (focus groups)
• Employing inductive questioning to probe reasoning, causation and psychology (in-depth interviews)

In short, it is knowing both the research question and what counts as an answer to the question that determines which “toolkit” the researcher will turn to for a specific project.
That said, the stage of our knowledge about a topic is often an important factor. When we first begin learning about a new topic we are usually in an exploratory mode. We may still be developing our vocabulary about the topic and couldn’t write a good survey about it if we tried – we don’t know enough to write good closed-end questions! Qualitative research is often used at this stage. The very “hands-on” qualitative techniques enable us to learn the lay of the land – how consumers think and speak about the topic. When we see research objectives using verbs like explore or identify – we are usually at this stage of knowledge and lean toward qualitative techniques. We may also use qualitative research at a later stage of knowledge to help interpret quantitative data. In these cases we use interviews or focus groups to understand the correlations that appear in survey data.
Quantitative studies are usually done only in areas where we already have the appropriate knowledge and vocabulary to design good survey questions. We have to know enough about what to ask about, how to ask it, and who to ask it of, to meaningfully quantify the responses. Research objectives that include words such as test, measure or evaluate are more typical of this stage of knowledge.
In short, thinking about qualitative and quantitative research is not about choosing methods or techniques – it is about understanding the nature of your research questions and having a good sense of what type of data – percentages, narratives, images, word counts, observations, maps of logic chains or decision processes would provide useful answers. If you are clear about what you are asking and what you need to know, the choice of methods will largely take care of itself.

How Should Market Research Be Used?

In its simplest form, market research is a decision making tool. Its primary goal is to enable more informed choices on a wide range of topics, especially for such key issues as branding, positioning, new product/service development, product use, segmentation, consumer attitudes and customer experience or behavior.

Researchers have developed a large number of tools and approaches to help answer the questions that marketers have. The two biggest “categories” are qualitative research and quantitative research. We’ll go into more depth on each of these in future posts but, basically, qualitative research is generally used to explore a topic in depth. Focus groups are the most well-known form of qualitative research, although one-on-one interviews, Laddering interviews, ethnographic interviews and “mini groups” are also often used.

Qualitative research relies on only a small number of participants, meaning that the results are not strictly “projectable” to the population as a whole. However, qualitative research is great at uncovering the linkages between consumer attitudes and behavior and really understanding how people feel about a particular topic. It can be used as the first step toward designing a more comprehensive quantitative survey or can help explain or further explore survey results that have already been collected.

Quantitative research can best be summarized as a measurement tool, with the results usually gathered through the use of a survey. Survey data can be collected in a wide variety of ways, such as online, by telephone, by mail and even in person. The two most critical factors when designing a quantitative survey are (1) sample definition, that is, who is being interviewed and (2) survey design, including the approach to be taken and what questions are being asked. We’ll delve into both of these topics in later postings.

Market research is most powerful when marketers have clearly defined their objectives. When defining a project, the decision makers don’t need to focus on the research approach, the discussion guide or the detailed survey questions– they just need to be certain about what they want to know. An experienced research can then take those objectives and create a study design to meet them. It’s really, really important for everyone involved to keep the study objectives front-and-center through all stages of any research project!