The “Dirty Dozen” – The Most Common Things That Cause Market Research To Go Wrong

Clients who are new to market research often worry that they will spend time and money on a market research project only to later realize that errors in the design or execution of the study have rendered their investment much less valuable than they had hoped. And they are right to worry – there are plenty of cases of market research blunders and even examples where market research findings have led to product design or marketing decisions that were worse than if there had been no research at all!
There are lots of individual events or mistakes that can knock market research projects off track and a short blog entry could never list them all. But there are a few types of errors that account for most of the problems. We think of these as the “dirty dozen,” – the common mistakes that we see over and over again and that rob market research studies of their potential value to decision makers. The tables below list these problems by the stage of the research project where they typically occur, along with the consequences the problem brings and – most importantly – ways to avoid them.
At the research design stage:

Problem	Consequences	Solution
Poorly formulated objectives or research questions	The data collected will not address the real issues, resulting in findings that are vague, inconclusive or even misleading.	Write out your objectives and research questions and think about what kind of data would count as an answer to each one. Don’t skip this step or assume that everybody on the project has a shared understanding of what the research is supposed to accomplish.
Poor choice of data collection method(s)	Lack of insight when needed depth or breadth (or sometimes both!) is missing from the collected data.	Make the method(s) suit the objectives and research questions. Don’t get locked into “standard” approaches or doing what is easy rather than what is best for the study.
Sample design issues	Asking questions of the wrong people can produce misleading answers in qualitative studies or statistically invalidate a quantitative project.	Be explicit about the sample parameters. Know who you are going to talk to and exactly what larger population the sample is supposed to represent.
Poorly designed/untested research instrument	Garbage in – garbage out. Failing to ask well-though-out questions, whether in a survey, a focus group or an in-depth interview, will produce poor quality results.	Every question you ask should have a purpose that relates back to the research objectives. Don’t neglect review and testing of the questionnaire or interview guide.

At the data collection stage:

Problem	Consequences	Solution
Inadequately trained or prepared data collectors	Data that is inconsistently gathered can produce gaps, validity problems and lack of depth.	Use professionals who know their jobs and have proven track records. Even experienced survey data collectors, interviewers and focus group moderators need to practice with the research instrument.
Failure to meet the sample specifications	If the sample you get is not the sample you intended, you may have data that is not pertinent or that misrepresents the views of the target population.	Have good quality control on the sample. If adjustments have to be made, be very sure you are not giving up the validity of your sample in order to fill your groups or meet numerical quotas.
Quality control issues	Poor quality control can result in errors in the data files, data that is missing or is mis- categorized.	Have a plan to check incoming data as it is collected. Don’t wait until data collection is over to begin the process of checking for errors or problems.
Loss of data	You can’t analyze data that has disappeared.	Have back-ups (and more-back-ups). Never let data reside longer than necessary in a single location or file. Have security and back-up procedures for all data storage media.

At the data analysis/interpretation stage:

Problem	Consequences	Solution
Lazy/incomplete review of the raw data	Important insights can be missed.	Have a data analysis plan that sets out how the raw data will be handled. Allow enough time for data review and processing. Don’t rely on human memory or quick skimming to capture all the meaning that the data holds.
Inappropriate data reduction techniques	Each time data is reduced, whether through coding of qualitative data or numerical consolidation of quantitative data, there is some potential loss of information or important details.	Make sure your chosen data reduction techniques capture the themes, ideas and categories that will answer the research questions. Don’t be afraid to recode or re-analyze if new issues emerge while data reduction is in progress. Remember recoding means you have learned something from your data – it’s a step forward not a step back!
Over-reliance on statistics	Interpretation that is guided only by statistical testing runs the risk of missing insights that didn’t quite pass the test criteria or finding “insights” that are really just an artifact of the statistical method.	Know the strengths and weaknesses of the statistics you use. Use them as guidelines and tools, not as the word of the research gods. Statistics are not a substitute for common sense of familiarity with your data and your research topic.
Canned answers	Having a bias toward a particular answer or type of interpretation can blind you to new themes and ideas that emerge from your data.	Keep an open mind. Let the data speak to you. Think about what would count as disproof of your preferred interpretation and make sure that there’s a way for that evidence to emerge.

As you can see, there are many potential pitfalls and room for error in market research projects of any type. Careful planning, design, oversight and analysis are absolutely key to getting the best value from the money you spend on market research!