Getting the Best Value from Open-Ended Questions

An article by Carolyn Lindquist called “For Better Insights From Text Analytics, Elicit Better Comments” in the most recent edition of Quirks Marketing Research (April 2012) gives three recommendations for improving the quality of consumer responses to open-ended questions. These three recommendations are:

1. Target your questions

2. Ask why

3. Be sensitive to placement

Based on my own experience, these are worthwhile considerations when designing surveys. I think most quantitative researchers – including me! – can fall into the twin traps of asking too many open-ends in a single survey and not defining those open-ends as clearly as possible.

I’m a strong believer in what I consider “directed open-ends,” which means that the wording is specific to the situation rather than a catch-all “please list comments below.” For example, in concept tests, I strongly believe in asking for strengths and weaknesses separately and this makes the survey both easier to answer and to analyze. This is consistently with Carolyn’s recommendation to “target your questions” – the example she gives is to link is to vary the open-ended question text according to the stated level of overall satisfaction.

I’m intrigued by Carolyn’s suggestion to ask “why” rather than “what questions,” as they have found that asking “why” (such as “please tell us why you were less than satisfied with your experience”) yields longer and more useful answers than asking “what” (as in “please tell us what we can do to improve your next experience”). She has found that the responses to “what” questions contain less detail and emotion than the answers to “why” questions. I think this suggestion is worth testing out. However, this does not mean we should ask “why” after every rating question, as we’ve had some clients request a few times over the years!

I also agree with her third recommendation on being sensitive to the placement of open-ended questions, although I don’t agree with her suggestion that open-ends should only be asked at the end of a survey. In my experience, open-ended questions should appear where they make the most sense in a survey and a nice balance of quantitative rating questions and open-ends makes for a more pleasant and natural survey-taking experience. One caveat though – I avoid having too many open-ended questions listed sequentially, as I believe that too many open-ends in a row can lead to a feeling that the survey is longer than it actually is and lead to respondent fatigue.