The Perception Dilemma, Or, What Can We Do About Self-Report Bias?

A recent article in the Sunday New York Times called “Why Waiting Is Torture” (http://www.nytimes.com/2012/08/19/opinion/sunday/why-waiting-in-line-is-torture.html?pagewanted=all) brought to mind one of key dilemmas in survey design – the simple fact that people often “misremember” their experiences (which is what we call “self-report bias”). How reliable can survey results be if respondents cannot accurately recall what happened?

The article itself is about the psychology of waiting in lines and some of the points are very interesting (although perhaps not surprising to researchers!):

1. According to Richard Larson at M.I.T., occupied time (such as walking to a specific location) feels shorter than unoccupied time (such as standing around waiting),

2. There is a tendency to overestimate the amount of time spent waiting in line (the article quotes an average of 36%),

3. A sense of uncertainty, such as not knowing how long you will be in line, increases the stress of waiting, while information and feedback on wait times or reasons for delays improve perceptions,

4. When there are multiple lines, customers focus on the lines they are “losing to” and not on the lines they are beating, and

5. The frustrations of waiting can be mitigated in the final moments by beating expectations, such as having the line suddenly speed up.

What implications do these findings have on survey design and analysis? In my experience, if we are trying to get an accurate record of an event – such as the amount of time waiting in line – a straightforward recall question is not always the best choice. There are actions we can take during research design, in developing our data collection tools and in analysis to deal with the problems or poor or inaccurate self-report of behavior.

At the research design stage, we should ask whether a self-report on a survey question is the best way to collect the data. In some cases, we are better off using direct measures, such as observations of the behavior, instead of asking about it. At the questionnaire development stage, we can explore which ways of asking a question are more likely to limit bias, for example asking people what hours they watched TV last night will produce a larger per night (and more accurate) answer than asking people to estimate their total viewing hours per week. In the analysis stage we often know which direction the self-report bias will tend to lean – for example, people generally under-report their consumption of alcohol and over-report their church attendance. When we know these tendencies we can deal with them either by adjusting the answers up or down – if we know the appropriate adjustment to make – or by mentioning them when we report the findings or make recommendations.

The key here is to take the possibility of self-report bias into consideration and to have a plan for dealing with it. The existence of self-report bias does not invalidate research efforts, it is merely one of the many factors that research vendors and clients must take into consideration as they approach their projects.

How Should You Choose A Focus Group Moderator?

An article by Naomi Henderson in the Summer 2012 edition of AMA’s Marketing Research magazine gives a worthwhile list of guidelines for choosing a moderator (You can read this article at http://www.marketingpower.com/ResourceLibrary/MarketingResearch/Pages/2012/Summer%202012/Qualitative-Reflections.aspx).

She points out that such a choice is not necessarily straightforward because “qualitative inquiry is a delicate balance of personality, experience and awareness of the nuances of group dynamics.” In other words, in choosing a moderator, you are not only selecting someone with a particular set of skills, you are also choosing a personality and all the risks that come along with such a choice.

Naomi’s article goes on to give some very practical advice on:

· What types of questions you should ask a prospective moderator

· What types of questions you should ask the references provided by that moderator,

· What types of work samples to request, and

· What to look for in a sample DVD from your prospective moderator.

This advice is worthwhile and useful but one important point she is missing is that there are very different styles of moderating which can have a huge impact on the perceived “fit” between clients and moderators.

In my experience, the two biggest styles are what I call the “laid back” style vs. the “in your face” style of moderating. Both are effective forms of moderating but each can impact the “back room” in different ways.

Over the years, I’ve worked with a number of moderators of the “laid back” variety. They tend to be very calm, which helps the group relax, and are very deliberate in their approach, which means that the topics get thoroughly explored. One moderator in particular made very good use of silences in the group – instead of filling each moment with questions, he let respondents essentially talk through the issues and build on each other without doing a lot of active probing. I think this approach works but, at times, the silences can make certain back room clients uncomfortable because they are not “getting what they want.”

Personally, I’m more of the “in your face” type of moderator. These moderators take a very active role in the group, tend to run very high energy sessions and work very hard to avoid silences. Because there is almost always something happening in these groups, clients tend to get a sense that it is a “good” group. However, clients can also miss some of the nuance in these groups or feel that certain topics were not fully addressed.

My main point is that, in addition to the Naomi’s practical suggestions for choosing a moderator, a good addition is to also ask a prospective moderator “how would you characterize your style of moderating?” In doing so, think about the team/internal clients you will have working on your project and what style of moderating might fit best with them.