A recent article in the Sunday New York Times called “Why Waiting Is Torture” (http://www.nytimes.com/2012/08/19/opinion/sunday/why-waiting-in-line-is-torture.html?pagewanted=all) brought to mind one of key dilemmas in survey design – the simple fact that people often “misremember” their experiences (which is what we call “self-report bias”). How reliable can survey results be if respondents cannot accurately recall what happened?
The article itself is about the psychology of waiting in lines and some of the points are very interesting (although perhaps not surprising to researchers!):
1. According to Richard Larson at M.I.T., occupied time (such as walking to a specific location) feels shorter than unoccupied time (such as standing around waiting),
2. There is a tendency to overestimate the amount of time spent waiting in line (the article quotes an average of 36%),
3. A sense of uncertainty, such as not knowing how long you will be in line, increases the stress of waiting, while information and feedback on wait times or reasons for delays improve perceptions,
4. When there are multiple lines, customers focus on the lines they are “losing to” and not on the lines they are beating, and
5. The frustrations of waiting can be mitigated in the final moments by beating expectations, such as having the line suddenly speed up.
What implications do these findings have on survey design and analysis? In my experience, if we are trying to get an accurate record of an event – such as the amount of time waiting in line – a straightforward recall question is not always the best choice. There are actions we can take during research design, in developing our data collection tools and in analysis to deal with the problems or poor or inaccurate self-report of behavior.
At the research design stage, we should ask whether a self-report on a survey question is the best way to collect the data. In some cases, we are better off using direct measures, such as observations of the behavior, instead of asking about it. At the questionnaire development stage, we can explore which ways of asking a question are more likely to limit bias, for example asking people what hours they watched TV last night will produce a larger per night (and more accurate) answer than asking people to estimate their total viewing hours per week. In the analysis stage we often know which direction the self-report bias will tend to lean – for example, people generally under-report their consumption of alcohol and over-report their church attendance. When we know these tendencies we can deal with them either by adjusting the answers up or down – if we know the appropriate adjustment to make – or by mentioning them when we report the findings or make recommendations.
The key here is to take the possibility of self-report bias into consideration and to have a plan for dealing with it. The existence of self-report bias does not invalidate research efforts, it is merely one of the many factors that research vendors and clients must take into consideration as they approach their projects.
Great Article.
ReplyDeleteMarket research services