Some Practical Advice on Statistical Testing

One thing that I am willing to admit is that I am a very “practical” researcher, meaning that I prefer to rely on the craft of analysis when constructing a story more than statistical analyses. This is not to say that advanced statistical tools do not have their place within a researcher’s tool box but they should not substitute for the attention required to carefully review the results and to dig deep through cross tabs to uncover the patterns in the data so as to create the a relevant and meaningful story. Remember the adage – “the numbers don’t speak for themselves, the researcher has to speak for the numbers.”
A great example of this is the use – and misuse – of statistical testing. I would never claim to be a statistician but, over the years, I’ve found that the type of statistical testing that often accompanies data analysis has very limited uses. In a nutshell, statistical testing is great for warning analysts when apparent differences in percentages are not significantly different. This is extremely important when deciding what action to take based on the results. However, such testing is no use on its own when determining whether statistically significant differences are meaningful. In my experience, statistical significance works as a good test of difference but such differences alone are insufficient when analyzing research data.
I love this comment from an article by Connie Schmitz on the use of statistics in surgical education and research that “Statistical analysis has a narrative role to play in our work. But to tell a good story, it has to make sense.” (http://www.facs.org/education/rap/schmitz0207.html) She points out that, with a large enough sample size, every comparison between findings can be labeled “significant,” as well as concluding that “it is particularly difficult to determine the importance of findings if one cannot translate statistical results back into the instrument’s original units of measure, into English, and then into practical terms.”
The idea of translating survey results into practical terms represents the very foundation of what I believe market research should be doing. This same idea is highlighted in an article by Terry Grapentine in the April 2011 edition of Quirks Marketing Research called “Statistical Significance Revisited.” Building on an even earlier Quirk’s article from 1994 called “The Use, Misuse and Abuse of Significance” (http://www.quirks.com/articles/a1994/19941101.aspx?searchID=29151901), he stresses that statistical testing does not render a verdict on the validity of the data being analyzed. He highlights examples of both sampling error and measurement error that can have major impacts on the validity of survey results that would not at all affect the decision that a particular difference is “statistically significant.” I agree wholeheartedly with his conclusion that “unfortunately, when one includes the results of statistical tests in a report, doing so confers a kind of specious statement on a study’s ‘scientific’ precision and validity” while going on to point out that “precision and validity are not the same thing.”
Personally, I find it especially frustrating when research analysis is limited to pointing out each and every one of the statistically significant differences, with the reader expected to draw their own conclusions from this laundry list of differences. How can that possibly be helpful in deciding what action to take? In this case, the researcher has simply failed to fulfill one of their key functions – describing the results in a succinct, coherent and relevant manner. In contrast, I believe that I follow the recommendation of Terry Grapentine (and of Patrick Baldasare and Vikas Mittel before him) that researchers should be seeking and reporting on “managerial significance,” by focusing on the differences in survey results “whose magnitude have relevance to decision making.” This is quite a different approach than simply reciting back the results that are statistically different.
Going back to Connie Schmitz’s article, she closes with a great observation conveyed by Geoffrey Norman and David Streiner in their book PDQ Statistics:
“Always keep in mind the advice of Winifred Castle, a British statistician, who wrote that, ‘We researchers use statistics the way a drunkard uses a lamp post, more for support than illumination’.”