Which Way to Compare: Part 2 – Why Percentage Distributions Are Also Better Than Indexing

Some of our clients are used to seeing comparative data presented in the form of an index – a standardized score set to 100. Their own score for the indexed item is then shown relative to the index. For example, if the national norm on a survey item is a score of 60, then 60 is indexed to 100. If the client’s score on that item is s72, that is 20% higher than 60, producing a relative score of 120.

With an index, data users can see at a glance how they stand relative to the comparative norm. This way of showing a comparison is conceptually easy to grasp and is common in types of research where the focus is heavily on the relative strength of whatever is being measured in comparison to some standard. Certain types of advertising research, in particular, uses indexing. When a proposed new ad is being tested, for example, the emphasis is not just on how well the ad does with its intended audience, but also on how well it does relative to other ads. A successful ad has to break through media clutter and there is no benefit in producing an ad and making an expensive media buy only to have the ad fail to stand out against the noise of other ads.

The strength of using an index, however, is also its weakness. By its very nature, an index puts the analytic focus on the comparison, not on whether the actual score is high or low, good, bad or indifferent. An example can help make this point. Imagine that we are designing a dessert menu for a restaurant and they survey their patrons and ask “Do you like chocolate flavored desserts?” We find that against the industry norm, the restaurant’s patrons index below the norm - only 90 against the index of 100. We might be tempted to limit the presence of chocolate in the dessert menu in favor of other flavors. An examination of the actual percentages, however, is more revealing. The norm shows that 80% of the population like chocolate desserts, while the restaurant patrons’ score was 72%. So despite the low index score, this is still a substantial majority of the folks dining at the restaurant and we would certainly want to make sure there are some chocolate goodies on that dessert menu.

As in the example above, using an index can be misleading. This is especially true if the index represents a very high percentage (like the example above) or a very low percentage. A client who “beats” an index based on a low score may feel proud, even though there is little to brag about, and may feel no changes or improvements are needed even when there is plenty of room to do so. Similarly, failing to match an index based on a very high percentage can cause clients to be upset or start wasting time and resources fixing a problem that isn’t really much of a problem at all.

So while we do feel that indexing has its place – wherever comparison is the whole point of the exercise – we think that it’s generally better to know a little bit more of the details. Most research is about making decisions and setting priorities that are about much more than whether or not you have outscored an index. A percentage distribution compared to a well-constructed norm lets data users really see where they stand, set goals that are achievable and meaningful, and take pride in legitimate successes.