How Many People Do I Need To Survey To Get Meaningful Answers?

A central decision for anyone considering a survey – or any other quantitative research – is figuring out how big the survey sample needs to be in order to produce meaningful answers to the research questions. Researchers focus on sample size because it ties together three core aspects of any research effort:

• Cost – the bigger the sample, the more it will cost to collect, process and analyze the data
• Speed – the bigger the sample, the longer it will take to collect it (big samples can sometimes be collected quickly, but usually only by further raising costs!)
• Accuracy – the bigger the sample, the more certain we can be that we have correctly captured the perceptions/opinions/behavior/beliefs/feelings of the population we are interested in (the technical term for this is statistical reliability)

As we see from these three bullets, the decision about sample size essentially boils down to a trade-off between cost, speed and accuracy. So when we pick a sample size we are making a decision about how much accuracy we are going to purchase, within the framework of our budget and timing.

Fortunately for researchers, quantitative samples do not have to be enormous to provide findings that are accurate enough to answer most market research questions. Any unbiased sample (we’ll talk about sample bias in another blog entry) of 50 or more stands a halfway decent chance of giving you a reasonable view of the population it is drawn from and, as we increase the sample size, our confidence that we have the correct answer increases. We can show this effect by looking at the margin of error – the plus or minus number – for some common sample sizes. To keep it simple, all of these are calculated using the assumption that the sample is drawn from a large population (20,000 or more) and that we are using the 95% confidence level of statistical reliability (the most typical standard for statistical reliability used in market research). If we are looking at percentages:

• A sample of 100 has a margin of error of ± 9.8%
• A sample of 250 has a margin of error of ± 6.2%
• A sample of 500 has a margin of error of ± 4.4%
• A sample of 1,000 has a margin of error of ± 3.0%
• A sample of 2,000 has a margin of error of ± 2.1%

Looking at these numbers you can see why national surveys, such as the big public opinion polls shown on TV or in newspapers, often have samples around 1,000 or so. Samples in that size range have small margins of error, and doubling the sample size wouldn’t make the margin of error much smaller – there’s no reason to spend money making the sample bigger for such a small gain in accuracy.

These numbers also show why we often urge clients to spend a bit to make a small sample bigger, but not too big! The gains in accuracy are all at the beginning – moving from a sample of 100 to something larger is almost always a good idea, while adding anything over 1,000 usually is not. So the rule of thumb is: 100 is probably too small and 1,000 is probably too big.

Of course, in real life it can be more complicated. We may need to examine sub-groups (age or income groups, political parties, geographic regions, etc.) within the population we are looking at. If a sub-group is small, we may need a bigger overall sample to capture enough of each of the sub-groups in order to provide an accurate picture of their views. So we have a rule of thumb about sub-groups, as well – don’t make decisions about any sub-group smaller than 30. For example, if we do a survey of households in a large urban area and we want to compare households by income level, we need to make our sample big enough to have at least 30 households in each of the income categories we want to compare. Assuming this is a normal city, there will be fewer households at the high end of the income distribution than at the low end, so we need to think about how to get enough of the high-end households to be able to make how comparison. So, if we want to be able to look at households with income over $100K, and 15% of the population has an income of $100K more, we need to have a sample of at least 200 households to ensure that 30 of the households would be in that category.

Using these rules of thumb, you can form an idea about how big your sample needs to be to answer your research questions - without spending more than you can afford.