Common Sample Size Mistakes
Sample size problems often come from planning shortcuts rather than math errors. This page covers the mistakes that most often lead to weak or misleading results.
Using a sample size with no context
A common mistake is copying a number from another project without checking the assumptions. A sample that was reasonable for one margin of error, population, or decision context may be wrong for yours.
Sample size should always be tied to a purpose.
Ignoring subgroups
A total sample may look large enough, but subgroup reporting can fall apart if each segment ends up too small. This happens often when teams want to compare departments, regions, channels, or customer tiers after the fact.
Plan subgroup needs before fieldwork starts.
Confusing responses with invitations
Another common mistake is treating the required completed sample as the number of people to contact. If response rate is low, the invite plan must be much larger than the target number of responses.
This is a fieldwork planning issue, not a sample size formula issue, but the two are often confused.
- Copying benchmark numbers blindly
- Ignoring subgroup requirements
- Using optimistic assumptions without evidence
- Stopping data collection too early
How to prevent these mistakes early
Most sample size mistakes become visible before data collection starts if teams write down their assumptions and test them against real project constraints. That short planning step catches weak logic earlier than a formula alone can.
It also helps to separate statistical concerns from operational ones. Underpowered results and low response rates can look similar in practice, but they come from different planning failures.
- Write down assumptions before the project launches
- Check subgroup needs instead of only the total sample
- Translate completed responses into the required outreach volume
- Set a stopping rule so the team does not quit early