How Many People Do You Really Need for a Survey?

One of the most important questions when designing a survey is ‘How many people do I need in my survey?’.  Get it wrong, and your survey will miss critical insights, be ignored by decision-makers, waste your resources and time, blow your budget, or exhaust your customers with ‘feedback fatigue’.    

Conducting research is a journey into the unknown.   For many, conducting large-scale surveys seems to be a safe bet.  But it is an expensive bet.  A better bet is one that efficiently and accurately captures the views and behaviour of you market or community.

At the opposite end is doing a survey that is too small.  A survey that interviews too few people will give results that are unreliable and heavily biased.

When it comes to finding the right survey size, it is a Goldilocks problem; you don’t want a sample size that is too small or too large.

If you are looking for a quick answer, the sample size of n=400 is commonly used because with a 95% confidence level, you will have a maximum margin of error of ±5%.  Most decision-makers can live with this for important decisions.  For strategic decision-making involving multiple segments, a sample size of n = 1,500 may be required.  However, smaller surveys can provide highly reliable results if done correctly and smaller surveys among small populations are statistically robust.  

Although a sample size that is too small is often seen as the primary problem, having surveys that are larger than necessary also comes with its own issues.

It may seem obvious, but why is a small sample size a problem?

A sample size that is too small increases the likelihood of obtaining a wrong result, a biased or random outcome.  In statistical terms, you have an unreliable result.  If your results are unreliable, then you have wasted your time and resources on doing the survey.

When a sample size is too small, even minor changes in who is interviewed can have a significant impact on the results, leading to findings that do not accurately reflect the actual situation.  For research that evaluates or tracks changes, this can lead to making incorrect decisions.

Takeaway:  Time spent ensuring you have the right sample size will save you money, time, and your credibility.

 

Why more responses are not always better?

Not always!

While interviewing more people is safer than interviewing too few, it comes at a cost: direct costs and indirect costs.  Increasing sample size also comes with rapidly declining statistical benefits.

If you’re paying for contacts, the increased cost is obvious; however, for businesses that survey their customers, members, or community from their panel, the costs are larger and often less obvious.

 

The hidden costs of oversized surveys

Below are ways larger sample sizes can increase survey costs, which you need to consider when deciding on the right sample size for your surveys:

• High Survey Software Costs. Survey software charges costs either per complete or based on usage levels.  The level of use looks like a fixed cost, but it is based on a maximum number of surveys, and you are charged either to ‘buy more interviews’ or charged at a high cost per complete, about your quota.   For organisations that have more than one survey, this can mean costs increase with each new survey you complete.  Every person who clicks on the survey link is charged to you.  If you overestimate the number of interviews you do, you still pay the minimum cost, which means your actual interview costs are higher.
• Higher Interview Costs. If you are using telephone and face-to-face surveys, every new interview incurs direct labour costs and facility costs.  While these costs may come down as you gain economies of scale, they can also go up dramatically as the cost of recruiting each person to complete the survey gets harder, especially if you have survey quotas.
• Lower Future Response Rates. When you survey a person about a topic, they can become less willing to ‘help again’.  As response rates fall, you need to send out more invitations and spend more resources to get the same amount of feedback.  A negative loop that drives up business costs.  Even sending survey invitations can be seen as reducing their overall willingness to help.
• Lost Future Engagement. Continuously demanding feedback from a high number of your customers can lead to fewer customers engaging with other messages you send.  Meaning lost sales, lower uptake on new services, reduced awareness of events or changes that you make that are important to your business.
• High Staff Costs. Larger surveys will increase time demands on staff.  Increasing time spent on trying to increase the survey sample size and in the subsequent analysis that comes with looking at more segments in a survey and reporting on more groups.
• Incentive Costs. Increasing response rates may involve offering additional incentives to encourage participation. If these are cash prizes, this can result in higher regulatory costs. If you are giving away products and services, this can lead to either lost sales or an increased bias in your surveys towards people who game the system to win.
• Increased Sample Costs. Even if you are not paying for a sample from an independent contact provider, larger survey sizes can result in higher costs when building your community panel.  The reduced response rates will necessitate increased spending on efforts to recruit participants for your panel.

 

Opportunity Cost

The most significant cost of having a sample size bigger than needed is what economists call ‘Opportunity Cost’.  This means that the costs incurred in conducting a larger survey mean you have fewer resources for addressing the problems identified in the research.

These higher survey costs are why many organisations work with independent market research companies.  What appears to be low survey costs can quickly become prohibitive, especially if additional staff or staff time are factored into conducting your research.

Takeaway: Don’t use large surveys when a more efficiently designed survey will yield the same results and be more sustainable.

 

The Cautionary Tale of the CEO who Killed Customer Feedback

A few years ago, while heading the Insights and Strategy for the region at a global bank, the CEO felt convinced that the only way he would believe the poor feedback we had received from our customers was to increase the sample size.  He wanted us to interview every customer whenever they engaged with us or made a change.  At face value, this made sense.  Allowing every customer to give feedback seemed the right thing to do.  It felt like we were being equitable and able to find more issues.  This meant that we transitioned from conducting surveys with an average of 400 interviews per month to over 6,000 per survey.

Our software costs increased from less than $10,000 per year to over $150,000 per year due to the surge in additional surveys and their impact, as well as the need to maintain other routine feedback surveys.

Our response rates quickly fell from an average of 34% to 5%.  The routine feedback surveys had a collapse in feedback after the initial account opening.

To rectify the situation, we were tasked with creating a customer panel that included only those customers who wanted to participate in a survey and began offering incentives for survey completion.  This led to increased workload, the need for additional staff, biased survey responses, and higher costs.

Customers became disengaged, and a significant source of complaints was the ceaseless number of messages being sent.  Statistically, this meant our average sampling error dropped from ±5% to ±1%  – a small drop for a very large cost.  Critically, the increase in sample sizes did not impact the decisions being made.

 

How to determine the right sample size

We’ve discussed the benefits of larger sample sizes and costs; now, let’s uncover the best practices for determining sample size for your research.

Determining the right sample size for any survey research is based on three areas:

• Decision-making needs
• Operational constraints
• Statistical requirements

Or, put another way . . .

• Step 1: Define what you need and the level of confidence you need
• Step 2: Determine what time, money and resources you have
• Step 3: Using maths to fine-tune the optimal sample size

 

Decision-making needs

Not all decisions come with the same level of relevance, risk, level of detail needed, or uncertainty about what you are researching.

 

Decision relevance

If 10% have a problem, would that be enough for you to invest in solving the problem?  What if that problem causes them to leave?  What if it reduces satisfaction by 20% but you already have high satisfaction levels?  What about understanding whether a strategy has improved results by at least 10%?  Maybe you need at least 25%.

The more important it is, the more critical it is to be able to identify smaller issues; the larger sample sizes you will need. This is called Effect Size, and its technical impact is covered in  Statistical Requirements.

 

Decision risk and confidence

When the risk of making a wrong decision is high, you need larger sample sizes to achieve smaller margins of error and to be more confident that you have the correct result.  This margin of error is typically shown as a range you feel comfortable with.  Statistical confidence is measured in terms of probability.  The most common is 95% but in some contexts, you may feel comfortable with 90%.  In a practical sense, a 95% confidence level means you are comfortable with a 1 in 20 chance of getting it wrong, while 90% means a 1 in 10 chance.

For example, if your results indicate that 40% of customers are interested in buying your new product, you can be 95% confident that the true result is within ±5% and lies between 35% and 45%.  The more confident you need to be or the smaller range you can accept, the larger your sample size needs to be.  How to technically use margin of error and confidence intervals for setting sample size is covered in the Statistical Requirements section of this article.

 

Level of detail

If you need to understand any subgroups and segments in your survey in detail, you will need a survey sample size large enough to understand those groups.  For example, if you know that you have a segment of your community that is only 10% of the population and that you need at least n=100 people in that segment. If you are using random sampling, your sample size will need to be at least n=1,000 just to ensure you find that many people in that segment.  Often, it is these smaller segments that determine the required sample size.

Segments can also include the need to know more about different behavioural or attitude groups.  This could mean understanding the needs of customers who are most or least interested in your products, or identifying the various issues that dissatisfied customers face.

The deeper you need to delve into specific groups in your analysis, the larger your survey sample size needs to be to understand those groups reliably.

Takeaway:  Your sample size needs are directly driven by your decision-making needs.  Understand what decisions you need to make from the survey and your risk appetite.

 

Operational constraints

Every research project has constraints.   The main constraints that affect sample size for any survey are often budget, time, and the availability of a survey sample.  After understanding what is needed and statistically what sample is ideal, our constraints determine what we can do.

When understanding the costs of larger sample sizes, we discussed some of the operational constraints.

Not having enough money for the ideal sample size requires compromising.  This could mean being unable to go into detail for smaller segments or lacking the level of confidence you would prefer.

Depending on your method, time constraints can significantly impact the sample size.  Larger sample sizes take longer to collect, especially for more labour-intensive research approaches like telephone and face-to-face interviewing.  Online customer surveys can often require reminder messages, which can add to your survey time and help achieve a larger survey sample size.

For customer and community panel-based research, the actual size of the available contact list is often the primary factor driving sample size choice.

 

Insight return on investment (Stop when you are not learning anything new!)

Although it sounds like difficult maths, this approach to determining your number of interviews and survey sample size is simple.  Start with a small number of interviews, and once you no longer see any significant change or discover anything new, stop!

This approach is also known as a marginal return on investment and is most effective for well-defined and straightforward issues, where there are no significant differences between groups.  It is ideal for user testing.   In academic research, this approach is used to determine when to stop searching for new cases and to begin analysis.

 

This marginal return approach has been used to justify the use of only five interviews for user testing.

The approach is simple, but it also places a significant burden on the researchers to make the right decision and defend it.

Takeaway:  If you have significant operational constraints, start by knowing these first and look at what methodology is a better fit for your needs and resources.

 

 

Statistical requirements

We have covered the broader managerial areas that we need to understand in setting the right survey sample size; however, there are also technical statistical requirements and methods for determining a sample size.

First, it is essential to note that the use of statistical tools for sample size setting is only valid for random sampling methods.  If you are not after a random sample, then read no further!

If your sampling is non-random, then your sample size is determined by your decision-making needs and operational constraints.  The Insight Return on Investment approach is ideal for non-random sampling research.

There are several formulas for determining sample size; however, the most common and straightforward is Cochran’s Formula.  This formula is used for results measured with percentages.

n = (Z² * P * Q) / E²

• ‘n’ is for the number of people in the sample size that the formula will estimate as optimal based on the parameters you give below.
• ‘Z’ is the Z-score, which is a transformation of the confidence level that you want from your survey. The most common is ‘1.96’ for a 95% confidence interval and ‘1.65’ for a 90% confidence interval.
• ‘P’ is the percentage score that you are testing or estimating, which is the main measure for the research. For example, if your focus is on checking to see if satisfaction has changed, and it was 70% in the last study, you would input ‘0.7’.   If you don’t know, then use 50% (0.5).  This is the score where the maximum error can occur.
• ‘Q’ is just the opposite of ‘P’ and is calculated as ‘1–P’. This is why 50% has the maximum error.
• ‘E’ is for the maximum margin of error that you are willing to accept for the target that you used for ‘P’. For example, you are happy with the true result being within ±5% of what the survey estimate.  For this equation, you use this range as ‘0.05’.

There are many sample size calculators available online if you don’t feel comfortable using the above formula in a spreadsheet.  You can also use the chart below to choose the most efficient sample size.

 

 

Other statistical considerations

Now that you know what the equation is, you may have more questions about how to know you are setting the correct parameters.  These parameters are strongly linked to both the characteristics of the population or market you are studying.

These considerations can have a strong impact on sample size and provide ways to

• Population Variation. The more diverse your market or community is in what you are studying, the larger your survey sample size needs to be to capture it accurately.
• Effect Size. For evaluation and monitoring studies, the bigger the change you are trying to detect, the smaller the sample size needs to be to detect that change, and vice versa.  Detecting a change of only ±1% will require a much larger sample size than one that only needs to detect changes of ±10%.
• Measurement Sensitivity. The better your research design is at reducing ‘noise’ and is sensitive to measuring variation, the fewer people you need to include in your community survey.  This sometimes refers to a survey’s study power.  For measurement scales, you can sometimes do this by having better scales, like moving from a two-point ‘Yes/ No’ to a five-point level agreement scale. However, larger scales can also increase ‘noise’ if people are trying to guess where they sit on a scale.  There are many techniques used in research design and scales that can significantly improve sensitivity beyond simply increasing scale size.
• Variable Ratio. If you are doing statistics, it is advisable to have at least 10 observations per item in your model.  For example, if you are running a regression model with two predictors, then you should have at least 20 interviews.

 

What if you don’t know?

Unless you are conducting a survey that involves tracking or monitoring, you are unlikely to know all the key factors that determine the ideal sample size.  If you did not have this information, you probably would not need to do the survey.

When this occurs, you can either do a pilot study, which gives you some initial estimates, or make best/ worst case assumptions on which to base your decision and to err on the side of larger sample sizes.

 

Takeaway:  Use scenarios to test different options to see what sample sizes emerge as the best options, and then choose a sample size that matches the most likely scenarios.

 

Evaluation and tracking surveys

If your survey is part of an evaluation or tracking changes over time, then knowing what Effect Size and Margin of Error are acceptable is critical.  You must set your margin of error to be smaller than your estimated effect size; otherwise, you may get results that are wrong or deceiving.

For example, if your strategy, campaign or program is expected to improve results by at least 10% for it to be judged a success, then you must have a margin of error less than 10%, say at least 7%.

 

What if you only have a small population to survey?

The population size only starts to have a noticeable impact on sample size when the population is less than 4,000 people.  For a small population, it is more likely that other factors will mean you won’t need to interview a large number of people.   In small community and business surveys, there is often less variation in views because of similar experiences.

The chart below illustrates how the sample size changes for different population sizes in a survey aiming to achieve a ±5% margin of error with a 95% confidence interval.

 

 

The Bottom Line: Finding the right survey size

Determining the right sample size for your survey is crucial in ensuring that you obtain accurate results, feel confident in the findings, and invest only what is necessary.  Finding that right sample size means understanding what you need from the research, the constraints you are working with, the characteristics of the population you are researching, and the discovery framework you are using.  By following the guidelines outlined in this article, you can determine the optimal sample size tailored to your specific needs.

 

FAQ: How Many People Do You Really Need for a Survey?

What is the ideal sample size for a survey?

The most commonly used sample size is n = 400, which at a 95% confidence level provides a ±5% margin of error. For strategic decision-making that involves multiple segments, a larger sample size of around n = 1,500 may be required.

 

Why is a small sample size a problem?

A small sample size produces unreliable and biased results, where minor changes in who is surveyed can dramatically alter findings. This increases the chance of making wrong business decisions and wastes time, money, and credibility.

 

Are bigger surveys always better?

No. Larger surveys deliver diminishing statistical benefits while significantly increasing costs. Hidden costs include higher survey software fees, staff workload, declining future response rates, customer disengagement, and greater incentive expenses. The biggest risk is opportunity cost – spending too much on oversized surveys instead of fixing identified problems.

 

How do you determine the right survey sample size?

The right sample size depends on three key factors:

1. Decision-making needs – How important and risky the decisions are.
2. Operational constraints – Budget, time, and access to participants.
3. Statistical requirements – Desired confidence level, margin of error, and population characteristics.

 

What formula is used to calculate survey sample size?

The most common method is Cochran’s Formula:

n = (Z² * P * Q) / E²

Where:

• Z = Z-score (1.96 for 95% confidence, 1.65 for 90%)
• P = estimated proportion (use 0.5 if unknown)
• Q = 1 – P
• E = margin of error (e.g., 0.05 for ±5%)

Many free online calculators are available if you don’t want to do the math manually.

 

How does population size affect sample size?

Population size only matters when it is less than 4,000. In smaller populations, fewer respondents may be needed because variation is lower, but statistical requirements still guide the minimum sample needed.

 

What are the risks of surveying too many people?

Oversized surveys can:

• Inflate software and labour costs
• Cause survey fatigue and lower response rates
• Reduce customer engagement with your business
• Increase incentive costs
• Waste resources through opportunity cost.