How can you design community research to deliver more reliable and actionable outcomes? Behavioural science can help improve research effectiveness and stakeholder engagement.
In market and community research, surveying everyone is rarely practical. Even if we invite everyone to participate in a study or survey, not everyone will respond. Conducting a census, which is a survey of everyone, is expensive.
Because we can only speak to a sample of people, we need to find ways to ensure our research is representative and has the best chance of achieving a high response rate that supports representation.
Having representative research is a fundamental part of quality research. Without good representation, there is a strong risk that results are biased and don’t represent the true needs and experiences of the community. Having research that is representative ensures the results that are valid, reliable and actionable. Providing results that can be acted upon with confidence and trusted by the community.
In this article, we cover how to improve response rates and representativeness in your community research. These suggestions also apply to other types of market research and social research.
Representation in community research is more than just being inclusive
At its heart, having representative research means having research that includes all relevant views, experiences and expectations.
- In exploratory research, the focus is on uncovering all relevant views and experiences, as well as their causes. In community research, this means understanding all the relevant views and the reasons behind them.
- In descriptive research, we move from wanting to ensure that we have covered all the relevant areas to also ensuring that we understand the relative distribution of those views. That is, we want to know how many people in our community share that view or have had that experience.
As an example, in an exploratory study on housing needs that we undertook, it meant higher representation of people in transition phases of their life – young, new to area, retiring, renters – to ensure their needs were surfaced and what was causing issues. While a community satisfaction study, our client needed more descriptive information to show how community perceptions of council performance was changing, who used services, and how this affected overall council perceptions. In this satisfaction survey the sample profile reflected the community population to prevent over and underrepresenting views. Because some community groups were relatively small but made up a disproportionate use of some facilities and services, we overrecruited these groups and then reweighted them back for total level reporting.
The representativeness of a study determines what insights we can uncover and how accurately we can generalise its results to the broader community.
Tip: Understand the purpose of representation in your what groups are least likely to engage, and why.
What does a representative sample look like?
We often rely on demographic matching to achieve representation because full behavioural insight is often not available until research is done. Demographics such as age, gender, household type, or housing status can help structure a representative sample.
The types of demographic variables need to reflect what is likely to impact views and experiences. In some community surveys, household type (such as family, shared living, or single-person households), housing type, ethnicity, or even employment status may be important factors.
A demographically representative sample increases the likelihood of a survey that is representative of opinions. Unless demographic representation is the goal, then you also need to consider what other factor you need to ensure correct representation.
If your research is for prediction or user feedback, then you will need to include recency and frequency of behaviour. For example, in library and sports facility research, not accounting for frequency of use will mean your results will reflect the views of infrequent users, leading to changes that will cause complaints or not deliver the intended outcomes.
Warning! The more things you need to control, the more complicated, lengthy, resource consuming and expensive a study becomes. Don’t overcontrol. Have study designs that increase the chances of success!
Post-Weighting isn’t a cure-all
Traditionally, sampling methods like simple and stratified random sampling aim for representativeness, but practical limitations mean they can fall short.
Sampling method alone falls short because different demographic, customer, and user groups have different likelihoods of being accessible, able, or wanting to engage when a survey is conducted.
To remedy this, surveys are often post-weighted to try to make them more representative based on census or another profile source.
Unless the underlying results already contain the needed insights, post-weighting results will not always work and can heavily rely on a few people being upweighted to have a significant effect on results. In community surveys where some groups are hard to survey, upweighting only a few to a larger number is common and can cause biased results.
Tip: Use post-weighting to correct deviations – not rescue – a poorly designed study.
Post-weighting can adjust for some imbalances, it isn’t a substitute for good design. Relying heavily on a few upweighted responses can lead to distortion.
Using behavioural science’s EAST Framework in Community Research
Behavioural science is a social science that aims to change behaviour. Drawing mostly from psychology and economics, behavioural science provides a range of tools that can be used to build better research. Specifically, we use behavioural science to improve sample sizes and representation in three areas:
- Improve the ability of people to complete the survey
- Improve their willingness to complete the survey
- Improve quality of responses and completion rates
A practical approach for investigating how to make these changes is the EAST framework. Developed by the UK government’s Behavioural Insights Team. The EAST Framework refers to Easy, Attractive. Social and Timely. These four factors are regularly found to affect willingness to uptake and engage across a wide range of situations, improving user experience, including responding to surveys and participating in research.
‘E’ if for Easy
Make it easy for people to find and complete the survey or interviews. Remove the ‘pain points’ in the journey from awareness to completion.
When designing research, be mindful that different community segments will face different types of barriers that can reduce completion. Consider the ways below on how to make it easier for each segment.
- Methodology. Is the methodology creating barriers that prevent some groups from participating in the survey?
- Access and Invitation. Is the way you engage confusing, intimidating, or just adding more steps than needed?
- Design and User Experience. Can people quickly understand what is needed and why?
Choose a methodology that is accessible and enjoyable to most people in your community, running parallel methodology if needed to improve accessibility. For example, in remote communities, among the elderly, internet access can be is less available or trusted. To address this, we have combined in-person interviewing with online or telephone surveys to improve coverage.
For access, utilise consider multiple approaches to enhance accessibility. For a study that needed high youth representation, we recruited young people through schools, libraries, sporting venues, and entertainment venues during their visits, using a combination of QR codes, posters, text messages, and emails with in-person intercepts.
Tip: Communication and instructions should take less time than completing the survey.
For user experience design – in-survey experience – also consider how the flow and layout of the survey help people complete it quickly. Long lists of agreement/ satisfaction/ importance scales are hard to complete, regardless of the method used.
Avoid long question lists, or people will ‘flat line’.
People often ‘flat line’ because they are either confused or want to finish and get it over with. If you have survey results that don’t change much over time and between people, you probably have flatlining.
Tips: Don’t torture people with repetitive and big grid questions.
‘A’ is for Attractive
Show participants why the survey matters to them. Relevance boosts engagement. Tailor messaging to show value and relevance for specific communities.
Luckily, community research is often viewed as relevant by the community. Although some of the areas’ questions are not.
A key driver of survey response rates is the perception of relevance, ease and enjoyment.
Visual design also matters. A well-designed survey appears easier and more respectful of participants’ time. However, avoid over-stylising or using patronising language, especially for younger audiences. For a study where we needed to improve representation among indigenous groups, we tailored the message in the communication channels we used to engage these groups about how their views were valued and counted.
Tip: Tailor messaging to different audiences to increase relevance and improve response rates across relevant cohorts.
When communicating to a group, remember we want to remain unbiased. For a youth survey, we found our client’s initial attempts at looking ‘relevant’ and using ‘in’ terms were dismissed as patronising . . . what they actually said was that it looked pathetic, lame, sad, embarrassing, a joke, and… you get the picture.
Tip: Making a survey look attractive should be done at the expense of making it easy to complete, or it may appear as if you are wasting their time.
For surveys that involve in-person interviewing or recruitment, we have staff clearly identified as undertaking the survey on behalf of the council, signage that draws interest, and staff who are professional and not perceived to bias the results. Being professional and courteous over the telephone is critical for effective engagement.
‘S’ is for Social
Humans are social creatures. We like what others like, and we like helping others.
Appeal to your community’s social needs.
- Sense of Community. How can you appeal to a person’s sense of community?
- Social Risk. How can you avoid triggering concern about personal or privacy risks?
Creating a sense of community by showing that people like them are participating.
At the invitation and awareness-building stage, use targeted communication that showcases people like them as being involved or included. For some groups, social media is effective, but we have found that using pop-up stalls with community members for recruitment are also effective. In reminder emails, show that others like them have completed the survey.
Some ethnic and community groups see research as risky and are hesitant to engage. Knowing what type of risk is important. For people from countries where the government is not trusted, using independent researchers and ensuring that community research is anonymous can help overcome some barriers. With these groups, using people from the same ethnic background can be counterproductive.
Tip: Surveys in which people see others like themselves complete or benefit from will increase response rates.
‘T’ is for Timely
Surveys related to a behaviour are more relevant when done closer to that experience.
For service experience studies, interviewing shortly after the encounter increases response rates because people perceive the direct relevance, and their memory is clearer. For services where an outcome is important, you may need to split the survey into two parts: one for service delivery and another for the overall service outcome. For these types of feedback and service quality studies, you can repeat the survey with someone, but be sure to inform them of this and explain the purpose.
Warning! Avoid asking people to complete experience feedback in front of staff. These surveys exhibit a strong positive bias, reflecting staff friendliness rather than service quality.
Another important factor when using time to improve response rates that can impact the willingness of different groups to complete a survey is the availability of time. When conducting a community survey of people working in an area, we found that recruiting them on their way home to complete the survey on their mobile device was significantly more effective than approaching them on their way to work. For a local tourism study, recruiting people while dining – with QR codes – meant they could complete the survey while waiting for their drink or meal.
Tip: Look at how to use timing to improve relevance, ease and attractiveness of completing the survey.
Final Thoughts: Better Design = Better Outcomes
Having community research and surveys that are representative of their community is critical. Critical to having reliable results for decision-making, and critical to having results accepted as representing community views.
In studies we have conducted for clients, using the behavioural science EAST Framework has provided an effective way to improved response rates across different community groups by reducing barriers, enhancing relevance, and increasing their motivation to complete surveys or assist in the research.
Enhance the chances of success in your projects by utilising the EAST Framework to more effectively engage with your community.