Demonstrating the impact of community engagement in research

In health and social care research, value is usually placed on certain ways of evidencing impact. In particular, quantitative measurements drawn from the sciences have come to be viewed as an accepted and expected standard for defining and measuring impact. Community engagement requires us to change our approach to what we measure and value when it comes to impact. This section covers this and provides examples of tools we can use to help us demonstrate the contribution our engagement is making to changes in our research or ways of working.

What does impact look like in community engagement?

Community engagement requires us to change our approach to what we measure and value when it comes to impact.

A different approach

The impact of community engagement happens at:

Individual level
Community level
Institutional and organisational level
Relationships level
Research infrastructure, systems and processes level

Change happens throughout the engagement process, rather than being something which only happens at the end of the process. Because community engagement is predominantly about relationships, the impact ripples out across different levels and systems, and it's important that our process for demonstrating impact helps to capture this.

In research, there is "...a tendency, particularly among academic researchers and research bodies/councils, to attribute a higher value to larger scales of impact i.e. to ‘national’ over ‘local’, ‘wide reach’ over ‘deep’, ‘institutional’ over ‘personal’ and to legitimise certain social changes/impacts above others.” (International Collaboration for Participatory Health Research). However, in community engagement, value is placed on micro-level impacts as well as macro-level impacts.

If we are changing our approach to how we do research and how we engage people in research, we have to change our approach to what success and impact looks like. We are changing the scale of our engagement, so we have to change how we value the size and scale of our work and any outcomes. In community engagement, bigger is not necessarily better, more effective or valuable.

If we want to see larger numbers of people from under-served backgrounds taking part in research, then we have to start with changes in individual behaviour and perceptions towards research. We cannot change individual behaviours and perceptions without investing time and resources in engaging people at this level. We therefore have to value change and impact at an individual level, as well as at a wider institutional and systems level.

If large-scale, broad engagement and recruitment practices were efficient in engaging under-served groups in research, we wouldn’t see different groups being under-served by research! We therefore have to change our mindset about what success 'at scale' means.

Levels of impact in community engagement

Below are some examples of the impact we may see as a result of community engagement work at different levels.

(i) Individual level:

Improved knowledge about their own community by taking part in involvement/community engagement activities
Personal and professional development, for example, gaining transferable skills as a result of their involvement
Growth in confidence as a result of engagement
Taking part in research
Increased awareness of research
Change in attitude towards research
Able to have their voice heard in research
Enjoyment and value placed on engagement
Involvement in research (for example, as a Research Champion, joining an advisory group, or funding committee)

(ii) Community level:

More of a culture of talking about research within the community - are there spaces where people can talk about health and care research?
More touchpoints to engage with and find out about research in the community - for example, is information visible in the community about research? Can people attend sessions to learn about research or speak to somebody about it, for example, their social prescriber?
Community-level activities for research
Research sites in the community - Is research taking place within the community itself?

(iii) Organisation/institution:

Better knowledge of how to involve the community in research
A better understanding of the local population the organisation serves
Established pathways for future recruitment and engagement if new study or project opportunities arise
Development of community engagement knowledge, skills and capacity among the workforce
Organisational priorities and activities align with the local population’s needs and interests

(iv) Relationships:

Improved relationship between the organisation/ institution and members of the community
Improved relationships and networks with others in the area, for example, with Local Authorities, NHS Trusts, schools
Improved ways of working and relationships across teams and organisations, if the engagement process goes well

(v) Research infrastructure, systems and processes:

Improved interconnectivity and relationships within the wider research infrastructure (if done well!)
Embedding of research within wider health and care infrastructure, for example, within local authorities or NHS Trusts
Improved systems for enabling community inclusion and involvement (for example, payment processes for individuals and community organisations )
New opportunities for study sites and recruitment routes
More diverse and inclusive research participant cohorts
Potential money and time saved in recruiting to time and target
More representation of community voices in research governance and decision-making

Making the case for community engagement

When we decide we want to do engagement or when we are asked to show the impact of our engagement work, we can often find ourselves having to navigating the tension between:

Knowing that engaging communities is morally and ethically the right thing to do, and having to present it as financially the ‘right’ or ‘valuable’ thing to do through business cases. This can often be the case when we are working in resource-stretched environments where difficult financial decisions are being made.
Knowing and seeing the value of changes in people’s individual perceptions towards research and resulting behaviour changes (for example, an individual signing up to take part in a research study). This often contrasts with a system where change that can be quantified at a larger scale through numbers (for example, the number of people signed up to a research registry) is often regarded as a higher value outcome.
We are therefore faced with the challenge of demonstrating impact and asking for sufficient resources in a system that is mismatched to the realities of community engagement work. There are a few ways we can approach this.

(i) Demonstrating benefits/outcomes vs costs/inputs

We can do a simple mapping out of what the proposed or demonstrated outcomes of an engagement project are, compared with the costs or inputs required.
We can do this to demonstrate both the financial benefits or outcomes of engagement, and the social or non-financial benefits or outcomes of engagement.
When we are under financial pressures, it can be tempting to focus on the financial ‘return on investment’ of engagement, but it is really important that we also show the social return on investment.
We also want to be able to show who benefits from each outcome. Is it your organisation? Individuals in the community? The research project you are doing?

(ii) Demonstrating new resources

We can also demonstrate how engagement might help us acquire new resources. For example, if we invest in a community, we may acquire a team of Research Champions who volunteer to spread the word about research opportunities.
We can show how this helps our organisation from a resourcing perspective, by, for example, contrasting this new resource with the cost of an alternative resource (for example, staff time needed to reach as many community members).

(iii) Demonstrating the alternatives to engagement

Another approach that we can take to demonstrating impact or potential impact of engagement, is by contrasting the potential impact of our approach with the impact of alternative scenarios:

What happens if we don’t do anything?
What happens if we carry on as normal, and maintain the status quo?
What would happen if we took a different approach to engagement than the one we are proposing or doing?
And what would happen if we used alternative means, such as a marketing campaign?

In Making the Case for Public Engagement (Involve and Consumer Focus 2011, page 20), an example from Luton's Neighbourhood Governance programme shows how it is possible to develop a product through public engagement and directly compare the cost of that engagement activity (in this case a slogan or branding) with a cost quotation from a commercial organisation to deliver the same output. Luton’s Neighbourhood Governance programme established branding through consultation – ‘Your Say, Your Way’. This saved the cost of a Public Relations firm developing the brand (demonstrating financial benefits) and also created more local ownership over the programme (demonstrating social benefits).

If we think about the risks and impacts of not doing community engagement in research, we may see:

Wasted resources on ineffective recruitment and engagement processes;
Fewer routes and pathways for recruiting and involving participants in our research, because we haven’t invested in building relationships and getting to know our communities;

We therefore risk prolonging our recruitment and making it a more difficult task for ourselves. We also make it more difficult to recruit from a sample that is representative of the populations we serve. As a result, we may continue to see the research evidence that informs our health and care services failing to represent different groups and geographies. This means we will continue to see health inequalities persist.

We can demonstrate what some of these risks might be if we don’t engage with our communities properly, as well as some of the potential risks that come with engagement. This can help us make an informed decision about how to proceed with the time and resources we have.

Contribution, not attribution

Community engagement does not exist in a vacuum. Even if we think that doing activity ‘A’ should lead to outcome ‘B’, there are lots of social, environmental, structural and other factors that may contribute to or affect Outcome B and the impacts of activity ‘A’. Community engagement initiatives are not like a randomised control trial!

The best approach is to therefore acknowledge that multiple factors are likely behind an observed change or changes. Our job is to understand the contribution of the community engagement activities to this change. We therefore need to be careful about directly attributing changes to our actions, but rather showing how we have contributed to that change through our actions.

Steps for demonstrating impact in community engagement collectively

There are a few steps we can take to collect and demonstrate impact in practice. This section covers these steps. The further resources section of this page provides a set of tools and methods you can use in your engagement processes to help you understand and demonstrate the impact.

Guidance for demonstrating the impact of community engagement

Design, collect and analyse impact together with our communities

As with all aspects of community engagement, we should follow the 'nothing about us without us' principle when it comes to designing how we collect information about the impact of this work. We should take a participatory approach to evaluating the impact of our work together with the community. At the beginning of our engagement project or work, we should discuss with our partners what they think success looks like for this work, and how they would like to capture and assess whether you have been successful in your work together.

This can also be an opportunity to develop skills and knowledge in the community (such as leadership and evaluation skills), by creating opportunities for a range of people to lead and participate in the different stages of evaluation and learning.

Embed creativity and different forms of expression

People are able to express themselves most comfortably in different ways. By using creative methods and opportunities for people to express themselves in different ways, we allow people to share what the impact of the engagement has been on them. The tools and methods provided in this section include some examples of creative evaluation methods that can be used in this way. Your communities and the partners you work with will be best able to tell you which forms of expression they would prefer to use, so ask them.

Have fun!

Everyone involved is likely to take the process more seriously and be committed if they are enjoying themselves. The previous point about embedding creativity can help us to inject fun into the evaluation process, and keep it engaging for people.

Mixture of methods and approaches

Given the impact of community engagement can happen at different scales and some of the changes that happen can be incremental, it can be beneficial to use a mixture of methods of data collection. We may use a mixture of quantitative methods (for example, keeping a log of the number of people engaged or attending community events) with qualitative methods (such as stories and narratives, case studies, surveys, interviews, and photographs).

Let go of perfect and embrace a ‘good enough’ approach. Getting an approximate measure of what really matters is better than getting a highly accurate measure of data that is less important.

Gather as you go

Community engagement is a long-term process. If we only measure a pre-defined set of outcomes at the end of the process, we risk losing a record of some of the valuable changes that have happened incrementally earlier on. By checking in regularly and keeping a narrative of how the engagement is having an impact, we can also check what is working well, and where we might need to change our approach as we move forward.

We can share this ongoing impact with colleagues to raise awareness and buy-in for this work. This could lead to further resourcing and support for your community engagement efforts. Community engagement is dynamic and open. This means that we cannot fully know what the impact or outcomes are going to be in advance. There may be unintended, additional outcomes that we need to capture.

Share the impact with communities and stakeholders

To be accountable and transparent, we need to share the impact with the community we are working with, and also with our other stakeholders and partners.

Consider ways you can share the impact in an engaging format. For example, could an exhibition or community event be an effective way to share the impact with people, rather than a report via email? Your partners in the community are best placed to advise on what will be most effective.

Resources, methods and tools for demonstrating impact

Evaluation guidance and frameworks

Durham University: Mapping Alternative Impact: Alternative approaches to impact from co-produced research. This report outlines the problems that arise when co-produced/participatory research is evaluated for impact. It describes the limitations of existing concepts and evaluation of impact and makes recommendations for changes that will better support co-produced research. The report’s main themes and recommendations also have relevance to other forms of co-production, and also to delivering impact through research that has not been co-produced.
El Ansari and Andersson (2004): The Costs and Benefits to Participants in Community Partnerships: A Paradox? This article presents the potential benefits and costs of engagement. It highlights the arguments for and against measuring the benefits and costs of engagement through a financial lens.
Here to There Consulting Inc: Developing a User Profile Aide for Action. A framework to help you identify who the primary users of your evaluation are, what they want the evaluation to explore, their preferences for how it should be conducted, and when they need the evaluation feedback.
International Collaboration for Participatory Health Research: Position Paper #3: Impact in Participatory Health Research. This paper explores how we define impact in participatory health research, and how we measure and demonstrate impact.
Involve and Consumer Focus: Making the Case for Public Engagement. A guide to help make the case for engagement in financial terms.
UK Standards for Public Involvement. The UK Standards for Public Involvement are designed to improve the quality and consistency of public involvement in research. Developed over three years by a UK-wide partnership, the standards are a description of what good public involvement looks like and encourage approaches and behaviours that are the hallmarks of good public involvement. The standards are for everyone doing health or social care research and provide guidance and reassurance for users working towards achieving their own best practice. They can be used as benchmarks to measure progress when evaluating community engagement processes in research.
Inspiring Communities: Evaluating and Learning About Community Led Change: A Principles-Based Approach. This guide provides a suggested set of principles for community-led change processes, and how to evaluate community-led change using these principles. The guide includes a number of ideas for participatory activities you can use to evaluate your engagement.
Inspiring Communities: The Quadrants of Change Framework. Provides a framework for identifying change in communities across four dimensions: personal, relational, structural and cultural.
Sue Holloway, Pro Bono Economics. Our Place Guide to Cost-Benefit Analysis. Detailed guidance on developing a cost-benefit analysis, which is a process to help demonstrate the value of an intervention.
What Works website. A website dedicated to providing guidance and resources to support organisations to demonstrate impact in community-level work.
What Works: Types of Evaluation. Summarises the different types of evaluation you can do and what they are used for.

Evaluation tools and methods

Evaluation Wheel method: A participatory evaluation activity that allows you to create and assess the measures of success for your engagement together with people and partners from the community.
Graffiti Wall method: A participatory and creative evaluation or feedback activity that allows people to write or draw their comments onto a public display. This works well for public events.
Wishes tree method: An interactive activity that allows members of the community to share their wishes. This is a useful exercise for understanding what matters to members of the community to help shape your evaluation.
Photograph Diary method: An interactive and participatory evaluation tool that invites participants to keep a photograph diary and to capture, select and discuss images relating to an engagement event or process that are especially meaningful to them. Responses in participants’ own words can provide great insights that are otherwise very difficult to attain. It is particularly helpful during the implementation of a new programme of activity, to check how things are going and provide formative feedback opportunities.
NIHR: Research Ready Communities, Group Community Conversations booklet. Provides a set of methods that can be used to consult members of the community for their views and feedback on a topic or project. The methods include word clouds, the H-diagram, counselling, voting with your feet, and sticker wall (also known as a sticker or dot voting, Photovoice).
Dotmocracy: How to Use Dot Voting Effectively. An overview of the dot voting or sticker voting method, including how and when to use it.
Our Locality: Tips for Trainers: Introducing the ‘H-form’ - a method for monitoring and evaluation. An overview of the H-diagram method for collecting community views and feedback.
Better Evaluation: Most Significant Change. The Most Significant Change (MSC) approach involves generating and analysing personal accounts of change and deciding which of these accounts is the most significant – and why. This approach allows you to understand some of the impact that has been achieved, but also what impact is of greatest value to different stakeholders involved in the engagement process.
Better Evaluation: Collect and/or Retrieve Data. This guide provides a list of different tools and methods that can be used to collect data to inform your evaluation.
The World Cafe: World Cafe Method. A methodology for facilitating discussions in larger groups, which could be used to facilitate feedback and evaluation from the community on the engagement process.
Photovoice: Photovoice is a participatory photography method that seeks to empower people to share their experiences through photographic storytelling.

The University of Kansas, The Community Toolbox: Chapter 3, Section 20. Implementing Photovoice in Your Community. Describes what Photovoice is, who should use it, why and when. Provides guidance on how to implement a Photovoice project.
Wang and Burris (1997): Photovoice: concept, methodology, and use for participatory needs assessment. Provides an overview of the development of the photovoice concept, advantages and disadvantages, key elements, participatory analysis, materials and resources, and implications for practice.

Page updated

Report abuse