Embedded Files
Large consultations often involve more than 30 people. User-testing is a term that refers to 'testing out' a product, a ready-made output or a prototype with a selected group of people and is commonly used in digital and marketing.
Why use a large public consultation?
A consultation is when you ask members of the public for their views, and then use these views to inform your decision-making about an issue/proposal. Consultation can be about any aspect of the research process – from identifying topics for research to thinking about the implications of research findings.
You can host a one-off meeting to ask people’s views on the importance of an issue or a study, identify concerns and/or barriers and highlight the positive benefits of something. Researchers can also consult the public about their research priorities, e.g. The James Lind Alliance.
Large consultations can be a good way to gain a wide variety of insights from a large group of people. However, please consider the following points:
What you are asking?
What you will do with the information?
How many times has this group of people/community been asked this question? Has this been done before?
Benefits of large consultations
It enables you to find out lots of different people’s views at once;
It can be useful when exploring sensitive and difficult issues;
You can get a wide range of views from different perspectives;
You can involve people in discussion and debate in a way that is not possible online or in smaller groups
The findings you get may be easier to scale up or apply to the wider population.
Challenges of consultation:
You might not get the broad views you are hoping for;
People might have previous bad experiences of consultation where their views were not listened to or acted upon;
You might require an experienced facilitator or company to recruit your participants and manage the process/facilitation;
You will need a considerable amount of planning, admin, data analysis and dissemination of findings support;
They can be very expensive, all participants will need payment for their time and contributions.
Examples of large consultations
There have been many examples of large public consultations for health and care research. Sometimes these can also be called citizens juries, public debates, workshops etc. Here are some recent ones with templates for you to use:
The One London Citizens’ Summit brought together 100 Londoners to debate and deliberate some of the complex issues around the uses of health and care data
Citizens Juries c.i.c. - provides examples of healthcare-based large public consultations
IPSOS Mori and Genomics England - A public dialogue on genomic medicine
Where to start
Large public consultations require expertise, a large budget and lots of support from public partners. You may want to consider approaching an external provider experienced in delivering this type of consultation and engagement activity.
Contact the RDNCC PPIE Team who will be more than happy to provide you with some advice and guidance. You can also refer to the University of Leeds approved supplier list for suitable agencies that can help.
User testing
User testing is often referred to when working on digital engagement platforms. We sometimes receive requests to find members of the public who can test out a process online or to review a website and/or its content.
User testing can be done in several ways, but the main aim is to identify whether a digital product or service is usable, effective and acceptable to users (the public/researchers/staff).
One of the most common user testing methods is a 1:1 usability testing session. During the session a researcher (called a “facilitator” or a “moderator”) asks a participant to perform a series of tasks, usually using one or more products or platforms. While the participant completes each task, the researcher observes the participant's behaviour and listens for feedback. Participants may be asked to think aloud as they complete tasks or reflect on their experience of using the product or prototype afterwards.
The insights you get are usually qualitative, but you can do quantitative usability testing by introducing metrics such as task completion time or clicks to completion. This activity can be online or in person.
Further guidance can be found on Gov. uk - Usability testing
Benefits of user testing:
It is a well-established way to identify common usability issues and problems;
It is relatively easy to do as you do not need any special equipment or many participants;
It is low-cost and can be done online
It is a way of gaining insights from people who cannot attend face-to-face events
Challenges of user testing:
It is a controlled environment. The insights you get may not reflect how people might naturally use the product or give any idea about the challenges they would face in real life;
It can be time-consuming to conduct usability testing with participants. You will also need to consider time for data analysis and support for participants before, during and after the sessions.;
It can be challenging to find the right participants. If participants are not right or not representative of your target audience, then the insights you gather will be limited and could be misleading.
How to carry out usability testing
Recruit around 5-6 participants (qualitative feedback) you might need more for quantitative insights
Have clear tasks
Have clear success criteria
Be impartial. Give the participant a task and then let them complete it. Try to resist influencing how they engage with the prototype or giving them too many instructions.
Remember, the task you ask participants to complete and the success criteria should reflect the stage of development of the product, e.g. you ask a user to find a specific piece of information on a web page. The success criteria would be their completing this task.
Recruit participants who are potential users. It is good to recruit people who may have been in a situation that would have led them to use your product in the last 6 months. This helps to make sure participants base their interactions on their own personal experiences rather than a hypothetical future or situation.
When running the session, it is best to have one person administering questions (facilitator) and one person taking notes (support). You may want to record the session so that it is easy to refer back to. You could record the audio, the screen and the users’ hands using the screen, but you must obtain consent for this prior to recording.
Follow up
Thank you
Always follow up with participants and provide them with a thank you. This should be done as soon as possible after the event and as a minimum by the next working day.
It is important to show public contributors that you appreciate their contributions. If you can, provide further information on any related follow-up activity where appropriate to do so.
Support
Public partners, whether taking part in user testing, large consultations or smaller activities, will still need a considerable amount of support to fulfil their role. This includes:
Admin support to organise payments, travel and accommodation if required and support with digital platforms;
A point of contact for questions before, during and after the planned activity;
An assessment of accessibility needs to allow them to contribute fully;
Assessment of content - Is it potentially triggering or upsetting for the participant?
Some content in user testing or larger consultations may involve sensitive topics and need special consideration before planning the activity. This is particularly true for health and social care-related projects. Many public partners will have personal experiences of many of the things we ask them about and may have lots of emotions attached to words/scenarios/disease types. Some questions you might want to consider before planning your activity are:
Would it be considered sensitive outside of the context of your work?
Does the topic ask about private issues that could be stressful? e.g. marriage, death, illness
Do any of the questions involve individuals revealing information that could be seen to be stigmatizing? e.g. gender
If you find that the content is sensitive, then you should consider ongoing support for participants and provide them with signposting information if required.
Payments
Large consultations are usually delivered and hosted by external organisations, who will manage payments for participants directly.
User testing participants will either be paid or not be paid for their time and input. NIHR RDNCC recommends that all public partners are remunerated for their contributions in line with NIHR payment guidance.
NIHR RDNCC teams hosted by the University of Leeds should follow this guidance: Process for payment of public partners for the University of Leeds hosted teams (NIHR RDNCC).
Remember that some public partners cannot receive payments or do not wish to due to personal circumstances or preferences. Please discuss this with your participants and contact the RDNCC PPIE Team with any queries in this instance.
Please also refer to the 'Budgeting and Payments' page of this toolkit for more information.
Page updated
Report abuse