Our Approach
Our approach to developing insights to share in this Knowledge Exchange has been pragmatic:
Researcher in residence: the evaluation teams are embedded in services in order to facilitate exchange of ideas and data. The researcher learns from the system they study and the system learns from the researcher.
Realist: our methodological approach was realist-informed, in that we aimed to understand what kind of impact new CMHF resources might have and how context has shaped these responses.
Collaborative interpretation and synthesis: the core knowledge exchange team examined findings across sites and then used a realist-informed approach to draw tentative conclusions, as well as to recognise where we have gaps in understanding. Several of the researchers on the team work from a lived experience perspective.
Researcher in residence
The researcher in residence (RiR) model was used in all four evaluation sites. In this model, a researcher in residence (also known as an 'embedded researcher') is invited to be part of each system. A member of staff from either McPin or the University of Plymouth conducted the evaluation in each site (with honorary contracts), alongside the support of the wider research team. RiR models capitalise on both the insider knowledge of in-house evaluations and the academic expertise provided by outsourcing evaluations. The teams RiRs become honorary members of Trust staff, building relationships with key parts of the community system. They were also able to access Trust and ICS data through the Trust’s usual governance procedures. This allowed the evaluations to be designed in a way that most benefited the local system, and to capture what was happening in the system, including what changes were like for staff, service users and carers.
Realist methods
Our research was realist-informed in approach. Realist Evaluation is a methodological approach to undertaking research and evaluation. It derives from Critical Realism (Bashkar, 2010) and Pawson and Tilley (1997). Often, traditional research methods ask only ‘what works?’, whereas realist approaches acknowledge that different things work for different people, in different places, and at different times. Realists aim to understand what works, under what circumstances, for whom it works, and why. Therefore, Realist evaluations aim to inform us how to adapt service changes to make them work in different situations. In our evaluations we explored how a range of important outcomes were generated, at least in part, by key new resources (roles, interventions, as well as leadership, culture and rules/protocols) and the responses to these (thinking and acting differently).
Collaborative interpretation and synthesis
In order to generate insights about transformation related to implementation of the CMHF we brought together results from each of the sites. The focus of enquiry in each site was different allowing us to identify common and distinct issues. Discussions within the evaluation team and with staff from sites led us to focus our overall interpretation on key issues such as leadership, roles, workforce and culture. We continued to use a broad realist method to explore how in different sites very different resource mechanisms (staff roles, leadership activities, ‘flow rules’) were put in place resulting in varied impact. The ideas we developed were then tested and refined through conversations, local and national presentations and engagement with a broad range of stakeholders.
Strengths and limitations
This work was carried out with the dual aims of supporting systems and generating more widely applicable knowledge. By working across a diverse range of systems with different models we were able to start to understand the wide range of factors which need attention. However, we were not funded to carry out detailed cross-case analysis so our conclusions are tentative, and from a realist perspective focus more on mechanisms (system resources and responses to those) and less on how contextual factors might determine or influence those responses. We have lived experience in the research team, but for a variety of operational reasons our ambitions to elicit experiences of individuals have only been met recently in the Buckinghamshire evaluation.
Ethics and governance
The three original CMHF evaluations were categorised as local evaluations according to the Health Research Authority (HRA) algorithm and local governance committees. The Buckinghamshire evaluation was initially approved as a local evaluation and then adapted and approved as a research evaluation by the University of Plymouth Faculty Research and Integrity Committee. Our interpretation also drew on research from the PARTNERS 3 project which was approved as a research study by the HRA.
References
Reclaiming Reality. A Critical Introduction to Contemporary Philosophy. Bhaskar 2010.
An introduction to scientific realist evaluation. Evaluation for the 21st century: A handbook. Pawson & Tilley 1997.