Non-systematic reviews

There are some topics that do not lend themselves easily to systematic review; a more creative and flexible approach is required. This may apply when research questions are theoretical, exploratory or scoping in nature as opposed to those seeking an estimate of effect in response to a focused question. 

Several types of non-systematic reviews are evident from the literature. These may be referred to in various ways, including ‘critical reviews’ or ‘scoping review’. Each label describes a different type of review activity with different methodologies. There is no real consensus about the required components for a critical review, in the way that there is for a systematic review. 

Critical reviews may be particularly well suited to hypothesis generation, and to other circumstances where more preliminary or exploratory consideration of an issue is appropriate or reviewing a topic from a humanities perspective such as a critical interpretive synthesis of evidence. Often, as in the examples given below, complex interventions or complicated observational problems may be involved or issues such as migration or stigma may be explored. This type of review would be used in circumstances where it is more important to give detailed consideration to a particular issue or a set of perspectives, than making sure that all available literature on the subject has been covered. It is similar to writing an essay that creatively develops an evidence-based argument. A critical review can demonstrate an appreciation of the limits of what is known about the subject, what research is now needed, and/or shed new light on the topic. It requires the early identification of a number of key texts that will form the substance of the review, and from which the research questions or issues can be initially defined and subsequently refined as the review progresses.

Sometimes critical reviews involve analysis of process (i.e. establishing how an approach might work as opposed to merely whether it works). Examples of this could include assessment of the experiences of those implementing interventions or elements of the context of implementation (e.g., impediments and facilitators to success). The latter may be particularly important in the evaluation of management and policy approaches, especially in relation to feasibility of implementation of complex strategies. The strengths and weaknesses of different arguments and approaches may be critiqued and areas of controversy identified. 

Scoping reviews may attempt to summarise the volume, nature and characteristics of research within a given field. Here, the emphasis may be on coverage of the literature and identification of gaps in research, including recognition of areas of research characterised by persistently poor methodological quality. Unlike a critical review, making sure that all relevant studies have been identified is what this type of review is really about. Scoping reviews are often used to map the available research literature. In so doing, they provide a readily usable guide to existing research. Typically, scoping reviews use systematic review principles and procedures to identify the literature. They differ from systematic reviews, however, in the extent to which they investigate data within included studies. It is rare for scoping reviews to do much more than describe and summarise data from primary studies: meta-analyses for example are not normally expected. A well-done scoping review, however, will be able to use the data included to answer research questions appropriate for this kind of study e.g. what research has been done on X.

Whatever type of review type is chosen, the marking criteria for dissertations must be fulfilled. This means that the treatment of primary material should not be merely descriptive. There should be suitable linking of theory and practice and reporting of methods should be transparent, as with a systematic review. The following aspects/sections would normally be recommended for inclusion in a dissertation based on a review though, because of the flexible nature of the range of review types allowed, the precise details should always be decided jointly with your dissertation supervisors:

• an addressable research question (may be broad in scope);

• a background section explaining the topic and justification of the need for the review;

• a methods section incorporating justification of methods used and citing of methodological literature if appropriate, with the following sub-sections:

  • • a guide for selecting primary studies, bearing in mind that selection may change as part of an iterative process; a range of different study designs may need to be considered;

    • a structured search strategy, possibly iterative in nature; 

    • a guide for data collection and presentation (sometimes referred to as ‘charting the data’ in scoping reviews);

    • a plan for detailed and critical analysis of some aspect of the topic (e.g. assessment of methodological quality, process or context of intervention, findings, reporting, suitability of application of theory and so on); 

    • a plan for a structured interpretation and synthesis of the assembled evidence;

• results section which should include the following:

  • • results of literature search;

    • description of primary studies, with summary of methodological quality if appropriate;

    • report of other aspects of findings, in accordance with the review question;

• discussion section which may include a summary of main findings, an account of resolution/remaining uncertainty in relation to the review question, quality of evidence (if appropriate), strengths and limitations of the review, context of findings in relation to other literature, recommendations for the future, in terms of practice and/or for research such as systematic review, dissemination plan (if appropriate) and concluding comments.

• there should be evidence of critical and analytical treatment of some aspect of the primary material;

• review methods, processes and findings must be reported in a transparent way.

Example research questions

• What are the potential impacts of global warming on population health and strategies for tackling them?

• What does the literature tell us about risk-targeted versus population-wide approaches to chronic disease prevention?

• How did stigma affect the lives of men who had sex with men during the AIDS epidemic 1980-1999?

Useful references (available via University of York Library, electronic journals)

Qualitative evidence synthesis (QES) in the Cochrane Handbook for Systematic Reviews of Interventions

Noyes J, Booth A, Cargo M, Flemming K. et al (2019) Qualitative evidence. Cochrane Handbook for Systematic Reviews of Interventions 2nd edition, pp.525-545. - https://training.cochrane.org/handbook/current/chapter-21

Overarching paper aimed at Master's level students 

Flemming K, Noyes J. (2021) Qualitative Evidence Synthesis: Where Are We at? International Journal of Qualitative Methods. 19;20:1609406921993276

Choice of QES method

Booth A, Noyes J, Flemming K , Gehardusd A, Wahlstere P ,van der Wiltf  GJ, Mozygembad K, Refolog P, Sacchinig D, Tummersf M, Rehfuess E (2018) Structured methodology review identified seven (RETREAT) criteria for selecting qualitative evidence synthesis approaches. Journal of Clinical Epidemiology 99:41-52

Reporting Guidelines for QES

Generic:

Tong, A., Flemming, K., McInnes, E. et al. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol 12, 181 (2012). https://doi.org/10.1186/1471-2288-12-181

Meta-ethnography:

France EF,  et al Improving reporting of meta-ethnography: The eMERGe reporting guidance. J Adv Nurs. 2019 May;75(5):1126-1139. doi: 10.1111/jan.13809

Additional training:

• Cochrane Learning Live webinar series on QES

• JBI Scoping Review Network

• PRISMA for Scoping Reviews Checklist