Cooper Booth 2018

Appraisal of: Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol, 2018; 18:85 

Reviewer(s): 

Andrew Booth

Full Reference:
Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol, 2018; 18:85. doi:10.1186/s12874-018-0545-3

Short description: 

This literature review examined guidance documents and supporting studies to determine whether a shared model of the literature searching process exists in systematic reviews and how this process is reported and supported. The authors reviewed nine key guidance documents identified based on their accessibility to and prominence within United Kingdom systematic reviewing practice, including the Cochrane Handbook, Campbell Handbook, CRD's guidance, NICE guidelines, and others covering different types of reviews including intervention reviews, health technology assessments, qualitative research reviews, social science reviews, and reviews for guideline development. Supporting studies were identified through the authors' topic knowledge, pearl growing, citation chasing, and a PubMed search using the systematic review methods filter.

The relevant sections within each guidance document were read and re-read to determine key methodological stages. These stages were extracted, compared across documents, and areas of consensus and unique guidance were identified. The authors determined eight key stages relating specifically to literature searching in systematic reviews, namely who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references, and reporting the search process. For each key stage, the authors extracted specific guidance and reviewed how this guidance was situated within the published literature, noting areas of agreement, unique guidance, and gaps requiring further research.

The review found consistent reporting of these eight key stages across the nine guidance documents, suggesting consensus on a shared implicit model of literature searching. Bibliographic database searching was consistently identified as the first search method, followed by supplementary methods, though the order and extent of supplementary methods varied. Eight of nine documents emphasized that literature searching should be thorough, comprehensive, transparent, and reproducible to minimize bias. The authors conclude that while a conventional approach to literature searching exists, further research is needed to determine the suitability of using the same process for all types of systematic reviews, particularly given differences between effectiveness reviews and qualitative evidence syntheses.

Limitations stated by the author(s): 

The authors acknowledge that their focus on guidance produced in Europe (particularly the UK) and Australia is a potential limitation, though they justify this selection based on the UK's prominent position in health information retrieval science. They note the existence of other guidance documents from North America and other countries that were not included. The authors acknowledge uncertainty about the extent to which the guidance documents inter-relate or provide guidance uniquely, noting that the Cochrane Handbook is a key reference source in many other guidance documents and reviews. They state it is unclear whether broadening the sample to include North American and other international handbooks would alter the findings, though their initial review suggests it would not significantly change the conclusions. The authors also note that their review of published studies is not a systematic review of the evidence for each key stage, and acknowledge that other relevant studies may exist that could contribute to the exploration and development of the identified key stages.

Limitations stated by the reviewer(s): 

1. Limited geographical and organizational scope [Selection Bias/Generalizability]: While the authors justify their focus on UK, European, and Australian guidance based on the UK's prominence in health information retrieval, this purposive sampling approach may not fully represent international best practices. The exclusion of prominent North American guidance from organizations such as the Agency for Healthcare Research and Quality (AHRQ), the Institute of Medicine, and the Canadian Agency for Drugs and Technologies in Health (CADTH) limits the comprehensiveness of the review. Although the authors acknowledge this limitation and suggest it would not significantly alter findings, this assertion remains untested. Different healthcare systems, research cultures, and methodological traditions may have developed alternative or complementary approaches to literature searching that could enrich or challenge the identified model.

2. Potential for circular reasoning in guidance interdependence [Methodological Concern]: The authors acknowledge but do not fully address the concern that guidance documents may reference and build upon each other, particularly the Cochrane Handbook which predates many other documents (first published 1994). This creates potential circularity where consensus may reflect common ancestry rather than independent convergence on best practices. The extent of cross-referencing between documents was not systematically analyzed, making it difficult to determine whether the identified consensus represents truly independent validation of the conventional approach or simply propagation of an established paradigm. This is particularly relevant given that several guidance authors may have contributed to multiple documents or been trained in the same methodological tradition.

3. Incomplete systematic review of supporting studies [Evidence Base Limitation]: The identification of supporting studies used convenience sampling methods including pearl growing, citation chasing, author knowledge, and a single PubMed search rather than a comprehensive systematic search across multiple databases. This approach may have missed relevant empirical studies, particularly those published in library and information science journals, non-English publications, or studies using terminology not captured by the search strategy. The authors acknowledge this as a literature review rather than a systematic review, but this limits confidence in the completeness of the evidence base used to evaluate how guidance corresponds to published research. Important contradictory evidence or alternative methodological approaches may have been overlooked.

4. Lack of quality assessment for guidance documents [Critical Appraisal Gap]: The review did not apply any quality assessment criteria to the included guidance documents themselves. Not all guidance is created equal; some documents may be based on more rigorous methodology development, broader stakeholder consultation, or more systematic evidence review than others. By treating all nine guidance documents as equivalent sources, the review may give equal weight to guidance of varying methodological rigor and evidence base. For example, guidance documents developed through formal consensus processes with diverse stakeholders might merit different consideration than those authored by smaller groups or individuals. The review does not examine or report the methodology used to develop each guidance document.

5. Temporal limitations and currency concerns [Relevance Limitation]: The publication dates of the guidance documents range from 2006 to 2017, with some being subsequently updated. The review does not systematically address how the literature searching landscape has evolved over this period, particularly regarding emerging technologies, automation tools, machine learning applications, and rapid review methodologies. The conventional approach identified may not adequately reflect recent innovations in search methodology or the growing debates about proportionate searching, abbreviated search strategies for rapid reviews, and the appropriate level of comprehensiveness for different review types and contexts. The supporting literature was searched only until August 2017, meaning developments in the field since that date are not captured.

6. Insufficient exploration of context-dependency [Generalizability Concern]: While the review identifies that different guidance documents target different types of reviews (effectiveness, qualitative, social science, environmental, HTA), the analysis does not adequately explore whether the "one size fits all" conventional approach is appropriate across these diverse contexts. The authors note in their conclusions that further research is needed to determine suitability across review types, but the review itself does not systematically compare and contrast the nuances in guidance across different review contexts. Areas where guidance diverges may be just as informative as areas of consensus, yet these differences are not thoroughly analyzed. The assumption that a single conventional approach should apply across all systematic review types may itself warrant critical examination.

7. Limited stakeholder perspective [Representation Bias]: The review focuses on guidance documents and published studies primarily authored by information specialists, librarians, and systematic review methodologists. The perspectives of review commissioners, end-users, policy makers, and other stakeholders who may have different priorities regarding literature searching (e.g., balancing comprehensiveness against timeliness and resources) are not systematically incorporated. The identified conventional approach may reflect professional norms within the information specialist community rather than optimal approaches for diverse decision-making contexts. User experience research examining how different search approaches meet or fail to meet stakeholder needs would strengthen understanding of the practical utility of the conventional approach.

8. Absence of economic analysis [Resource Consideration]: The review does not systematically address the resource implications of the identified conventional approach, including time, personnel, and financial costs. While some supporting studies discuss efficiency and cost considerations, the guidance documents and the review itself do not provide clear frameworks for making evidence-based decisions about when comprehensive searching justifies its resource requirements versus when more targeted approaches might be appropriate. In an era of increasing demand for rapid evidence synthesis and limited research budgets, the lack of economic evaluation of different search approaches is a significant gap. The relationship between search comprehensiveness and review quality, and whether incremental investments in search yield proportionate improvements in review conclusions, deserves more systematic examination.

9. Unclear operationalization of key concepts [Definitional Ambiguity]: Terms like "comprehensive," "thorough," "systematic," and "transparent" recur throughout the guidance and review but lack clear operational definitions or measurable criteria. What constitutes a comprehensive search remains ambiguous—is it searching a certain number of databases, using specific supplementary methods, or achieving a particular level of sensitivity? This ambiguity may contribute to inconsistent application of the conventional approach and difficulties in evaluating search quality objectively. The review identifies this issue in discussing what constitutes a comprehensive literature search but does not resolve it or propose frameworks for standardization.

10. Limited engagement with alternative paradigms [Conceptual Limitation]: The review accepts the premise that systematic, comprehensive literature searching should be the goal and examines how to achieve it, rather than critically evaluating whether this paradigm is always appropriate or optimal. Alternative approaches such as purposive sampling for qualitative evidence synthesis, iterative searching based on information saturation, or targeted searching for rapid reviews receive limited attention. The review could benefit from more explicit examination of when and why departures from the conventional approach might be methodologically justified rather than treating them primarily as resource-driven compromises. Emerging debates about "good enough" evidence synthesis and proportionate methods are not fully engaged.

Study Type: 

Literature review (methodology review)

Related Chapters: 

• Qualitative research

• Value of using different search approaches

Tags: 

• Literature searching

• Search methodology

• Systematic reviews

• Guidance documents

• Search process

• Bibliographic databases

• Supplementary search methods

• Search strategy development

• PICO

• Comprehensive searching

• Search reporting

• Reference management

• Quality assurance

• PRESS

• Search filters

• Information specialists

• Librarians

• Evidence synthesis

• Health technology assessment

• Qualitative evidence synthesis

• B. Designing strategies - general