Documenting and reporting the search process

Last updated: 8 April 2020

Introduction

Thorough documentation and clear reporting are essential when conducting a comprehensive literature search for a health technology assessment or systematic review. The search encompasses one of the most important aspects of the review as the evidence base it identifies forms the foundation for the project. A faulty search can result in a flawed review, therefore, documenting all aspects of the search to showcase that it is methodologically sound aids in establishing a review’s credibility (1, 2). It can then be said that the ultimate goal of the reporting/documenting process is transparency and reproducibility with the added benefit of increasing the reader’s confidence in the research. Thorough documentation of the search also facilitates future updating of a review and may also serve as a resource for research team members should they need to refer to this information during the production of other sections of the report (1, 3).

Unfortunately, it has been found that within the health sciences literature large numbers of systematic searches (including those used in Cochrane Reviews) (4) are inadequately documented and described in the final reports of the project, leading to questions surrounding the validity of the search methodology used and rendering them irreproducible for readers (4-7).

Publication appraisals were not prepared for handbooks and guidance documents.

Background

To date, there exist many reporting and searching standards/published recommendations for systematic reviews and HTAs that touch upon what components of the search strategy to describe to achieve transparency and replicability for readers. In terms of standards, these include: The Cochrane Handbook Chapter 4 (8), MECIR (9), The PRISMA Checklist (10), The Institute of Medicine’s Standards for Systematic Reviews (11), HugeNet (12), RAMSES (13), CRD Guidance (14), ENTREQ (15), The DACEHTA Handbook (16), EUnetHTA guideline for information retrieval (17), The INAHTA Checklist (18) and the Campbell Collaboration guide (19).

In addition, a number of published studies aim to set forth guidelines for search strategy reporting, notably Booth’s article on the mnemonic STARLITE (20). The work of Rader et. al. (3), Sampson et al. (1), and Atkinson (21) involve reviewing current guidance and/or surveying search practitioners on search strategy documentation. They also provide additional suggestions in areas where information is lacking (1, 3, 21-25).

There is, however, high variability among all these recommendations and sometimes little consensus as to what elements of the search are essential to describe and also what level of detail should be provided for each element (1, 22). The publication of PRISMA-S, an extension to the internationally accepted PRISMA statement, may address some of the variability of reporting of searches in systematic reviews. Publication of PRISMA-S is expected sometime in 2020 (26).

An Inventory of Search Documentation/Reporting Elements and Methods

The attached table represents a synthesis of the literature (reporting standards and research articles) surrounding the documentation and reporting of search strategies. It only presents recommendations as found in the literature and therefore it may seem lacking in some areas due to a paucity of published information on those areas. The literature describes search strategy documentation and reporting for various types of products that relate to a systematic review or HTA, mainly the full report, the abstract of the report, the project protocol and the report appendices. Information for each of these types of documents is represented in the table. In addition, while sometimes used interchangeably, the activity of recording the search strategy is often divided up by the literature into two separate processes, that of documenting the search strategy and that of reporting the search strategy. Documenting the search can be seen as recording information about the search simultaneously as it is run. This information is often used for internal purposes and records. Reporting the search involves the final formal write up of the search methodology that will be included in the published version of the review/HTA. Information from the documenting phase feeds into the reporting phase (8, 9, 14). Once again information for each of these processes is described in the table below. Finally, a portion of the literature covering this area looks specifically at documenting and reporting the search process for reviews of the qualitative literature. Search strategies for qualitative reviews may encompass a different process than traditional systematic reviews of clinical interventions. The elements and methodology for reporting/documenting thus vary in some aspects and can be unique to these types of reviews. These items are also flagged in the table as being relevant for the description of search strategies with a qualitative emphasis.

Reference list

(1) Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748-54. [Publication appraisal]

(2) Booth A, Papaioannou D, Sutton A. Systematic Approaches to a Successful Literature Review. Thousand Oaks, CA: Sage; 2012. [Publication appraisal]

(3) Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Method. 2014;5(2):98-115 [Publication appraisal]

(4) Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. J Med Libr Assoc. 2009;97(1):21-9. [Publication appraisal]

(5) Faggion CM, Atieh MA, Park S. Search strategies in systematic reviews in periodontology and implant dentistry. J Clin Periodontol. 2013;40(9):883-8. [Publication appraisal]

(6) Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61(5):440-8. [Publication appraisal]

(7) Roundtree AK, Kallen MA, Lopez-Olivo MA, Kimmel B, Skidmore B, Ortiz Z, et al. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: a systematic review. J Clin Epidemiol. 2009;62(2):128-37. [Further reference details] [Publication appraisal]

(8) Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies. 4.4.8 Peer review of search strategies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook. [Publication appraisal]

(9) Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews. Cochrane: London, 2016. [Publication appraisal]

(10) Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264-9. [Publication appraisal]

(11) Eden J, Levit L, Berg A, Morton S, editors. Finding What Works in Health Care: Standards for Systematic Reviews. Washington (DC): National Academies Press; 2011. [Publication appraisal]

(12) Little J, Higgins J, Bray M, Ioannidis J, Khoury M, Manolio T, et al. The HuGENet™ HuGE review handbook, version 1.0. Ottawa, Ontario, Canada: HuGENet Canada Coordinating Centre. 2006. [Publication appraisal]

(13) Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Med. 2013;11(1):1. [Publication appraisal]

(14). Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. York, UK: Centre for Reviews and Dissemination (CRD); 2009. [Publication appraisal]

(15) Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12(1):1. [Publication appraisal]

(16) Kristensen FB, Sigmund H, editors. Health Technology Assessment Handbook. Copenhagen: Danish Centre for Health Technology Assessment, National Board of Health 2008. [Publication appraisal]

(17) European Network for Health Technology Assessment (EUnetHTA). Guideline: Process of Information Retrieval for Systematic Reviews and Health Technology Assessments on Clinical Effectiveness. EUnetHTA; 2015. [Publication appraisal]

(18) INAHTA Secretariat. A Checklist for Health Technology Assessment Reports. INAHTA; 2007. [Publication appraisal]

(19) Kugley S, Wade A, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: The Campbell Collaboration; Version 1.1 Feb 2017. [Publication appraisal]

(20) Booth AC. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94:421-9, e205. [Publication appraisal]

(21) Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Method. 2015(1):87. [Publication appraisal]

(22) Niederstadt C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. Int J Technol Assess Health Care. 2010;26(4):450-7. [Publication appraisal]

(23) Briscoe S. Web searching for systematic reviews: a case study of reporting standards in the UK Health Technology Assessment programme. BMC Res Notes. 2015;8:153-. [Publication appraisal]

(24) Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Acad Med. 2011;86(8):1049-54. [Publication appraisal]

(25) Finfgeld-Connett D, Johnson ED. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. J Adv Nurs. 2013;69(1):194-204. [Publication appraisal]

(26) Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB, PRISMA-S Group. PRISMA-S extension: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Version 5, March 02, 2020. [Publication appraisal]