Clinical effectiveness

Last updated: 4 April 2024

What's new in this update

No new research relevant to the chapter was identified during the March 2024 update. 

Introduction

This domain focuses on the identification of evaluations of the efficacy or effectiveness of a technology (device, medicine, vaccine, procedure or system) or intervention. These evaluations focus on whether a technology works, as well as the magnitude of health benefits or harms caused by the technology. Searching for adverse effects is summarized in the SuRe Info Safety chapter, and diagnostic tests in the SuRe Info Diagnostic Accuracy chapter. Study designs used to assess clinical effectiveness of a technology or intervention include randomized controlled trials (RCTs), quasi-experimental studies, and observational studies. Clinical effectiveness searches will focus on the identification of reports of these study types (1,2).

This chapter is primarily based on Cochrane’s 2021 update of the Searching for and selecting studies chapter (Chapter 4) of the Cochrane Handbook for Systematic Reviews of Interventions (3) and the EUnetHTA guideline on information retrieval for systematic reviews and health technology assessments of clinical effectiveness (4).

This chapter is the result of extensive Cochrane work, and appraisals have not been prepared for studies cited in the Cochrane Handbook.

Sources to search

The choice and number of sources selected to search will depend on the research question, which resources can be accessed, and time and budget constraints (3,5). The Cochrane Handbook states the following databases should be searched as a minimum to identify as many relevant efficacy studies as possible and minimise the risk of publication bias:

• Cochrane Central Register of Controlled Trials (CENTRAL)

• MEDLINE

• Embase

While CENTRAL includes records from MEDLINE, Embase, clinical trial registers and other bibliographic databases, the Cochrane Handbook (section 4.3.1.1) recommends supplementary searches of MEDLINE and Embase for comprehensiveness and currency since there is a time delay between records being indexed in MEDLINE/Embase and appearing in CENTRAL.

There are various interfaces for these databases, some free and some fee-paying, check your own institution library to see which resources you can access. For more information see the SuRe Info Service providers and search interfaces page.

Subject specific databases

The 2021 Cochrane Handbook (section 4.3.1.4) states that it is highly desirable to search appropriate subject-specific databases (3). There is mixed evidence on whether searching subject-specific databases beyond Embase and MEDLINE adds value in terms of identifying additional unique references, and the decision about whether to search subject specific databases may depend on the research question or topic area (6–12).

Examples of subject-specific bibliographic databases are listed in an Appendix. Further information is available in the Technical Supplement associated with the Cochrane Handbook (13).

National/regional databases

The 2021 Cochrane Handbook (section 4.3.1.4) recommends searching national and regional databases (3), as these may index journals not included in international bibliographic databases such as MEDLINE and Embase, and could minimize the risk of language bias (14–16). There may be particular relevance in searching regional databases for certain topic areas, for example, searching Chinese databases may identify additional trials in searches on Chinese traditional medicine (17). Examples of regional and national bibliographic databases are listed in the Appendix. Further information is available in the Technical Supplement associated with the Cochrane Handbook (13).

Bethel and Rogers (2019) recommend using search summary tables to evaluate the impact of decisions regarding bibliographic database selection (5). Search summary tables make it easier to identify the resources that contribute unique records and may aid decision-making for update searches.

Clinical trial registers, ongoing studies and unpublished data

Sources of ongoing and unpublished studies should be included in the search. Searching for such studies can prove challenging, but should be undertaken to minimise bias (18–20). Ongoing studies can be found in trials registries, whose records are updated when the trials are completed and published. 

The following is a selected list of trial registries and a search engine:

• Clinicaltrials.gov

• The World Health Organization International Clinical Trials Registry Platform Search Portal (WHO ICTRP)

• The EU Clinical Trials Register (EUCTR)

Some clinical trials registries are region or topic specific (for a more exhaustive list see Chapter 4 of the 2021 Cochrane Handbook (3) or the YHEC clinical trials website).

Investigators’ documentation and clinical study reports are another source of unpublished data so contact with principal investigators or researchers may be considered. Isojarvi et al 2018 have summarised research evidence around identifying unpublished data (21) and research highlights the need to search multiple trials registers (22). Reports of RCTs and quasi-RCTs from ICTRP and ClinicalTrials.gov are now included within the CENTRAL database, however, research suggests searching CENTRAL alone may not be sufficient to identify all relevant clinical trial registrations (34).

Supplementary searching

Consider which supplementary search methods to use; sources will depend on the topic of the search (2021 Cochrane Handbook section 4.3.5). These methods may locate publications that were not found in the original search or identify other concepts that have been missed. They may also identify comments, errata, retractions or related studies (23–25).

There is evidence that suggests hand searching may identify unique studies not identified through bibliographic database searches, particularly in non-English language journals (26). However, the value of hand searching may vary across subject areas (27).

Citation index searching can help identify studies particularly where subject searches are challenging (28,29). The 2021 Cochrane Handbook (section 4.3.5) recommends screening the reference lists of previous reviews on the same topic to identify studies (3). For further information on citation searching, see the SuRe Info section on the Value of using different search approaches.

Web-searching may identify individual studies or organisations such as academic units or research bodies that publish relevant material. The usefulness of this is also topic dependent and further information is available in the Technical Supplement associated with the Cochrane Handbook (13).

Searching for grey literature (or literature not controlled by traditional commercial publishing including reports, dissertations, theses, databases of conference abstracts) is considered highly desirable for Cochrane Reviews (Cochrane Handbook section 4.3.5).  Levay (2022) describes and tests a method for systematically searching Ovid Embase for conference abstracts (35).  Other sources of grey literature are listed in the Appendix. The current Cochrane Handbook states that searching for grey literature is highly desirable (3), however, there is discussion regarding the value of grey literature for certain topics, so the time and effort spent searching grey literature sources should be carefully considered when planning the search (28,30).

Regulatory agency, manufacturer websites and clinical study reports may be useful for providing more extensive detail on interventions than is available in journal articles (21). Regulatory agencies provide access to detailed pharmaceutical submission documents such as the EMA’s public assessment reports (EPARs), FDA drug approval package or DAPs, as well as product recalls, market withdrawals or safety alerts. 

Health technology assessments (HTAs) published by national health technology assessment agencies can provide detailed information on the clinical effectiveness, economic analysis and patient related issues around new health technologies.

Designing search strategies

The 2021 Cochrane Handbook (section 4.4) provides guidance on issues to consider when designing search strategies for systematic reviews (3). The PICO model (Patient or Population or Problem; Intervention; Comparison; Outcome) is commonly used to develop the structure of a search strategy for clinical effectiveness research questions, but other frameworks are available (31).

In many bibliographic database strategies, the search is likely to have 3 sets of terms, using a combination of subject headings and free-text terms to describe:

·         the population or health problem of interest,

·         the intervention(s) evaluated,

·         eligible types of study design (for e.g. randomized controlled trials).

For further information, see the SuRe Info chapter on Search strategy development.

Search filters

Search filters are combinations of search terms designed to retrieve particular types of references, including specific methodological study designs. The 2021 Cochrane Handbook (section 4.4.7) and Glanville et al (2020) recommend using specially designed and tested filters (such as the Cochrane Highly Sensitive Search Strategies for identifying randomized trials in Ovid MEDLINE) when appropriate (3, 32). A search filter should not be used in a pre-filtered database such as CENTRAL.

Some systematic reviews may include non-randomized controlled trials (2021 Cochrane Handbook Chapter 24) (3). Search strategies for these study types can be problematic as they are not well defined or indexed consistently (33).

For further information on the sources of methodological filters (including filters to identify randomized controlled trials and observational studies), see the SuRe Info section on Search filters.

Documenting searches

For further information regarding documenting and reporting searches, refer to the SuRe Info chapter.

Reference list

(1)  Centre for Reviews and Dissemination. Systematic reviews. CRD’s guidance for undertaking reviews in health care [Internet]. 2009. 

(2) EUnetHTA. HTA Core Model. Version 3.0 for the full assessment of diagnostic technologies, medical and surgical interventions, pharmaceuticals and screening technologies [Internet]. European Network for Health Technology Assessment; 2016.  [Publication appraisal]

(3) Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf MI, et al. Chapter 4: Searching for and selecting studies. Cochrane handbook for systematic reviews of interventions. Version 6.2 (updated February 2021) . 2021. [Publication appraisal]

(4) EUnetHTA. Guideline. Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness. Version 2.0 [Internet]. European Network for Health Technology Assessment; 2019.  [Publication appraisal] 

(5)   Bethel A, Rogers M. Choosing the right databases and the right search techniques. In: Systematic searching: practical ideas for improving results. London, UK: Facet Publishing; 2019. p. 73–94. [Publication appraisal]

(6) Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complement Ther Med. 2004;12(4):228–32. [Publication appraisal]

(7)  Levay P. The contributions of MEDLINE, other bibliographic databases and various search techniques to NICE Public Health Guidance. Evid Based Libr Inf Pract. 2015;10(1).  [Publication appraisal] 

(8)  Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16:127. [Publication appraisal]

(9) Aagaard T, Lund H, Juhl C. Optimizing literature search in systematic reviews – are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol. 2016;16:161. [Publication appraisal] 

(10)  Beckles Z, Glover S, Ashe J, Stockton S, Boynton J, Lai R, et al. Searching CINAHL did not add value to clinical questions posed in NICE guidelines. J Clin Epidemiol. 2013 Sep;66(9):1051–7. [Publication appraisal]

(11) Kwon Y, Powelson SE, Wong H, Ghali WA, Conly JM. An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks. Syst Rev. 2014 Nov 11;3(1):135. [Publication appraisal]

(12) Frandsen TF, Eriksen MB, Hammer DMG, Christensen JB, Wallin JA. Using Embase as a supplement to PubMed in Cochrane reviews differed across fields. J Clin Epidemiol. 2021 Jan 8;133:24–31.  [Publication appraisal] 

(13)  Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf MI, et al. Technical supplement to Chapter 4: Searching for and selecting studies. In: Higgins, JPT, Thomas, J, Chandler, J, Cumpston, MS, Li, T, Page, MJ, Welch, VA (eds). Cochrane Handbook for Systematic Reviews of Interventions. Version 6. [Internet]. 2019.  [Publication appraisal] 

(14) Manriquez JJ. Searching the LILACS database could improve systematic reviews in dermatology. Arch Dermatol. 2009;145(8):947–8. [Publication appraisal]

(15) Atsawawaranunt K, Adams CE, Roberts S. Searching for randomised controlled trials and clinical controlled trials in Thai online bibliographical biomedical databases. Health Inf Libr J. 2011;28(1):68–76. [Publication appraisal] 

(16) Bonfill X, Osorio D, Posso M, Solà I, Rada G, Torres A, et al. Identification of biomedical journals in Spain and Latin America. Health Inf Libr J. 2015 Dec 1;32(4):276–86. [Publication appraisal]

(17) Wu XY, Tang JL, Mao C, Yuan JQ, Qin Y, Chung VC. Systematic reviews and meta-analyses of traditional chinese medicine must search chinese databases to reduce language bias. Evid Based Complement Altern Med. 2013;2013:812179. [Publication appraisal] 

(18)  Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA. 1993 Jun 2;269(21):2749–53. [Publication appraisal]

(19)  Fuller K, Bowers A, Vassar M. Clinical trial registry use in minimally invasive surgical oncology systematic reviews and meta-analyses. BMJ Evid-Based Med. 2020 Apr;25(2):1–2. [Publication appraisal]

(20)  Greiner B, Corcoran A, Wheeler D. Clinical trial registry searches are under-utilized in systematic reviews from critical care journals: A bibliometric analysis. J Crit Care. 2020 Sep 17; [Publication appraisal]

(21) Isojarvi J, Wood H, Lefebvre C, Glanville J. Challenges of identifying unpublished data from clinical trials: getting the best out of clinical trials registers and other novel sources. Res Synth Methods. 2018;9(4):561–78. [Publication appraisal] 

(22) Banno M, Tsujimoto Y, Kataoka Y. Studies registered in non-ClinicalTrials.gov accounted for an increasing proportion of protocol registrations in medical research. J Clin Epidemiol. 2019 Sep 12;116:106–13. [Publication appraisal]

(23) Allahbadia GN. Why Correcting the Literature with Errata and Retractions is Good Medical Practice? J Obstet Gynaecol India. 2014;64(6):377–80. [Publication appraisal] 

(24)  Hopewell S, Clarke MJ, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database Syst Rev 2007 Apr 1. [Publication appraisal]

(25) Farrah K, Rabb D. Errata for trial publications are not uncommon, are frequently not trivial, and can be challenging to access: a retrospective review. J Med Libr Assoc [Internet]. 2019 Apr 15 [cited 2019 Dec 10];107(2). [Publication appraisal]

(26)  Nasser M, Al Hajeri A. Comparison of handsearching versus Embase searching of the Archives of Iranian Medicine to identify reports of randomized controlled trials. Arch Iran Med. 2006;9(3):192–5. [Publication appraisal] 

(27)  Stovold E, Hansen S. Handsearching respiratory conference abstracts: a comparison with abstracts identified by an EMBASE search. In Madrid, Spain: John Wiley & Sons; 2011. [Publication appraisal]

(28)  Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017 Nov 28;6(1):234. [Publication appraisal] 

(29)  Janssens ACJW, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol. 2015 Oct 13;15:84. [Publication appraisal] 

(30)  Delaney A, Tamas PA. Searching for evidence or approval? A commentary on database search in systematic reviews and alternative information retrieval methodologies. Res Synth Methods. 2018 Mar;9(1):124–31. [Publication appraisal] 

(31)  Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Info Libr J. 2008;25:313–7. [Publication appraisal] 

(32) Glanville J, Kotas E, Featherstone R, Dooley G. Which are the most sensitive search filters to identify randomized controlled trials in MEDLINE? J Med Libr Assoc 2020,108(4):556-563. [Publication appraisal]

(33)  Li L, Smith HE, Atun R, Tudor Car L. Search strategies to identify observational studies in MEDLINE and Embase. Cochrane Methodology Review Group, editor. Cochrane Database Syst Rev [Internet]. 2019 Mar 12 [cited 2019 Dec 10]. [Publication appraisal]

(34) Banno M, Tsujimoto Y, Kataoka, Y. Using the Cochrane Central Register of Controlled Trials to identify clinical trial registration is insufficient: a cross-sectional study. BMC Medical Research Methodology, 2020; 20, 200. [Publication appraisal]

(35) Levay P. Systematic searching in Ovid Embase: understanding the MEDLINE document collection and identifying conference records. medRxiv, 2022. [Publication appraisal]