What's new in this update
Update searches completed in October 2022 identified one new study for inclusion:
Levay P. Systematic searching in Ovid Embase: understanding the MEDLINE document collection and identifying conference records. medRxiv, 2022.
This domain focuses on the identification of evaluations of the efficacy or effectiveness of a technology (device, medicine, vaccine, procedure or system) or intervention. These evaluations focus on whether a technology works, as well as the magnitude of health benefits or harms caused by the technology. Searching for adverse effects is summarized in the SuRe Info Safety chapter, and diagnostic tests in the SuRe Info Diagnostic Accuracy chapter. Study designs used to assess clinical effectiveness of a technology or intervention include randomized controlled trials (RCTs), quasi-experimental studies, and observational studies. Clinical effectiveness searches will focus on the identification of reports of these study types (1,2).
This chapter is primarily based on Cochrane’s 2021 update of the Searching for and selecting studies chapter (Chapter 4) of the Cochrane Handbook for Systematic Reviews of Interventions (3) and the EUnetHTA guideline on information retrieval for systematic reviews and health technology assessments of clinical effectiveness (4).
This chapter is the result of extensive Cochrane work, and appraisals have not been prepared for studies cited in the Cochrane Handbook.
Sources to search
The choice and number of sources selected to search will depend on the research question, which resources can be accessed, and time and budget constraints (3,5). The Cochrane Handbook states the following databases should be searched as a minimum to identify as many relevant efficacy studies as possible and minimise the risk of publication bias:
While CENTRAL includes records from MEDLINE, Embase, clinical trial registers and other bibliographic databases, the Cochrane Handbook (section 184.108.40.206) recommends supplementary searches of MEDLINE and Embase for comprehensiveness and currency since there is a time delay between records being indexed in MEDLINE/Embase and appearing in CENTRAL.
There are various interfaces for these databases, some free and some fee-paying, check your own institution library to see which resources you can access. For more information see the SuRe Info Service providers and search interfaces page.
Subject specific databases
The 2021 Cochrane Handbook (section 220.127.116.11) states that it is highly desirable to search appropriate subject-specific databases (3). There is mixed evidence on whether searching subject-specific databases beyond Embase and MEDLINE adds value in terms of identifying additional unique references, and the decision about whether to search subject specific databases may depend on the research question or topic area (6–12).
Examples of subject-specific bibliographic databases are listed in an Appendix. Further information is available in the Technical Supplement associated with the Cochrane Handbook (13).
The 2021 Cochrane Handbook (section 18.104.22.168) recommends searching national and regional databases (3), as these may index journals not included in international bibliographic databases such as MEDLINE and Embase, and could minimize the risk of language bias (14–16). There may be particular relevance in searching regional databases for certain topic areas, for example, searching Chinese databases may identify additional trials in searches on Chinese traditional medicine (17). Examples of regional and national bibliographic databases are listed in the Appendix. Further information is available in the Technical Supplement associated with the Cochrane Handbook (13).
Bethel and Rogers (2019) recommend using search summary tables to evaluate the impact of decisions regarding bibliographic database selection (5). Search summary tables make it easier to identify the resources that contribute unique records and may aid decision-making for update searches.
Clinical trial registers, ongoing studies and unpublished data
Sources of ongoing and unpublished studies should be included in the search. Searching for such studies can prove challenging, but should be undertaken to minimise bias (18–20). Ongoing studies can be found in trials registries, whose records are updated when the trials are completed and published.
The following is a selected list of trial registries and a search engine:
Some clinical trials registries are region or topic specific (for a more exhaustive list see Chapter 4 of the 2021 Cochrane Handbook (3) or the YHEC clinical trials website).
Investigators’ documentation and clinical study reports are another source of unpublished data so contact with principal investigators or researchers may be considered. Isojarvi et al 2018 have summarised research evidence around identifying unpublished data (21) and research highlights the need to search multiple trials registers (22). Reports of RCTs and quasi-RCTs from ICTRP and ClinicalTrials.gov are now included within the CENTRAL database, however, research suggests searching CENTRAL alone may not be sufficient to identify all relevant clinical trial registrations (34).
Consider which supplementary search methods to use; sources will depend on the topic of the search (2021 Cochrane Handbook section 4.3.5). These methods may locate publications that were not found in the original search or identify other concepts that have been missed. They may also identify comments, errata, retractions or related studies (23–25).
There is evidence that suggests hand searching may identify unique studies not identified through bibliographic database searches, particularly in non-English language journals (26). However, the value of hand searching may vary across subject areas (27).
Citation index searching can help identify studies particularly where subject searches are challenging (28,29). The 2021 Cochrane Handbook (section 4.3.5) recommends screening the reference lists of previous reviews on the same topic to identify studies (3). For further information on citation searching, see the SuRe Info section on the Value of using different search approaches.
Web-searching may identify individual studies or organisations such as academic units or research bodies that publish relevant material. The usefulness of this is also topic dependent and further information is available in the Technical Supplement associated with the Cochrane Handbook (13).
Searching for grey literature (or literature not controlled by traditional commercial publishing including reports, dissertations, theses, databases of conference abstracts) is considered highly desirable for Cochrane Reviews (Cochrane Handbook section 4.3.5). Levay (2022) describes and tests a method for systematically searching Ovid Embase for conference abstracts (35). Other sources of grey literature are listed in the Appendix. The current Cochrane Handbook states that searching for grey literature is highly desirable (3), however, there is discussion regarding the value of grey literature for certain topics, so the time and effort spent searching grey literature sources should be carefully considered when planning the search (28,30).
Regulatory agency, manufacturer websites and clinical study reports may be useful for providing more extensive detail on interventions than is available in journal articles (21). Regulatory agencies provide access to detailed pharmaceutical submission documents such as the EMA’s public assessment reports (EPARs), FDA drug approval package or DAPs, as well as product recalls, market withdrawals or safety alerts.
Health technology assessments (HTAs) published by national health technology assessment agencies can provide detailed information on the clinical effectiveness, economic analysis and patient related issues around new health technologies.
Designing search strategies
The 2021 Cochrane Handbook (section 4.4) provides guidance on issues to consider when designing search strategies for systematic reviews (3). The PICO model (Patient or Population or Problem; Intervention; Comparison; Outcome) is commonly used to develop the structure of a search strategy for clinical effectiveness research questions, but other frameworks are available (31).
In many bibliographic database strategies, the search is likely to have 3 sets of terms, using a combination of subject headings and free-text terms to describe:
· the population or health problem of interest,
· the intervention(s) evaluated,
· eligible types of study design (for e.g. randomized controlled trials).
For further information, see the SuRe Info chapter on Search strategy development.
Search filters are combinations of search terms designed to retrieve particular types of references, including specific methodological study designs. The 2021 Cochrane Handbook (section 4.4.7) and Glanville et al (2020) recommend using specially designed and tested filters (such as the Cochrane Highly Sensitive Search Strategies for identifying randomized trials in Ovid MEDLINE) when appropriate (3, 32). A search filter should not be used in a pre-filtered database such as CENTRAL.
Some systematic reviews may include non-randomized controlled trials (2021 Cochrane Handbook Chapter 24) (3). Search strategies for these study types can be problematic as they are not well defined or indexed consistently (33).
For further information on the sources of methodological filters (including filters to identify randomized controlled trials and observational studies), see the SuRe Info section on Search filters.
For further information regarding documenting and reporting searches, refer to the SuRe Info chapter.