Value of using different search approaches

Last updated:  15 February 2023

What's new in this update

The chapter "automated search methods" was revised during the update in Feburary 2023. 

Introduction

The conventional search approach of applying Boolean logic to subject heading and free-text queries continues to dominate literature reviews as it remains an effective method for searching bibliographic databases. However, sensitivity and specificity issues relating to Boolean searching have led researchers to investigate a variety of alternative search approaches: citation searching (direct or indirect citation relationships), the use of the “similar articles” function, hand searching, and methods of automated retrieval are some examples.

Similar articles

There is evidence that the “similar articles” link in PubMed (formerly known as “related citations”) is an efficient search approach (1-3) that can be used for scoping searches or for identifying the need to update a systematic review. The results of three analyses suggest that there is real benefit in using the 'similar articles' link and a simple Boolean search in PubMed. In the study by Waffenschmidt et al (3) the combination of these two search methods resulted in 98% sensitivity when searching for randomized controlled trials (RCTs) of drugs. Sampson et al searched for various topics and the combination of the two search methods reached 90% sensitivity in the original testing (1) and 91% (2) when retested.

Citation searching

How well citation searching works depends on whether the citation network between articles is closed. Robinson et al. (4) examined citation networks of related trials and evaluated 259 meta-analyses. They concluded that citation networks are often disconnected. However, they only checked citation links between RCTs; especially in the case of indirect citation relationships, all citations in a reference list should be analysed.

Direct citation relationships: Checking reference lists and citation tracking

A review article by Cooper et al. (5) discusses the current literature on supplementary search techniques, such as citation searching, including the advantages and disadvantages of these techniques. 

A Cochrane review (6) included 12 studies examining manually checking reference lists and concluded that there is some evidence to support this method when traditional searching is difficult, but that the studies were heterogeneous and at a high risk of bias. A recent study by Preston et al. (7) showed that search strategies in MEDLINE and Embase identified 85% of studies on diagnostic test accuracy included in 9 systematic reviews; 24 further studies (8%) were identified by checking reference lists. A single study (8) comparing an automated system using the Scopus database with manual reference checking found the automated system to be equally sensitive but considerably more (62.5%) time efficient than the manual method.

Citation tracking (e.g. “cited by” function in Google Scholar) seems to identify only few additional references when used in addition to traditional searching. Four case studies of systematic or scoping reviews (9-12) showed that citation tracking as a supplementary approach found between 0 and 14% more unique references. However, citation tracking appears to offer some added value over Boolean searching in certain review types (e.g. scoping review (15)) or in more challenging searches (e.g. for outcome measures (13) or complex evidence (14)). 

Indirect citation relationships: Co-citations searching

Since 2015, four studies have assessed the co-citations (i.e. the reference lists of articles citing key articles) of systematic or meta-analyses (sample size between 10 and 250) across a range of topics. Janssens and Gwinn (16) analysed co-citations and found that sensitivities between 79 and 82% were achieved when combining co-citation searching and direct citation searching. The number needed to screen (NNS) was reduced by 50 to 89% (median) compared with traditional searching. The approach was validated in a subsequent study (17) and achieved a median sensitivity of 75%. The CoCites tool was recently released for public use (18).

Belter (19, 20) used a combination of approaches (similar articles, direct and indirect citation relationships) in the Web of Science database. He tested and modified the method in two studies and retrieved a median of 74 to 87% of the references included in Cochrane reviews.

Full-text search

Linder (13) evaluated Google Scholar's full-text search feature in order to find studies on 'outcome measurement instruments' - terms that are often omitted from the title, abstract and subject headings of articles. The keyword (full text) search using Google Scholar yielded the highest sensitivity (70%). However, one must bear in mind that searching Google Scholar is time-consuming and difficult, as its functionality (e.g. incomplete bibliographic information, no reference export) remains limited (21).

Automated retrieval methods

At present automated retrieval methods implemented e.g. in public available interfaces have not been sufficiently evaluated to decide whether they are a useful approach for performing sensitive searches. One study comparing Boolean searching with ranked querying in MEDLINE (Ovid) reported that ranked retrieval alone was not reliable for a search task requiring high recall (22, 23).

Some new approaches even dispense with the conventional Boolean search altogether, for example by combining the search and the screening process. Adam et al. (24) use an interactive screening technique with ranking of PubMed citations based on relevance followed by manual screening. However, the authors conclude that due to low sensitivity, it is not yet suitable for widespread use. Shemilt et al. (25) also dispense with the Boolean search and use a "machine learning-based recommendation model" ("AutoUpdate") to update living reviews. However, this part of the process was not part of the evaluation . 

Hand searching

Several current studies comparing hand searching with electronic database searching concluded that there was little or no benefit offered by hand searching. This was found to be the case when searching for additional RCTs (26), diagnostic test accuracy (DTA) studies (27), and occupational health intervention studies published in a language other than English (28). This contradicts the conclusions of a 2007 Cochrane review which found hand searching to be an effective approach when searching for systematic reviews; however, this review is now outdated as its most recent search was performed in 2002 and its main comparison was between hand searching and the "old" (1994), three-section Cochrane Highly Sensitive Search Strategy for RCTs (29).

Reference list

(1) Sampson M, Shojania KG, McGowan J, Daniel R, Rader T, Iansavichene AE, et al. Surveillance search techniques identified the need to update systematic reviews. J Clin Epidemiol. 2008;61(8):755-62. [Publication appraisal]

(2) Sampson M. Complementary approaches to searching MEDLINE may be sufficient for updating existing systematic reviews. J Clin Epidemiol. 2016;78:108-15. [Publication appraisal]

(3) Waffenschmidt S, Janzen T, Hausner E, Kaiser T. Simple search techniques in PubMed are potentially suitable for evaluating the completeness of systematic reviews. J Clin Epidemiol. 2013;66(6):660-5. [Publication appraisal]

(4) Robinson KA, Dunn AG, Tsafnat G, Glasziou P. Citation networks of related trials are often disconnected: implications for bidirectional citation searches. J Clin Epidemiol 2014. p. 793-9. [Publication appraisal]

(5) Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017;6(1):234. [Publication appraisal]

(6) Horsley T, Dingwall O, Tetzlaff JM, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2009(1):MR000026. [Publication appraisal]

(7)  Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking. Syst Rev. 2015;4:82. [Publication appraisal]

(8) Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Info Libr J. 2010;27(1):22-7. [Publication appraisal]

(9) Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Libr J. 2010;27(2):114-22. [Publication appraisal]

(10) Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. J Public Health Res. 2014;3(2):177. [Publication appraisal]

(11) Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014;14:73. [Publication appraisal]

(12) Kuper H, Nicholson A, Hemingway H. Searching for observational studies: what does citation tracking add to PubMed? A case study in depression and coronary heart disease. BMC Med Res Methodol. 2006;6:4. [Publication appraisal]

(13) Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015;68(4):412-7. [Publication appraisal]

(14) Greenhalgh T, Peacock RC. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005;331(7524):1064-5. 

(15) Rogers M, Bethel A, Briscoe S. Resources for forwards citation searching for implementation studies in dementia care: A case study comparing Web of Science and Scopus. Res Synth Methods. 2020;11(3):379-86. 

(16) Janssens AC, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol. 2015;15:84. [Publication appraisal]

(17) Janssens A, Gwinn M, Brockman JE, Powell K, Goodman M. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol. 2020;20(1):25. [Publication appraisal]

(18) CoCites 2021 [cited 28.10.2021]. Available from: http://cocites.com.

(19) Belter CW. Citation analysis as a literature search method for systematic reviews. J Assn Inf Sci Tec. 2016;67(11):2766-77. [Publication appraisal]

(20) Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731-46. [Publication appraisal]

(21) Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2015;7(1):34-45. [Publication appraisal]

(22) Karimi S, Pohl S, Scholer F, Cavedon L, Zobel J. Boolean versus ranked querying for biomedical systematic reviews. BMC Med Inform Decis Mak. 2010;10:58.  [Publication appraisal]

(23) Pohl S. Boolean and ranked information retrieval for biomedical systematic reviewing [Dissertation]. Melbourne: University of Melbourne; 2012. 

(24) Adam GP, Pappas D, Papageorgiou H, Evangelou E, Trikalinos TA. Title of Project: A Novel Tool that Allows Interactive Screening of PubMed Citations Showed Promise for the Semi-Automation of Identification of Biomedical Literature. J Clin Epidemiol. 2022. 

(25) Shemilt I, Arno A, Thomas J, Lorenc T, Khouja C, Raine G, et al. Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research [version 1; peer review: 2 approved with reservations]. Wellcome Open Research. 2021;6(210). 

(26) Armstrong R, Jackson N, Doyle J, Waters E, Howes F. It's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health. 2005;27(4):388-91. [Publication appraisal]

(27) Glanville J, Cikalo M, Crawford F, Dozier M, McIntosh H. Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation. Res Syn Meth. 2012;3(3):202-13. [Publication appraisal]

(28) Mattioli S, Farioli A, Cooke RM, Baldasseroni A, Ruotsalainen J, Placidi D, et al. Hidden effectiveness? Results of hand-searching Italian language journals for occupational health interventions. Occup Environ Med. 2012;69(7):522-4. [Publication appraisal]

(29) Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007(2):MR000001. [Publication appraisal]