Value of using different search approaches

Last updated:  19 December 2025

What’s new in this update

The chapter "Citation searching " was revised during the update in December 2025.

Introduction

The conventional search approach of applying Boolean logic to subject heading and free-text queries continues to dominate literature reviews as it remains an effective method for searching bibliographic databases. However, sensitivity and specificity issues relating to Boolean searching have led researchers to investigate a variety of alternative search approaches: citation searching (direct or indirect citation relationships), the use of the “similar articles” function, hand searching, and methods of automated retrieval are some examples.

Similar articles

There is evidence that the “similar articles” link in PubMed (formerly known as “related citations”) is an efficient search approach (1-3) that can be used for scoping searches or for identifying the need to update a systematic review. The results of three analyses suggest that there is real benefit in using the 'similar articles' link and a simple Boolean search in PubMed. In the study by Waffenschmidt et al (3) the combination of these two search methods resulted in 98% sensitivity when searching for randomized controlled trials (RCTs) of drugs. Sampson et al searched for various topics and the combination of the two search methods reached 90% sensitivity in the original testing (1) and 91% (2) when retested.

Citation searching

How well citation searching works depends on whether the citation network between articles is closed. Robinson et al. (4) examined citation networks of related trials and evaluated 259 meta-analyses. They concluded that citation networks are often disconnected. However, they only checked citation links between RCTs; especially in the case of indirect citation relationships, all citations in a reference list should be analysed.

The quality of forward or backward citation search techniques is significantly influenced by the coverage offered by the various citation indices. For this purpose, Gusenbauer (5) examines the citation coverage of the citation indices of 59 databases in a study based on a comprehensive analysis of 259 journal articles from various disciplines. He concludes that Google Scholar, Semantic Scholar, and Lens, among others, are leading citation indices for forward searches, while Web of Science Core Collection,Scopus, Semantic Scholar and Lens are especially useful for backward searches.

Direct citation relationships: Forward and backward citation searching

A review article by Cooper et al. (6) discusses the current literature on supplementary search techniques, such as citation searching, including the advantages and disadvantages of these techniques. A Cochrane review (7) included 12 studies examining manually checking reference lists and concluded that there is some evidence to support this method when traditional searching is difficult, but that the studies were heterogeneous and at a high risk of bias. A study by Preston et al. (8) showed that search strategies in MEDLINE and Embase identified 85% of studies on diagnostic test accuracy included in 9 systematic reviews; 24 further studies (8%) were identified by checking reference lists. A single study (9) comparing Scopus database with manual reference checking found Scopus to be equally sensitive but considerably more (62.5%) time efficient than the manual method.

Forward citation searching (e.g. “cited by” function in Google Scholar) seems to identify only few additional references when used in addition to traditional searching. Case studies of systematic or scoping reviews (10-14) showed that forward citation searching as a supplementary approach found between 0 and 14% more unique references. However, forward citation searching appears to offer some added value over Boolean searching in certain review types (e.g. scoping review (15)) or in more challenging searches (e.g. for outcome measures (16) or complex evidence (17)).

Indirect citation relationships: Co-citations searching

Since 2015, four studies have assessed the co-citations (i.e. the reference lists of articles citing key articles) of systematic or meta-analyses (sample size between 10 and 250) across a range of topics. Janssens and Gwinn (18) analysed co-citations and found that sensitivities between 79 and 82% were achieved when combining co-citation searching and direct citation searching. The number needed to screen (NNS) was reduced by 50 to 89% (median) compared with traditional searching. The approach was validated in a subsequent study (19) and achieved a median sensitivity of 75%.

Belter (20, 21) used a combination of approaches (similar articles, direct and indirect citation relationships) in the Web of Science database. He tested and modified the method in two studies and retrieved a median of 74 to 87% of the references included in Cochrane reviews.

Full-text search

Linder (16) evaluated Google Scholar's full-text search feature in order to find studies on 'outcome measurement instruments' - terms that are often omitted from the title, abstract and subject headings of articles. The keyword (full text) search using Google Scholar yielded the highest sensitivity (70%). However, one must bear in mind that searching Google Scholar is time-consuming and difficult, as its functionality (e.g. incomplete bibliographic information, no reference export) remains limited (22).

Automated retrieval methods

At present automated retrieval methods implemented e.g. in public available interfaces have not been sufficiently evaluated to decide whether they are a useful approach for performing sensitive searches. One study comparing Boolean searching with ranked querying in MEDLINE (Ovid) reported that ranked retrieval alone was not reliable for a search task requiring high recall (23, 24).

Some new approaches even dispense with the conventional Boolean search altogether, for example by combining the search and the screening process. Adam et al. (24) use an interactive screening technique with ranking of PubMed citations based on relevance followed by manual screening. However, the authors conclude that due to low sensitivity, it is not yet suitable for widespread use. Shemilt et al. (25) also dispense with the Boolean search and use a "machine learning-based recommendation model" ("AutoUpdate") to update living reviews. However, this part of the process was not part of the evaluation  

Hand searching

Several current studies comparing hand searching with electronic database searching concluded that there was little or no benefit offered by hand searching. This was found to be the case when searching for additional RCTs (25), diagnostic test accuracy (DTA) studies (26), and occupational health intervention studies published in a language other than English (27). This contradicts the conclusions of a 2007 Cochrane review which found hand searching to be an effective approach when searching for systematic reviews; however, this review is now outdated as its most recent search was performed in 2002 and its main comparison was between hand searching and the "old" (1994), three-section Cochrane Highly Sensitive Search Strategy for RCTs (28).

Reference list

(1) Sampson M, Shojania KG, McGowan J, Daniel R, Rader T, Iansavichene AE, et al. Surveillance search techniques identified the need to update systematic reviews. J Clin Epidemiol. 2008;61(8):755–62. [Publication appraisal]

(2) Sampson M. Complementary approaches to searching MEDLINE may be sufficient for updating existing systematic reviews. J Clin Epidemiol. 2016;78:108–15. [Publication appraisal]

(3) Waffenschmidt S, Janzen T, Hausner E, Kaiser T. Simple search techniques in PubMed are potentially suitable for evaluating the completeness of systematic reviews. J Clin Epidemiol. 2013;66(6):660–5. [Publication appraisal]

(4) Robinson KA, Dunn AG, Tsafnat G, Glasziou P. Citation networks of related trials are often disconnected: implications for bidirectional citation searches. J Clin Epidemiol. 2014;67(7):793–9. [Publication appraisal]

(5) Gusenbauer M. Beyond Google Scholar, Scopus, and Web of Science: An evaluation of the backward and forward citation coverage of 59 databases' citation indices. Res Synth Methods. 2024;15(5):802–17. [Publication appraisal]

(6) Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Systematic Reviews. 2017;6(1):234. [Publication appraisal]

(7) Horsley T, Dingwall O, Tetzlaff JM, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews 2011(1):MR000026. [Publication appraisal]

(8) Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking. Syst Rev. 2015;4:82. [Publication appraisal]

(9) Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Info Libr J. 2010;27(1):22–7. [Publication appraisal]

(10) Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Libr J. 2010;27(2):114–22. [Publication appraisal]

(11) Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. J Public Health Res. 2014;3(2):177. [Publication appraisal]

(12) Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014;14:73. [Publication appraisal]

(13) Kuper H, Nicholson A, Hemingway H. Searching for observational studies: what does citation tracking add to PubMed? A case study in depression and coronary heart disease. BMC Med Res Methodol. 2006;6:4. [Publication appraisal]

(14) Cantrell A, Booth A, Chambers D. A systematic review case study of urgent and emergency care configuration found citation searching of Web of Science and Google Scholar of similar value. Health Info Libr J. 2024;41(2):166–81.
[Publication appraisal]

(15) Rogers M, Bethel A, Briscoe S. Resources for forwards citation searching for implementation studies in dementia care: A case study comparing Web of Science and Scopus. Res Synth Methods. 2020;11(3):379–86.

(16) Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015;68(4):412–7. [Publication appraisal]

(17) Greenhalgh T, Peacock RC. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005;331(7524):1064–5.

(18) Janssens AC, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol. 2015;15:84. [Publication appraisal]

(19) Janssens A, Gwinn M, Brockman JE, Powell K, Goodman M. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol. 2020;20(1):25. [Publication appraisal]

(20) Belter CW. Citation analysis as a literature search method for systematic reviews. J Assn Inf Sci Tec. 2016;67(11):2766–77. [Publication appraisal]

(21) Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731–46. [Publication appraisal]

(22) Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2015;7(1):34–45. [Publication appraisal]

(23) Karimi S, Pohl S, Scholer F, Cavedon L, Zobel J. Boolean versus ranked querying for biomedical systematic reviews. BMC Med Inform Decis Mak. 2010;10:58. [Publication appraisal]

(24) Pohl S. Boolean and ranked information retrieval for biomedical systematic reviewing [Dissertation]. Melbourne: University of Melbourne; 2012.

(25) Armstrong R, Jackson N, Doyle J, Waters E, Howes F. It's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health. 2005;27(4):388–91. [Publication appraisal]

(26) Glanville J, Cikalo M, Crawford F, Dozier M, McIntosh H. Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation. Res Syn Meth. 2012;3(3):202–13. [Publication appraisal]

(27) Mattioli S, Farioli A, Cooke RM, Baldasseroni A, Ruotsalainen J, Placidi D, et al. Hidden effectiveness? Results of hand-searching Italian language journals for occupational health interventions. Occup Environ Med. 2012;69(7):522–4. [Publication appraisal]

(28) Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007(2):MR000001. [Publication appraisal]