Frandsen 2025

Appraisal of: Frandsen TF, Moos C, Herrera Marino CIL, Eriksen MB. Supplementary databases increased literature search coverage beyond PubMed and Embase. Journal of Clinical Epidemiology, 2025;181:111704. 

Reviewer(s): 

Andrew Booth

Full Reference:
Frandsen TF, Moos C, Herrera Marino CIL, Eriksen MB. Supplementary databases increased literature search coverage beyond PubMed and Embase. Journal of Clinical Epidemiology, 2025;181:111704

Short description: 

This study investigates database coverage of publications not indexed in PubMed and Embase by analysing included references from 343 Cochrane reviews across four specialty groups (public health, incontinence, hepato-biliary, and stroke) published between 2017 and 2022. The researchers manually searched 11,822 included publications first in PubMed, then Embase, and subsequently in additional databases including Cochrane Library, PsycInfo, CINAHL, ClinicalTrials.gov, Proquest Dissertations, and Scopus. The study found that PubMed and Embase together covered 71.5% of included publications on average, with coverage varying by specialty group from 64.5% to 75.9%. Supplementary databases increased retrieval to 94.2%, with Cochrane Library providing the highest additional coverage at 62.9% of residual publications. The authors conclude that while PubMed and Embase provide substantial coverage, supplementary databases are essential for comprehensive literature searches, particularly databases relevant to specific specialties and document types.

Limitations stated by the author(s): 

The authors acknowledge several important limitations. First, the study uses Cochrane reviews as the gold standard, meaning only studies captured in these reviews were examined, which may not represent all relevant literature. The overlap between Cochrane and non-Cochrane reviews addressing the same question is generally poor, making the selection of reviews crucial. Second, Cochrane Library coverage is likely overestimated because CENTRAL indexes studies from published Cochrane reviews that are not indexed elsewhere, including personal communications. This indexing occurs after review publication, so the coverage would be lower during the actual review preparation phase, though the exact reduction cannot be determined. Third, the selection of Cochrane review groups to define topics may impact results on database coverage, as different databases may prioritize certain study types. Fourth, the study focuses solely on coverage rather than actual retrieval and search efficiency. Coverage alone does not guarantee successful retrieval, as indexed articles may not be retrieved depending on search strategies employed. Fifth, Google Scholar was excluded despite its high coverage due to limitations in systematic review contexts, including lack of advanced search features, limited export capabilities, reproducibility issues, and lack of quality control.

Limitations stated by the reviewer(s): 

Strengths: The study demonstrates several methodological strengths. The large sample size of 343 reviews and 11,822 references provides robust data across multiple specialty areas, enhancing the generalizability of findings. The systematic approach to database searching, beginning with PubMed and progressively examining additional databases, creates a clear hierarchy that reflects practical searching behaviour. The inclusion of multiple specialty groups allows for important comparisons across different medical domains, revealing meaningful variation in database coverage patterns. The authors demonstrate transparency by explicitly acknowledging the overestimation of Cochrane Library coverage and explaining why this occurs. The study addresses a practically important question for systematic reviewers by quantifying the value added by supplementary databases. The manual searching approach, while labor-intensive, ensures accuracy in determining whether publications were truly indexed. Additionally, the allowance for minor bibliographic errors in references reflects realistic conditions in published literature.

Weaknesses: Several methodological limitations warrant consideration. The restriction to only four Cochrane review groups with previously identified low PubMed/Embase coverage introduces selection bias, as these groups may not be representative of all systematic review topics. The findings may not generalize well to specialty areas with higher baseline coverage in PubMed and Embase. The retrospective design using published reviews means the study cannot fully simulate the prospective search process that reviewers actually conduct. The exclusion of Google Scholar, while justified for systematic review contexts, limits understanding of comprehensive database coverage since it is widely used by researchers. The study does not examine the incremental value of each additional database in terms of time investment versus unique citations retrieved, which is crucial information for resource-constrained reviewers. 

No inter-rater reliability assessment was reported for the manual searching process, raising questions about consistency and reproducibility of indexing determinations. The study does not analyse the characteristics of publications missed by all databases (the 5.8% not covered), which could provide valuable insights. The focus solely on coverage rather than retrieval means the practical utility of findings for informing actual search strategies remains unclear. Publication type differences between registries and conventional bibliographic databases complicate direct coverage comparisons. Finally, the study does not consider the quality or relevance of included studies, meaning that additional databases may retrieve publications that do not ultimately affect review conclusions or evidence strength.

Study Type: 

Methodological study; Database coverage analysis; Cross-sectional study

Related Chapters: 

• Qualitative research

• Value of using different search approaches

Tags: 

• Database coverage

• PubMed

• Embase

• Cochrane Library

• Systematic reviews

• Literature searching

• Information retrieval

• Search strategies

• Bibliographic databases

• CINAHL

• PsycInfo

• ClinicalTrials.gov

• Scopus

• Supplementary databases

• Evidence synthesis