Embedded Files
Appraisal of: "Ewald, H., Klerings, I., Wagner, G., Heise, T.L., Stratil, J.M., Lhachimi, S.K., Hemkens, L.G., Gartlehner, G., Armijo-Olivo, S., & Nussbaumer-Streit, B. (2022). Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study. Journal of Clinical Epidemiology, 149, 154–164."
Appraisal of: "Ewald, H., Klerings, I., Wagner, G., Heise, T.L., Stratil, J.M., Lhachimi, S.K., Hemkens, L.G., Gartlehner, G., Armijo-Olivo, S., & Nussbaumer-Streit, B. (2022). Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study. Journal of Clinical Epidemiology, 149, 154–164."
Reviewer(s):
Juilet Brown
Alan Lovell
MS Copilot
Full Reference:
Ewald, H., Klerings, I., Wagner, G., Heise, T.L., Stratil, J.M., Lhachimi, S.K., Hemkens, L.G., Gartlehner, G., Armijo-Olivo, S., & Nussbaumer-Streit, B. (2022). Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study. Journal of Clinical Epidemiology, 149, 154–164. DOI: 10.1016/j.jclinepi.2022.05.022.
Short description:
This meta-research study evaluated the impact of limiting database searches on the coverage, recall, and conclusions of systematic reviews. The authors analysed 60 randomly selected Cochrane reviews, assessing how many included references were indexed and retrievable using combinations of MEDLINE, Embase, and CENTRAL. They examined whether reduced database coverage affected review conclusions and certainty.
The study found that searching at least two databases significantly improved coverage and recall, with MEDLINE + Embase + CENTRAL achieving a median recall of 90%. However, even this combination did not guarantee complete retrieval of relevant studies. Reviews with unchanged conclusions had higher recall and coverage, while those with altered or indeterminate conclusions had lower metrics. Notably, references not found were often older, lacked abstracts, or were grey literature. The authors recommend using supplementary search methods when relevant studies may be difficult to locate.
Limitations stated by the author(s):
Recall was calculated independently of database coverage, potentially underestimating performance.
Indexation assessments were limited to MEDLINE-only and Embase-only, possibly excluding additional records from other interfaces.
CENTRAL coverage may have been overestimated due to Cochrane indexing practices.
Search strategy quality was assumed based on Cochrane standards but not comprehensively evaluated.
Some references may have influenced narrative synthesis rather than meta-analysis, complicating recall-conclusion relationships.
Limitations stated by the reviewer(s):
The study does not fully resolve how many databases are sufficient for all review types.
The reliance on Cochrane reviews may limit generalizability to non-Cochrane contexts.
The analysis does not account for the impact of supplementary methods like citation tracking or expert consultation.
Recall metrics may be affected by the presence of non-indexed references, which are not uniformly addressed.
The study assumes high-quality search strategies without validating comprehensiveness or sensitivity.
Study Type:
Meta-research Study (Quantitative Evaluation of Database Search Performance in Systematic Reviews)
Related Chapters:
Tags:
Databases
Page updated
Report abuse