Justesen 2021

Appraisal of: Justesen T, Freyberg J, Schultz ANØ. Database selection and data gathering methods in systematic reviews of qualitative research regarding diabetes mellitus - an explorative study. BMC Medical Research Methodology (2021) 21:94 . 

Reviewer(s): 

Andrew Booth

Full Reference:
Justesen T, Freyberg J, Schultz ANØ. Database selection and data gathering methods in systematic reviews of qualitative research regarding diabetes mellitus - an explorative study. BMC Medical Research Methodology (2021) 21:94

Short description: 

This exploratory study investigated optimal database combinations for conducting systematic reviews (SRs) of qualitative research regarding diabetes mellitus. The authors identified 26 SRs published between 2010 and 2020 through PubMed and systematically hand-searched 501 included references across seven databases: CINAHL, MEDLINE/PubMed, PsycINFO, Embase, Web of Science, Scopus, and Google Scholar. They calculated recall rates to determine which database combinations yielded the highest retrieval rates.

The study found that SRs searched an average of five databases, with MEDLINE/PubMed being universally used (100%) and CINAHL second most common (81%). When Google Scholar was excluded, the combination of MEDLINE/PubMed and CINAHL achieved the highest recall rate for two databases (96.4%), while MEDLINE/PubMed, CINAHL, and Embase yielded the highest for three databases (98.8%). CINAHL retrieved the most unique references (n=15) when Google Scholar was excluded. Additionally, 69% of SRs hand-searched reference lists, which contributed a median of 2.5 additional references per SR. Only one SR involved an information specialist in database selection.

The authors concluded that MEDLINE/PubMed combined with CINAHL provides optimal coverage for qualitative diabetes research, with Google Scholar serving as a useful supplement for grey literature retrieval.

Limitations stated by the author(s): 

The authors identified several important limitations. First, they only searched PubMed to identify eligible SRs, potentially missing relevant reviews indexed in other databases. Second, their search strategy used only MeSH terms without free-text terms, which may have resulted in incomplete retrieval of all qualitative diabetes SRs. Third, the findings are specific to qualitative research in diabetes mellitus and may not generalize to other disease areas or research types. Fourth, they did not investigate all commonly used databases, such as Social Science Citation Index and British Nursing Index, which could have altered their conclusions. Fifth, database presence does not guarantee retrieval with a specific search strategy, limiting direct transferability to actual SR conduct. Finally, and most significantly, they calculated recall rates based only on references actually included in the SRs rather than all potentially relevant available references, which could have substantially affected their findings and recommendations.

Limitations stated by the reviewer(s): 

Methodological limitations: The study employed a retrospective analysis of published SRs rather than prospectively testing database combinations, which means the results reflect what was found by others rather than what could be found with optimal searching. The lack of independent dual screening for both SR selection and reference extraction introduces potential selection and extraction bias. The study would have been strengthened by having two independent reviewers verify database recall for a subset of references to ensure reliability.

Sampling and scope concerns: The exclusive reliance on PubMed with only MeSH terms for identifying SRs creates significant selection bias, as qualitative methodology SRs may be better indexed in databases like CINAHL or PsycINFO. The restriction to diabetes mellitus limits generalizability. The exclusion of unpublished references and grey literature (except through Google Scholar) contradicts best practices for comprehensive systematic reviews and may underestimate the value of specialized grey literature databases.

Measurement and analysis issues: The fundamental flaw in using included references rather than all available relevant references as the denominator for recall calculations means this study measures "retrieval of what other reviewers found" rather than "retrieval of what exists in the literature." This significantly undermines the validity of the recall rate calculations. The study also does not account for database coverage overlap or examine whether unique references were substantively important to the review conclusions. The aggregation of recall rates across SRs with different quality and comprehensiveness levels may mask important variations.

Search strategy considerations: The authors acknowledge but do not adequately address that presence in a database differs from findability with actual search strategies. The study does not examine the search strategies used by the original SRs, their quality, or whether poor searches might explain why some references were missed. There is no analysis of whether the unique references found in particular databases represented important findings or were peripheral to the review questions.

Information specialist involvement: The finding that only one SR involved an information specialist is concerning but not adequately explored. The authors recommend consulting information specialists but provide no evidence about whether specialist involvement correlated with better database selection or higher recall rates.

Practical applicability: While the study provides recall percentages, it offers limited guidance on the practical trade-offs between comprehensiveness and efficiency. The recommendation to search MEDLINE/PubMed and CINAHL lacks nuance regarding different types of qualitative research questions or when additional databases become essential. The treatment of Google Scholar as supplementary may undervalue its contribution, particularly for grey literature and non-Western publications.

Study Type: 

Cross-sectional bibliometric study / Methodological research

Related Chapters: 

• Qualitative research

• Search filters

Tags: 

• Database comparison

• Systematic reviews

• Qualitative research

• Diabetes mellitus

• MEDLINE/PubMed

• CINAHL

• Embase

• Google Scholar

• Recall rates

• Literature searching

• Information specialists

• Grey literature

• Reference list searching

• Bibliographic databases