Burns 2021

Appraisal of: Burns CS, Nix T, Shapiro RM, 2nd, Huber JT. MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLoS ONE. 2021;16(5):e0234221.

Reviewer(s): 

Anna Brown

Su Golder

Full Reference: 

Burns CS, Nix T, Shapiro RM, 2nd, Huber JT. MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLoS ONE. 2021;16(5):e0234221.

Short description: 

This study builds on a previous study (Burns et al, 2019; see below) that identified differences in the numbers of results retrieved when running equivalent MEDLINE searches across 5 different platforms: PubMed, ProQuest, EBSCOhost, Web of Science and Ovid. This later research additionally examines how results, and variations across platforms, vary over time, by re-running the same 29 searches (adjusted according to the search syntax requirements of each platform) monthly between October 2018 and September 2019. The authors also attempted to gain insights into the reasons for some of the differences between platforms, through close inspection of some individual records/search results.

Key findings included:

• results retrieved from a single query were often inconsistent both between platform and within each platform over time (indicating a change in platform search functionality in some cases)

• even for queries restricted by publication date 1950 to 2015, different search results were sometimes returned at different dates between October 2018 and September 2019. This suggests that bibliographic records can be changed even several years after initial publication

• a difference in lag time between updates from PubMed/MEDLINE and other platforms

• electronic before print publication of articles was one cause of inconsistency between platforms and over time

• as in the earlier study, specific search queries using multiple metadata fields were more likely to produce more consistent results across all 5 platforms.

Limitations stated by the author(s): 

This study did not examine which individual papers were included or omitted by each platform for each search, and was therefore unable to draw conclusions about how inconsistencies across platforms might impact clinical care or research. The remaining unknowns make it difficult to offer concrete recommendations for searchers. Relatively simple search strategies were used, rather than ‘real world’, complex searches, which could have provided insights into how differences in retrieval between platforms might affect the results of a systematic review or meta-analysis.

Comprehensive examination of differences in retrieval across platforms for each individual search term, field, limiter and different combinations of these was not possible. Future in-depth examination of each variable would be useful. PubMed search counts were chosen as the point of reference, because NLM is responsible for both MEDLINE and the PubMed interface, but this is a somewhat arbitrary selection. Using other platforms as a point of reference might be useful in explaining differences in indexing, Boolean logic, and other aspects of searching.

PubMed’s interface and search algorithms have been updated since this research (which used what is now ‘legacy’ PubMed); the findings may be different if the study was re-done using the new interface. 

Limitations stated by the reviewer(s): 

The study is quite complex and written up in high-level, academic language, making it at times difficult to understand.

Study Type: 

Single study

Related Chapters: 

Service providers and search interfaces

Supplemental publications to the study:

A talk by the author can be found here: https://guides.lib.umich.edu/TaubmanTalks/MedlinePlatforms.

Burns CS, Shapiro RM, 2nd, Nix T, Huber JT. Search results outliers among MEDLINE platforms. J Med Libr Assoc. 2019 Jul;107(3):364-73.

Tags: 

• Databases

• Search engines

• MEDLINE

• PubMed

• ProQuest

• EBSCOhost

• Web of Science

• Ovid