Burns 2019

Appraisal of: Burns CS, Shapiro RM, 2nd, Nix T, Huber JT. Search results outliers among MEDLINE platforms. J Med Libr Assoc. 2019 Jul;107(3):364-73.


Anna Brown

Su Golder

Full Reference:

Burns CS, Shapiro RM, 2nd, Nix T, Huber JT. Search results outliers among MEDLINE platforms. J Med Libr Assoc. 2019 Jul;107(3):364-73.

Short description:

This study examined the variation in numbers of results when searching MEDLINE via 5 different platforms: PubMed, ProQuest, EBSCOhost, Web of Science and Ovid. The authors created 29 sets of search queries using various combinations of record fields and search operators, modified only to match the search syntax requirements of each platform.

Key findings were:

  • a range of differences in result counts between the 5 platforms for the majority of searches

  • inconsistency in variation across platforms and across different metadata/search operator combinations

  • Web of Science and ProQuest searches were the most likely to deviate from the equivalent searches in PubMed, whereas Ovid’s were the most consistent with PubMed (though with Ovid consistently retrieving fewer results than PubMed)

  • the authors were able to rule out delays between MEDLINE updates on the different platforms as the main cause of these differences in results

  • a problem was identified with explosion of MeSH terms in Web of Scienc, causing large differences in results counts compared to the other platforms for some searches

  • specific search queries using multiple metadata fields were more likely to produce more consistent results across all 5 platforms, for example combining author names with MeSH terms appeared to help disambiguate the names.

In conclusion, the different MEDLINE platforms are not equivalent to each other, and this affects the reproducibility of search strategies.

Limitations stated by the author(s):

This study used simple searches designed to compare numbers of results retrieved via different platforms; it did not use ‘real world’, complex searches. It would be useful for researchers to know how much the differences in retrieval between platforms might affect the results of a systematic review or meta-analysis.

This study examines the differences between platforms at one moment in time. Changes over time may affect the findings and conclusions and provide insight into the reasons behind variations across platforms.

PubMed search counts were chosen as the point of reference, because NLM is responsible for both MEDLINE and the PubMed interface. Using other platforms as a point of reference might be useful in explaining differences in indexing, Boolean logic, and other aspects of searching.

Whilst this research highlighted differences in results counts between platforms, it did not examine the content of records retrieved, which would have provided additional insight into the reasons behind the differences.

Limitations stated by the reviewer(s):

No additional limitations detected by the reviewers.

Study Type:

Single study

Related Chapters:

Service providers and search interfaces


  • Databases

  • Search engines


  • PubMed

  • ProQuest

  • EBSCOhost

  • Web of Science

  • Ovid