Author year

Appraisal of: Bedi RP, Kumar RP. Reproducibility of PsycInfo searches across platforms. Canadian Psychology-Psychologie Canadienne 2023. http://dx.doi.org/10.1037/cap0000365


Reviewer(s): 

Anna Brown, Su Golder

Full Reference: 

Bedi RP, Kumar RP. Reproducibility of PsycInfo searches across platforms. Canadian Psychology-Psychologie Canadienne 2023. http://dx.doi.org/10.1037/cap0000365

Short description: 

This study compares PsycInfo searched via APA PsycNet, Ebsco, ProQuest and Ovid platforms. 16 search queries, relating to three different topics, were run in both the basic and advanced interfaces on each platform. Numbers of citations retrieved using the “Times cited in this database” and “Find citing articles” functions were also compared for three publications. As suspected from experience and anecdotal evidence, the authors found often large (thousands of results) discrepancies between numbers of records retrieved via different platforms. Variant spellings of words (such as Canadian vs. American spellings), use of Boolean “AND,” phrase searching with or without quotation marks, truncation using an asterisk (*) and use of parentheses all appeared to be handled differently by different platforms, and in the case of Ovid, between the basic and advanced search screens. 

Consistent findings across three different topics suggests discrepancies are the norm and findings generalizable. The authors describe this as a “method bias” because searches are unreliable and irreproducible across platforms. They conclude that: “PsycInfo users should be aware that their initial search results are largely a function of which platform they have used” and that: “highly relevant articles could be missed through no fault of the searcher but rather because of the proprietary algorithms used by the vendor of the platform being used.”

Limitations stated by the author(s): 

Whilst a set of 16 search strings using different syntax across three topics (two chosen for Canadian relevance, but one for broader international relevance) was used to ensure some generalizability, further testing with different search strings should be undertaken to confirm the findings. Additionally, findings would differ over time as records are added/updated and search algorithms tweaked on each platform.

This research was designed to identify discrepancies between platforms, not to evaluate the quality or relevance of results. More research is needed to identify the effects of using different platforms on studies included in systematic reviews.

This study was able to identify that different platforms processed different syntax (such as parentheses and quotation marks) in different ways, but was not designed to explore how and why they did so.

Limitations stated by the reviewer(s): 

Test searches used for this study were identical in each platform and designed to reflect searches run by a “typical”, non-expert user. However, information specialists or other expert users familiar with the workings of different platforms might be expected to tailor their search strategies accordingly, thereby minimizing the effects of platform idiosyncrasies on the number and relevance of their search results. Note that the study authors do recommend involvement of a librarian for “high-stakes searches (e.g., for a meta-analysis)”.

The authors found that using quotation marks, asterisks, or parentheses in Ovid advanced search returned an error message and that using truncation (*) in Ovid basic search unexpectedly retrieved fewer results. The reviewers tested this and did not get an error message, but found that, as Ovid advanced search converted all search strings without an “AND” to phrases, 0 results were returned. The exact phrase “counselling psychology Canada” must not be present in any records in the database. In many similar cases, an expert searcher would be aware of what syntax works where, be able to ascertain reasons for errors or anomalies, and amend their searches accordingly.

Study Type: 

Single study

Related Chapters: 

Service providers and search interfaces

Tags: