Levay 2015
Appraisal of: "Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2015;7(1):34-45."
Reviewer(s):
Elke Hausner
Siw Waffenschmidt
Full Reference:
Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2015;7(1):34-45.
Short description:
To examine how effectively forwards citation searching with Web of Science (WOS) or Google Scholar (GS) identified evidence to support public health guidance published by the National Institute for Health and Care Excellence.
The searches identified 43 relevant publications. The WOS process had 86.05% recall and 1.58% precision. The GS process had 90.7% recall and 1.62% precision. There were nine highly important publications. WOS had 100% recall and GS 88.89% recall.
WOS is recommended over GS, as citation searching was more effective, while the administrative and screening times and costs were lower.
Limitations stated by the author(s):
The comparison is limited by what might be called linking lag. GS was searched in March 2012, and more citations might have been linked to the base set by the time WOS was searched in June 2012. The WOS records do not show the date on which a citation link has been added, and so it is impossible to replicate which citations it would have retrieved in March 2012. One of the disadvantages of citation searching is that it takes time for more recent publications to be cited and for these links to be picked up in WOS or GS. It would be worth investigating the extent to which citation searching would benefit from being supplemented by some focussed searches of PubMed to ensure very recent publications are included.
It is plausible that screening higher yields would have identified further relevant studies. Google does not provide access to the full set of 15 068 results, but additional, currently unknown, publications could have been located in the available set of 7692. This study began with the realisation that it was not feasible to screen the 26 385 MEDLINE results and that it is not known whether they contain other relevant publications. The resources were not available to screen the additional GS and MEDLINE results either for the systematic review or this supplementary study.
The functionality and scope of WOS made it the most appropriate comparator with GS for the NICE review. Other resources are available for citation searching, and Scopus (http://www.scopus.com/) has been compared with WOS and GS in other studies (Falagas et al., 2008; Kulkarni et al., 2009; Li et al., 2010; Wright et al., 2014). The resources were not available to conduct a wider study, but a direct comparison between WOS, GS and Scopus would have merit.
This case study has analysed the effectiveness and efficiency of using citation searching in a single review. A comparison of how citation searching performs in other topic areas would be beneficial to assess whether these conclusions have more general applicability, for example across the other 50 areas of NICE public health guidance. It would also be worth pursuing these methods in other disciplines that deploy complex interventions, have a disparate evidence base or lack controlled trials. It would be particularly useful to examine whether citation searching can help to overcome some of the difficulties experienced in finding evidence on social care topics (Clapton, 2010) since NICE gained additional responsibilities in this area in April 2013.
The costs of developing the base set were not calculated for this study because they were the same for WOS and GS. It would be interesting to compare the cost of developing a base set for citation searching, reference harvesting or other methods with the time taken to develop, test and implement a database search strategy. Further research is required to explore how the timing of citation searching and its relationship to database searching affect its value, given the differences between this study and Wright et al. (2014).
Limitations stated by the reviewer(s):
Development of reference set not systematic: 6 relevant publications by contacting experts; 40 relevant publications by checking reference lists. Study compared performance of WOS vs GS, but not against Boolean search e.g.
Study Type:
Single study
Related Chapters:
Tags:
citation searching