Haddaway 2015

Appraisal of: Haddaway NR, Collins AM, Coughlin D, Kirk S (2015) The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PLoS ONE 10(9): e0138237.

Reviewer(s): 

Melissa Walter

Carolyn Spry

Full Reference: 

Haddaway NR, Collins AM, Coughlin D, Kirk S (2015) The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PLoS ONE 10(9): e0138237.

Short description: 

The authors sought to answer five questions about the retrieval of grey literature from Google Scholar as opposed to Web of Science, using strategies and included studies from 7 published environmental systematic reviews. They assessed the types of grey literature retrieved from Google Scholar; examined the overlap in retrieval between the two databases, and the extent of duplication within each database; and assessed the ability of Google Scholar to retrieve 1) included articles from the sample SRs and 2) grey literature located in organizational websites. They concluded that Google Scholar is not an appropriate substitute for database searching, since it has low overlap with Web of Science and therefore misses a large number of studies. However, it is a useful addition, particularly for grey literature searching – the authors found that title searching in Google Scholar is particularly likely to pull up grey literature results such as conference proceedings and theses.

Limitations stated by the author(s): 

None listed. 

Limitations stated by the reviewer(s): 

While the study uses seven published SRs as its sample, some of the five research questions do not make use of the full set of seven SRs; one part of question 4 used four of those seven SRs as well as two others not previously listed; and question 5 used only one of the seven SRs. It is not clear why the authors chose different case studies for the different research questions but it may mean that the results are less generalizable (particularly for question 5).

Two different versions of Web of Science seem to be used at different points in the article (for instance, one version includes MEDLINE and the other does not). Table 2 seems to suggest that question 1 used the MISTRA EviEM subscription to WoS, while the methods for question 3 state that Bangor University’s subscription was used. It is not clear which version of Web of Science was used for questions 4 and 5.

The authors seem to suggest that WoS contains only 'black', or non-grey, literature; however, the WoS core collection includes both conference proceedings and books/book chapters. It is not clear how many of the duplicates between Google Scholar and WoS are grey as opposed to black literature.

Since the authors used environmental systematic reviews as their evidence base, the results may not be applicable to the medical literature.

It is worth noting that the authors consider published books and book chapters to be grey literature.

Study Type: 

Single study

Related Chapters: 

Tags: 

• Grey literature

• Google Scholar

• Systematic reviews