Grey literature

Last updated:  3 April 2024

What's new in this update
Formatting and text have been streamlined throughout the chapter and additional references have been added.

Introduction

This chapter discusses current evidence on health-related grey literature to inform systematic reviews and health technology assessments (HTA), including sources to search, designing search strategies, and search documentation and reporting.

GreyNet International defines grey literature as “a field in library and information science that deals with the production, distribution, and access to multiple document types produced on all levels of government, academics, business, and organization in electronic and print formats not controlled by commercial publishing i.e., where publishing is not the primary activity of the producing body” (1).

Searching grey literature sources is one component of an extensive search approach to support a systematic review or HTA (2). Grey literature can be included in all elements of HTA (3-5). Elements included in an HTA can include “clinical effectiveness, safety, costs and economic implications, ethical, social, cultural and legal issues, organizational and environmental aspects, as well as wider implications for the patient, relatives, caregivers, and the population” (6).

The search for grey literature can help identify unique studies as well as provide additional information on published studies (4, 7, 8). The Cochrane Handbook for Conducting Systematic Reviews of Interventions considers a search of grey literature sources as highly desirable to minimize the potential impacts of publication bias and to gather as large an evidence base as possible (4).

Sources to search

Based on a 2014 survey, the most commonly used grey literature sources for systematic reviews are (9):

●        The international Health Technology Assessment (HTA) database

●        WHO International Clinical Trials Registry Platform (ICTRP)

●        ClinicalTrials.gov

●        Proquest Dissertations & Theses Global

●        Conference Proceedings Citation Index

*NHS Economic Evaluation Database (NHS EED) and Database of Abstracts of Reviews of Effects (DARE) were also cited in the survey results, however both are closed to new records as of December 31, 2014.

Depending on the topic of the systematic review or element of the HTA, a combination of grey literature sources can be searched (4, 5, 9-19).

Research on grey literature sources has been conducted for the following topics:

• Addictions research (20)

• New and emerging nondrug health technologies (12)

• Nursing (21)

• Public health (10)

• Public health decision-making (22)

• Qualitative research (5)

• Religion and mental health (15)

Research on grey literature sources has been conducted for the following types of grey literature:

• Conference abstracts (23)

• Electronic theses and dissertations (24)

• Trial protocols (8)

• Unpublished adverse effects data (17)

• Unpublished data (25)

A study of 423 Cochrane systematic reviews found 39 reviews searched one or more websites (26). The websites included:

• relevant charities and non-governmental organizations

• drug manufacturers

• various government health and social care departments

In a small study, Google Scholar and Scopus were not effective sources for retrieving recent conference abstracts or other grey literature documents such as newly updated government-issued health guidelines (18).

Following an investigation of the included studies in seven environmental science systematic reviews, Haddaway et al. (19) note that some relevant information will be missed if Google Scholar is the only grey literature source searched. Instead, Google Scholar could be used to supplement a search of websites of relevant organizations (19).

Identifying and searching grey literature sources as well as managing potentially large search yields can be a challenging and time-consuming process (14, 23). To identify grey literature sources, it has been suggested we could conduct an internet search, contact experts, and hand search the websites of known relevant organizations for additional grey literature sources (2, 22). To aid in the identification of grey literature sources the Canadian Agency for Drugs and Technologies in Health (CADTH) has developed Grey Matters, an extensive grey literature search tool that outlines grey literature sources to ensure the retrieval of international HTAs, government, and evidence-based agency reports that often are not indexed in bibliographic databases (27). There are other detailed guides for conducting grey literature searches that outline numerous sources and search approaches (3, 4, 16, 28, 29).

Sources to search for several types of grey literature have been summarized in the SuReInfo Clinical Effectiveness chapter. See appendix for more details.

Grey literature sources are also suggested in topic-specific SuReInfo chapters:

●        Organizational aspects

●        Costs and economic evaluation

●        Diagnostic accuracy

●        Health problem and current use of the technology

●        Description and technical characteristics of technology

●        Safety

Designing search strategies

While there is no ‘gold standard’ for conducting a systematic grey literature search, there are strategies to develop a robust method to identify grey literature (2, 9). Effective grey literature search strategies that include a search of clinical trial registries, grey literature databases, websites of relevant organizations, conference proceedings and an internet search engine have been described in the literature (4, 10, 13, 16, 19, 22).

An important first step in designing a grey literature search strategy is to develop a search plan to document potential search sources, search terms, and limits (4, 22). Decisions can be made regarding how many pages of search results will be screened and if the scope of the grey literature search will take a narrower focus than the bibliographic database searches due to time and/or resource constraints (2, 22, 30). Stopping rules for web searching have been discussed in the literature (30).

Grey literature search strategies can utilize the search terms included in the full bibliographic database search (30, 31). However, additional effort and testing can be required when adapting search strategies for grey literature searching (14). In a sample of Cochrane reviews, the number of search terms used to search websites or search engines was reduced when compared to the corresponding Medline search (30). Searches in grey literature databases may require the search strategies to be adapted to the databases’ available functionality (2). Some databases may not provide Boolean operators, subject headings, phrase searching, wildcards, and/or truncation (14).  Mahood et al. (14) employed a two-term search strategy for grey literature databases that did not allow the full or modified search strategy to be run. Searches in Google may benefit from using several search term combinations, phrase searching, and limiting the search to the title field (2, 14).

Due to the varying nature of sources, the grey literature search strategy will need to implement a mix of search techniques, such as browsing website menus and subpages, keyword searching using multiple combinations of search terms, and tracking recurring themes, articles, and information to identify when data saturation (the point at which no new information is being found) is reached (2, 22, 26).

Grey literature searching may benefit from an iterative search process (2). After a grey literature search in Google and Google Scholar yielded only one additional eligible study, Enticott et al. (31) conducted an additional targeted grey literature search that focused on searching the websites of selected government health departments and statistical agencies. The second grey literature search identified four additional peer-reviewed publications and four high-quality government reports (31).

To effectively search for grey literature in Google Scholar research suggests limiting the search to title only (19). This search approach retrieved more conference proceedings, theses, organizational papers, and government reports than full text searching in Google Scholar (19). Haddaway et al. (19) notes that in their study sample the bulk of grey literature peaked at page 80 for full text searches and page 35 for title searches.

A survey of systematic review searchers revealed the average time to conduct a grey literature search was 7 hours (9). Half of the survey respondents spent less than 1.5 hours searching for grey literature with an average of four grey literature sources searched per systematic review (9). Overall, 27% of total searching time was dedicated to searching grey literature (9). Godin et al. (2) report that the time spent on targeted web-based searching of relevant organizations was 9-11 hours.

Documenting and reporting grey literature searches

Adams et al. (10) draws attention to the paradox of grey literature search documentation. With proper documentation a grey literature search can be replicated, but it is unlikely to lead to the same set of search results as is expected with the replication of a bibliographic database search (10). This is due in part to internet search engines returning results that are influenced by recent popularity, the user’s geographical location and search history, and other factors including the undisclosed process of result ordering in Google Scholar (10, 19, 26). Cooper (32) suggests that searchers report their geographical location when conducting Google searches.

To ensure the grey literature search is transparent and reproducible, is the Cochrane Handbook recommends to document the process carefully and keep good records (16). Using spreadsheets as a recording system to track where and how grey literature is identified, when the resource was searched, and how many relevant items were found is recommended (2, 10, 22). Keep a record of the keywords used in the grey literature search and document the various strategies used while searching internet search engines and websites (4). Briscoe (26) summarizes five reporting elements to document when searching internet search engines or websites of relevant organizations: name of search engine or website, URL of search engine or website, date of search, search strategy, and total number of results or number of results screened.

As a method to increase transparency and improve reporting of systematic reviews, research suggests saving screenshots of the first 1,000 search results in Google Scholar since only the first 1,000 results can be viewed or downloaded (19). Printing or saving electronic copies of documents or webpages found from an internet search can safeguard against losing relevant data if the information is removed or updated (4). The CADTH Grey Matters tool can also be used to ensure that grey literature searching is transparent and potentially reproducible while being conducted according to international standards (4, 27).

The reporting of grey literature searches in systematic reviews is a component of the PRISMA-S checklist (33). The PRISMA 2020 flow diagram template includes a separate section for reporting grey literature sources and the corresponding number of search results (34).

Reference list

(1)   GreyNet International. GreyNet International 2022.  

2) Godin K, Stapleton J, Kirkpatrick SI, Hanning RM, Leatherdale ST. Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Syst Rev. 2015; 4(138).  [Publication appraisal]

(3)  Giustini D. Retrieving grey literature, information, and data in the digital age. 2019. In: The handbook of research synthesis and meta-analysis. New York (NY): Russell Sage Foundation. 3rd ed. [101-26]. 

(4)  Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Marshall C, et al. Chapter 4: searching for and selecting studies. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022) 2022.  

(5) Stansfield C, Brunton G, Rees R. Search wide, dig deep: literature searching for qualitative research. An analysis of the publication formats and information sources used for four systematic reviews in public health. Res Syn Methods. 2014;5(2):142-51. [Publication appraisal]

(6) O'Rourke B, Oortwijn W, Schuller T. The new definition of health technology assessment: a milestone in international collaboration. Int J Technol Assess Health Care. 2020; 36(3):187-90. 

(7) Isojärvi J, Wood H, Lefebvre C, Glanville J. Challenges of identifying unpublished data from clinical trials: getting the best out of clinical trials registers and other novel sources. Res Syn Methods. 2018;9(4):561-78. [Publication appraisal]

(8) Sutton A, Galvan De La Cruz MC, Leaviss J, Booth A. Searching for trial protocols: a comparison of methods. Res Syn Methods. 2018;9(4):551-60. [Publication appraisal]

(9) Saleh AA, Ratajeski MA, Bertolet M. Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized. Evid Based Libr Inf Pract . 2014; 9(3):28-50. [Publication appraisal] 

(10)  Adams J, Hillier-Brown FC, Moore HJ, Lake AA, Araujo-Soares V, White M, et al. Searching and synthesising 'grey literature' and 'grey information' in public health: critical reflections on three case studies. Syst Rev. 2016; 5:164. [Publication appraisal] 

(11) Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017; 6:234. [Publication appraisal] 

(12) Farrah K, Mierzwinski-Urban M. Almost half of references in reports on new and emerging nondrug health technologies are grey literature. J Med Libr Assoc. 2019; 107(1):43-8.  [Publication appraisal] 

(13) Kreis J, Panteli D, Busse R. How health technology assessment agencies address the issue of unpublished data. Int J Technol Assess Health Care. 2014; 30(1):34-43].  [Publication appraisal] 

(14) Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Syn Methods. 2014;5(3):221-34. [Publication appraisal]

(15) Wright JM, Cottrell DJ, Mir G. Searching for religion and mental health studies required health, social science, and grey literature databases. J Clin Epidemiol. 2014; 67(7):800-810. [Publication appraisal]

(16) Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Marshall C, et al. Technical supplement to chapter 4: searching for and selecting studies. In: Cochrane Handbook for Systematic Reviews of Interventions. Version 6.3 (updated February 2022) 2022. [Publication appraisal].

(17)  Golder S, Loke YK, Wright K, Sterrantino C. Most systematic reviews of adverse effects did not include unpublished data. J Clin Epidemiol. 2016; 77:125-133. [Publication appraisal].

(18)  Bonato S. Google Scholar and Scopus for finding gray literature publications. J Med Libr Assoc. 2016; 104(3):252-4. [Publication appraisal] 

(19) Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One. 2015; 10(9):e0138237.  [Publication appraisal] 

(20) Shrivastava R, Mahajan P. Analysis of the usage and diversity of grey literature in addiction research: A study. Collection and Curation. 2021;40(3):93-9. [Publication appraisal] 

(21) Woods S, Phillips K, Dudash A. Grey literature citations in top nursing journals: a bibliometric study. J Med Libr Assoc. 2020;108(2):262-9. [Publication appraisal] 

(22) Alajajian S, Ho T.  Optimizing the identification of grey literature: a rapid review. Brampton (ON): Peel Public Health; 2018. [Publication appraisal] 

(23) Cooper C, Snowsill T, Worsley C, Prowse A, O'Mara-Eves A, Greenwood H, et al. Handsearching had best recall but poor efficiency when exporting to a bibliographic tool: case study. J Clin Epidemiol. 2020;123:39-48. [Publication appraisal] 

(24) Loan FA, Parray UY, Khan AM. Performance of search engines in harvesting grey literature. Library Hi Tech News. 2022;2:21-3. [Publication appraisal] 

(25) Ziai H, Zhang R, Chan AW, Persaud N. Search for unpublished data by systematic reviewers: an audit. BMJ Open. 2017;7(10):e017737. [Publication appraisal] 

(26) Briscoe S. A review of the reporting of web searching to identify studies for Cochrane systematic reviews. Res Syn Methods. 2018;9(1):89-99. [Publication appraisal]

(27) Grey matters: a tool for searching health-related grey literature. Ottawa (ON): CADTH; 2022. 

(28) Bonato S. Searching the grey literature: a handbook for searching reports, working papers, and other unpublished research. Lanham (MD): Rowman & Littlefield; 2018. 

(29) Balshem H, Stevens A, Ansari M, Norris S, Kansagara D, Shamliyan T, et al. Finding grey literature evidence and assessing for outcome and analysis reporting biases when comparing medical interventions. AHRQ and the Effective Health Care Program. Methods guide for comparative effectiveness reviews. Rockville (MD): Agency for Healthcare Research and Quality; 2013. 

(30) Briscoe S, Nunns M, Shaw L. How do Cochrane authors conduct web searching to identify studies? Findings from a cross-sectional sample of Cochrane Reviews. Health Info Libr J. 2020;37(4):293-318.

(31) Enticott J, Buck K, Shawyer F. Finding "hard to find" literature on hard to find groups: a novel technique to search grey literature on refugees and asylum seekers. Int J Methods Psychiatr Res. 2018; 27:e1580.  [Publication appraisal]

32. Cooper C, Lorenc T, Schauberger U. What you see depends on where you sit: The effect of geographical location on web-searching for systematic reviews: A case study. Res Synth Methods. 2021;12(4):557-70.

(33) Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: an extension to the PRISMA Statement for reporting literature searches in systematic reviews. Syst Rev. 2021; 10:39. 

(34) Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021; 372:n71.