Module 2

Module 2 Objectives

Module Objective: Review appropriate resources for inclusion in a desk review.

By the end of the module, the learner will have achieved the following learning objectives:

• Learning Objective 1: Learner will be able to distinguish between primary and secondary sources.

• Learning Objective 2: Learner will be able to identify merits and challenges associated with different types of sources. (i.e., peer reviewed journals, reports, public speeches, grey literature, literature reviews).

• Learning Objective 3: Learner will be able to define inclusion criteria for reference materials.

• Learning Objective 4: Learner will be able to distinguish between reputable and questionable sources.

• Learning Objective 5: Learner will be able to locate information from reputable sources.

Primary and Secondary Sources

Primary sources are most commonly used in historical research and writing. Primary sources capture information that is close to the original source and are created by witnesses to or participants in a certain event. Examples of primary sources are original scientific studies, speeches, letters, maps, and photographs. Quantitative data sets can also be primary sources if they have not been manipulated by outside sources. For example, the datasets produced from the 2015 Demographic and Health Survey (DHS) in Afghanistan that report on the data collected without secondary analysis would be considered primary sources.

Secondary sources often include information from a variety of primary and other secondary sources. They typically incorporate additional analysis or new interpretations of information that have already been made available. Examples of secondary sources include articles in scholarly journals, encyclopedias, books, and reports. For example, if someone created a report after performing a statistical analysis on the datasets from the 2015 DHS in Afghanistan, that would be considered a secondary source.

Both primary and secondary sources can be utilized in the desk review process. You may need to directly reference a government data set on maternal mortality, which would be a primary source, in addition to an analysis of maternal health incorporating data, policies, and perspectives from multiple sources, which would be a secondary source. You may not be able to locate primary sources for all pieces of information; however, if a primary source is available, it is preferable to include it because it is tied more directly to the source of the information. For example, it is preferable to reference an original research article than to reference a summary of the article.

Secondary sources, particularly scholarly journal articles and existing reports from governments and non-governmental organizations, will be key pieces of information for the desk review. However, not all secondary sources are equally reliable, which will be reviewed in another section of this module.

Types of Resources

Desk reviews should rely upon a wide variety of resources to ensure thorough representation of available information. The following is a list of common resources you should consider reviewing and including in your desk review. However, this list is not exhaustive, and there may be other materials that you should review based on the focus and scope of your desk review. Remember, all of the resources you choose to use in your desk review should be directly relevant to the research framework you developed (discussed in Module 1).

• Peer-reviewed journals provide scholarly articles that have been reviewed for quality and adherence to standards of research. It is helpful to be familiar with well-known journals – for example, The Lancet, New England Journal of Medicine, and PLOS One – that are respected for rigorous publication standards. You may need certain credentials to access journals, which will be discussed in the section “Where to Find Resources?”

• Grey literature is an expansive category of resources that are not formally published but are still collected and saved for their content. Examples could include presentations, data from unpublished research, working papers, theses or dissertations, and policies. For example, the Southern African Development Community Protocol on Gender and Development is a policy that would be classified as grey literature. While grey literature is often published by reputable entities, it has not undergone independent peer review. Organizations and governments may keep grey literature on their websites or in databases. For example, you can search USAID’s Development Experience Clearinghouse for reports, evaluations, and other publications.

• Databases of statistics can be helpful sources to gather pieces of information like population size, health statistics, labor statistics, or other measures. Examples of statistical databases include World Bank Data and UN World Population Prospects. It is important to rely on databases from well-known organizations that publicize the source of the information and whether it was extrapolated. You should note if statistics have been extrapolated, or calculated through statistical methods. Extrapolated statistics are developed through a process of estimation using previously observed data. The population of a country is a commonly cited statistic in many desk reviews. Population can be directly observed in a census, estimated through a representative survey, or it can be projected using past date and estimations about growth in the country. Thus, when presenting a statistic like population, it is important to explain how the value was determined.

• Other desk reviews can serve as a starting place to locate content or identify where to look for information or resources. Searching for other desk reviews on your topic can also help you confirm that the chosen topic or area of focus has not been already exhausted. It is important that your desk review advances beyond the information collected and analyses made in existing desk reviews. The purpose of your desk review is to synthesize existing information, analyze it, and explain to your audience what the collection of information is saying or demonstrating, citing the evidence that supports your conclusions as you discuss them.

• Other forms of secondary research, like scoping reviews, systematic reviews, and literature reviews can also help you identify publications that should be included in your desk review. These forms of research were reviewed in Module 1. Because secondary research compiles and reports on a large number of existing sources, it is important to review the documents cited in the publication. If you are reading a literature review that references the conclusion of a peer-reviewed journal article, you should do your best to access the original article and ensure you understand what is presented in the original article and that it has been accurately portrayed in the literature review. In this case, you should cite both the literature review and the original article in your desk review. In addition to providing a summary of existing resources, secondary research typically adds additional analysis, conclusions, or opinions from the authors; if you are referencing a point of analysis from the secondary research publication, you should cite the secondary research publication.

• As appropriate, you may also include information from newspapers, public speeches, legal documents, polls, and websites to provide deeper context for your desk review. However, it is important to determine the reliability of the source and present that information within the desk review. If the information you find has not been corroborated in additional sources, you can include this information with a caveat about its reliability. You should be particularly careful when including information from newspapers. Newspapers may provide the only publicly available account of statements made at events. In this case, it is acceptable to cite a newspaper. However, if the newspaper article is reporting on information that can be found in a more reputable publication, you should cite that publication and not the newspaper article.

How to Find and Access Resources

There are a wide variety of methods to find resources. It is helpful to begin with large databases that host a large number of reputable resources. The following are examples of databases that you can search for publications that also provide connections to open access articles where available.

Websites for government departments or non-governmental organizations may contain reports, information on programming, or statistics that can be incorporated into your desk review. Many of these websites will have search functions to help locate sources related to your area of focus. For example, from the website for the Ministry of Health and Wellness of Botswana, you can access weekly disease surveillance and health policies.

The table below was developed by WI-HER staff to identify common, reputable sources for information that may be useful in a desk review. It is beneficial to begin looking for information with these sources but remember to verify any information you find to ensure it is accurate and up-to-date.

Open Access Content

If you do not have credentials to access certain databases or journals, there are a number of methods for sourcing or accessing open access content. The term open access means anyone is able to access the information. It is important to pursue legal options that do not violate copyright laws. The Directory of Open Access Journals lists and provides access to over 13,000 open access, peer-reviewed journals. The Open Access Button helps researchers locate legally open access versions of published articles, and also submits requests to authors to make their publications open access if they are not. You may also directly email article authors to request them to grant free access, but this process may take time. Additionally, you can install the Unpaywall extension for the Google Chrome browser, which will link directly to legal, open access versions of articles in your search process in any database or search engine. You may also be able to work with local libraries, educational institutions, or colleagues to seek out access to articles that require credentials for access.

Researches associated with educational institutions or professional societies in developing countries may be able to secure access to free published content through Research4Life, an initiative to provide access to high-quality research in health, agriculture, environment, innovation, and justice.

Evaluating Resources using Inclusion Criteria

As discussed, it is necessary to review materials critically before including them in the desk review. Before beginning the review, you should set parameters for inclusion, called inclusion criteria, to help evaluate each resource you locate. Every desk review you complete will likely have different inclusion criteria.

Publication date is a common inclusion criterion. Establishing inclusion criteria relating to publication dates helps ensure you include the most recent information available. It is typical to include information published in the last ten years, but some topics may require a broader or narrower time window. Foundational or historical literature that fall outside your parameters may need to be referenced for your review to be thorough. Further, if there is a dearth of information in your area of focus, you may need to expand your date parameters to have enough information in your review.

The reputation of the source of the material may be used to determine if it can be included. Publications that have undergone peer review, are authored by well-known individuals or organizations, or that are corroborated by other publications tend to be more reputable. It is important to research the source of the publication, including its authors, to better understand if the information can be considered to be reputable. An example of an inclusion criterion related to source reputation is only including resources that have been peer-reviewed.

Citation metrics can also be used as inclusion criterion. Citation metrics indicate how often an article has been cited. Citation metrics are often only available for formally published materials, like journal articles. These metrics can help you determine if the resource is an influential document in the field. The more an article has been cited, the more likely it has been influential – especially if it has been cited often and has been recently published. However, you should also determine if a resource has been cited frequently because it has been disproven or has controversial findings. If so, you may consider excluding the document from your review, or providing a caveat with the information.

The most important inclusion criteria for your desk review is relevance to the research framework. Sources that are not directly tied to your research question or aims should not be included in the desk review. It is important to keep this framework in mind as you locate sources.

Resource Management

As you begin collecting resources, you should track the information you are reviewing. This will help you quickly identify the source of information when you are writing the desk review. You should keep a record of the resource’s citation (to be discussed in detail in Module 3) and what the resource adds to the desk review. It will also be helpful to track the section of the desk review into which the resource should be incorporated – this will be based on the outline you develop, as discussed in Module 1. You can also make note of any missing information or critiques based on the resource.

It may be helpful to create a table or spreadsheet to track your resources. An example spreadsheet you can utilize is linked below.

Identifying Gaps in Research

As you proceed through the research process, you may note gaps in research relating to your topic of focus. If you have exhausted your resources and cannot find the information, or cannot find a reputable source of the information, you should make note of or explain the gap in the body of your desk review. Gaps in research help identify opportunities for new primary research to contribute to knowledge in your area of focus.