CHECK YOU'RE
UP-TO-DATE
Please bookmark this page

5 September 2024

For the Outcome (DEP) variable added guidance about how to code IEs and SRs for different levels of outcome specificity (and one more clarification for SRs):

IEs: Replaced "Outcomes should be captured at the aggregation of the taxonomy. If there are multiple outcomes at lower levels of aggregation to the taxonomy, we should capture them under one outcome. (For example if a study measures expenditure on rice, maize and wheat, we would capture these under  "Household expenditure on food")" with "Outcomes should be captured at the level of specificity of the taxonomy. Typically, there are three scenarios: 

• When a paper reports multiple outcomes more specific than any of the ones that we have in the taxonomy, and there is a single code at the lowest level of our taxonomy that captures what all the more specific outcomes are measuring, then choose that single code that captures all of them. For example, a study might report separate outcome measures for household expenditures on rice, maize and wheat. While the DEP taxonomy doesn’t include outcomes that are this specific, since these outcome measures are all examples of "Household expenditure on food", we would use that code for capturing all of these more specific outcome measures. 

• When a paper reports outcomes that fit multiple codes in our taxonomy, then use the most specific code available. You should select all applicable codes from the lowest level even if a higher code exists. For example, if a study reports “On-farm work time” and “Non-farm work time”, those should be coded as separate outcomes even if a higher level outcome such as “Overall work time” exists in the taxonomy. You would only use the “Overall work time” code if a study (also) contains a measure of work time which combines time spent on non-farm and on-farm activities.  

• When a paper reports outcomes at a level more general than the lowest level of the taxonomy, and hence cannot be classified under any existing codes, please use the “Other” option and report it to your coordinator. Occasionally, the DEP team may decide to create codes at a higher level, but in most cases there are only codes at the lowest level represented in the dropdown list. "

SRs: 

1. Replaced: "Outcomes should be captured at the aggregation of the taxonomy. If there are multiple outcomes at lower levels of aggregation to the taxonomy, we should capture them under one outcome. (For example if a study measures expenditure on rice, maize and wheat, we would capture these under  "Household expenditure on food"). Outcomes should be captured at the lowest, disaggregated level. If a review is only interested in the effects of different interventions to reduce child labour you should use the ‘Child labour’ disaggregated outcome NOT use the outcome group code (e.g. Employment) or broad group (e.g.  Human development & welfare) that it may be classified under in the taxonomy hierarchy." with a similar guidance as for the IE protocol.

2. Replaced: "You would only code outcome codes found in the included studies (not the inclusion criteria). E.g. if a review only includes studies..." with "You would only code outcome codes based on the analysis, or in the case of narrative synthesis, based on findings reported from the included studies (not the inclusion criteria). E.g. if a narrative synthesis only reports findings from studies..." [change made as the previous instruction was not entirely consistent with column G from the SR protocol which specified that it should be based on the analysis]

4 September 2024

Added the following instructions for a new Source project field which was just added to the IE and SR protocols:

New field: Source project (above the title) How to extract? "Select the project for which you are creating the record. When you select the project name from the dropdown list and click 'Yes' in the confirmation pop-up, the record will be auto-assigned to that project in the ‘Manage projects’ tab. If you have any doubts about what project to choose, please contact your project coordinator for guidance." 

15 AUGUST 2024

Updated information on the auto-populated fields under the Transparency tab of the Impact Evaluation protocol:

• If the Primary Dataset Available field has 'Does not use primary data' selected, then the 3 fields - 'Primary Dataset Location, Primary Dataset URL and Primary Dataset Format - will all be auto-populated with 'Does not use primary data'.

• If the Secondary Dataset Disclosure field has 'Does not use secondary data' selected, then the 3 fields - 'Secondary Dataset Name, Secondary Dataset Location and Additional Dataset Info- will all be auto-populated with 'Does not use secondary data'.

4 JULY 2024

Clarified the definition of a linked publication:

Linked publications – impact evaluation or systematic review* papers (articles, reports, etc.) that report findings for the same study, usually in different versions. Our definition is adopted from the What Works Clearinghouse Procedures handbook 4.1 (Section V., Subsection A., p. 9).  The key question you should ask yourself is: do these papers provide largely independent contributions to what we know about how this intervention works, or is this basically one research report split across multiple papers? If the latter is the case, we would consider two or more reports as linked publications for the purpose of the DEP and 3ie produced Evidence Gap Maps**.  

*Updates of existing systematic reviews would also be considered as a linked publication. Those are defined as “a new edition of a published systematic review with changes that can include new data, new methods, or new analyses to the previous edition” (Higgins et al. 2023 citing Garner et al .2016). 

**For DEP data extraction, we define linked publications differently than when extracting studies included in a systematic review. When producing a systematic review the key use case is not whether the papers provide a largely independent contribution but whether the effect estimates in these papers are statistically independent. 

If you are a coordinator/PM please read further here about how to identify and deal with linked/duplicate publications in the DEP admin panel.

19 JANUARY 2024

For coding SDGs:

There are a some SDG indicators whose connection to the SDG is not obvious (e.g., country-level exports being an indicator for SDG 17: Partnerships for the Goals). On the SDG coding guidance page, we have added a table showing some of these non-obvious connections, with guidance on how to code them correctly.

23 NOVEMBER 2023

For the Education outcome taxonomy:

In the DEP taxonomy, outcomes related to enrollment & attendance, educational outcomes and learning may be separated based on education level (pre-primary, primary, secondary, tertiary). In most cases it is straightforward to code, as authors specify both the grade or age of the students and if it would be considered as primary or secondary education in a given population (typically country, less often: a multicountry study). However, occasionally authors only specify the grade or age, e.g. 8th graders, or 14 year olds without mentioning if this would be considered as primary level or secondary level in a given country. In such cases, where authors only specify the grade or age and its not clear from the paper if it would be considered as primary level or secondary level in a given population, then look up what grade/age would fall under primary or secondary school for a given country in the latest standardized UNESCO ISCED 2011 classification available under the following link: https://isced.uis.unesco.org/visualizations/ . If a study does not specify if it measures primary or secondary education constructs or clearly includes studies that cover multiple levels, then use the 'Overall student attendance' or 'Overall student enrolment' code. 

02 NOVEMBER 2023

For Ethics approval:

Changed: "If an IRB certificate is found, but is not mentioned anywhere in the paper, still code “No” for this field." into: "If an IRB certificate is found, but is not mentioned anywhere in the paper, code “Yes” for this field." 

04 SEPTEMBER 2023

For Pre-Analysis Plan (impact evaluation records):

Added: "Code "Yes" if authors specify all of the following: a description of the sample to be used in the study, key data sources, hypotheses to be tested throughout the causal chain, how variables will be constructed, and the analysis method. Five points from the following checklist need to be scored as yes: https://blogs.worldbank.org/impactevaluations/a-pre-analysis-plan-checklist . " 

04 SEPTEMBER 2023

For SDG coding:

There could be multiple SDGs relevant to a paper, and the link to them may not always be explicit. For example, interventions to reduce anaemia are only mentioned in the Zero hunger section of the SDG guidance but would also be relevant for the Health and wellbeing SDGs. They ultimately aim to reduce maternal morbidity and mortality but that may not be easy to estimate and is not always specifically mentioned in the paper.  If you have any doubts please share your doubts using the procedure outlined by your coordinator.

31  AUGUST 2023

For Intervention (DEP):

Clarified points 1 and 3 (added text in bold) in the IE DEX protocol: 

1. When there are three study arms, comprising two different treatment arms and a "pure control" arm (which receives no intervention at all). The two different treatment arms should be extracted as two separate interventions.

3. When a study has two treatment arms, each of which receives a different intervention, but does NOT have a "pure control" group (which receives no intervention at all), code these as separate interventions. However, when there are two arms, both of which receive some intervention component (e.g., folic acid), while only one group receives an additional component (e.g., folic acid + iron supplements), only code the additional component received (e.g. iron supplements).

24  AUGUST 2023

For "Other" Interventions/Outcomes:

In rare cases you may find that the code does not fit into any codes from the DEP Intervention/Outcome taxonomy. There are options in the intervention and outcome dropdown list named 'Other intervention' and 'Other outcome', respectively. When you choose that option, these are the rules you MUST follow:

DEP Team:

•Log it in the DEP DEX Tracker in the ‘Other Category Suggestions’ tab. Where you MUST mention intervention/outcome, sector, broad group, definition and disaggregated intervention/outcome. There may be cases where it is difficult to specify this. You can always ask the DEP Coordinator for help in such cases.

Other projects:

•Follow the instructions of your project manager/coordinator.

21  APRIL 2023

1. We are now asking all research assistants to consult the DEX Guidance website while doing your extractions. It contains all the necessary files you need to complete the extraction, like the protocol, taxonomies, SDG guidance and equity guidance (which were previously located in your individual files). Please do NOT use any old downloaded versions of the protocol/taxonomies that you may still have saved to your computer (it might be helpful if you delete any such copies). The protocol that is linked on the website will always be the most up-to-date version

2. Please ensure you have familiarised yourself with the following pages on the website: 1) Introduction to DEX, 2) Essential Tips for Data Extraction and 3) How to Read Outcome Tables. They contain many useful hints and tips for DEX

• Reading outcome tables and identifying outcomes can be tricky – if you’re confused or you're struggling to identify the outcomes, please let me know and I will be happy to help you interpret the tables

3. Remember to look out for dark blue boxes in the taxonomies entitled “DEP Guidance”. This guidance contains many of the important DEP team decisions on how different interventions/policies should be coded where the taxonomies are unclear or make no mention of them.

4. Please remember to write the date you completed the study extraction in the DEP DEX Tracker. All assigned studies are listed in the first sheet (IE Tracker)

5. Remember if you have any questions you can use the FAQ sheet in the tracker

6. If you come across a study that is a Health Sector study, please DO NOT extract the study and let me know.

9 FEBRUARY 2023

• Added DEP guidance in the Sectors Taxonomy – any microfinance/microcredit study should be coded as Financial Sector – Other Non-Bank Financial Institutions

  • •  Some microfinance programs will be operated through banks – however in the interest of consistency, we'd like all such programs to be captured under the Other Non-Bank and Financial Institutions codes.

22 DECEMBER 2022

• Please ensure you provide sufficient detail whenever your write a question/comment in the FAQ. Fully explain your thoughts or confusions and use excerpts from the text where necessary.

17 DECEMBER 2022

• Please ensure you ALWAYS refer to the taxonomies for full and detailed definitions while you're coding - even the most experienced coder will refer to the taxonomies throughout every study extraction. You cannot get sufficient detail to justify a code by browsing the dropdown menu's on the admin panel.

18 NOVEMBER 2022

• Please ensure to always capture the minutes it takes you to extract each tab on the admin panel as accurately as possible. We are now closely using this data to help our forecasts for DEP.

11 NOVEMBER 2022

• When coding interventions and outcomes,  we have added the following sentence to the protocol

• Note: If you're absolutely sure that there is no appropriate intervention/outcome code, you can select "Other intervention/outcome". If you select this option you must suggest the following for a new intervention/outcome code: name, sector, broad group, intervention/outcome group, intervention/outcome group definition, disaggregated intervention/outcome and definition in the "Other category suggestions" sheet in the DEP DEX Tracker (or notify your core team). 

21 OCTOBER 2022

• All study titles must be in title case. This means all words should be capitalised except short prepositions, articles, and conjunctions (a, the, of, from, and, but, by, …). E.g., "Examining the Reliability of Self-Reported Data  on school Participation". Please see the DEP protocol for more detail.