Course Materials

LECTURE SLIDES, CASE STUDIES AND EXERCISES

Lectures

• Lecture 1: What is Evaluation?

• Lecture 2: Theory of Change & Measurement

• Lecture 3: Why Randomize

• Lecture 4: How to Randomize

• Lecture 5: Choosing the Right Sample Size

• Lecture 6: Comparative Cost-Effectiveness Analysis

• Lecture 7: Threats and Analysis

• Lecture 8: Randomized Evaluation from Start to Finish

• Lecture 9: Applying and Using Evidence

• Lecture 10: The Generalizability Framework

Case Studies and Exercises

• Case Study 1: Community-based Monitoring of Healthcare in Uganda

• Case Study 2: Evaluating the Learn to Read Program in India

• Case Study 3: Promoting economic inclusion and resilience in Niger

• Case Study 4: Rural Sanitation in Indonesia

• Exercise 1: How to do Power Calculations

• Exercise 2: Navigating Design Challenges

extra resources for Lectures

Theory of Change and Measurement

• 2020 webinar series lecture on Theory of Change and Measurement

• J-PAL Research Resource: The Elements of a Randomised Evaluation

• J-PAL Research Resource: Introduction to Measurement and Indicators

• IPA’s “Guiding Your Program to Build a Theory of Change”

Why Randomise

• J-PAL Research Resource: Introduction to Randomised Evaluations

• J-PAL’s table of Impact Evaluation Methods

• J-PAL’s Advantages of Randomised Evaluations

• J-PAL’s Common questions and concerns about Randomised Evaluations

How to Randomise

• J-PAL Research Resource: Randomisation

• J-PAL’s Real-World Challenges to Randomisation and Their Solutions

• J-PAL Research Resource: Data Analysis - resource on Threats and Analysis

Sampling and Power

• J-PAL Research Resource: Power calculations

• J-PAL Research Resource: Quick guide to power calculations

• J-PAL's Sample size and power calculations code in Stata and R

• J-PAL’s Six Rules of Thumb for Understanding Statistical Power

• EGAP’s Power calculator

• EGAP’s 10 Things to know about statistical power

Cost Effectiveness Analysis

• J-PAL Research Resource: Conducting Cost-effectiveness Analysis

• J-PAL's Costing Guidelines

• Paper by Dhaliwal et al (2012): Comparative Cost-effectiveness Analysis to Inform Policy in Developing Countries

• J-PAL blog: Infinitely cost-effective? Common misconceptions in cost-effective analyses

• J-PAL Policy Bulletin: Roll call: Getting children into school

• J-PAL policy insight: Tailoring instruction to students’ learning levels to increase learning

Generalisability

• Bates and Glennerster (2017): “The Generalisability Puzzle,” Stanford Social Innovation Review

• J-PAL Evidence to Policy

• J-PAL Self-Guided Case Study on Applying the Generalisability Framework to Complex Health Care

extra resources for Case studies

The course booklet contains the case study guide and questions; some additional readings are provided below.

Case Study 1: Community-based Monitoring of Healthcare in Uganda

• J-PAL evaluation summary: Community-Based Monitoring of Primary Healthcare Providers in Uganda

• Related paper by Bjorkman et al (2009): Power to the people - Evidence from a randomized field experiment on community-based monitoring in Uganda

• Related paper by Bjorkman et al (2017): Experimental Evidence on the  Long-Run Impact of Community-Based Monitoring

Case Study 2: Evaluating the Learn to Read Program in India

• Related paper by Banerjee et al (2010): Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in India

Case Study 3: Promoting Economic Inclusion and Resilience in Niger

• J-PAL evaluation summary: Promoting Productive Inclusion and Resilience in National Safety Nets: A Four-Country Evaluation in the Sahel

• Related paper by Bossuroy et al (2022): Tackling psychosocial and capital constraints to alleviate poverty

Case Study 4: Rural Sanitation in Indonesia

• J-PAL evaluation summary: The Impact of a Large-Scale Community-Led Total Sanitation Program in Indonesia 

• Related paper by Cameron et al (2019): Scaling up sanitation: Evidence from an RCT in Indonesia

• Related paper by Cameron et al (2022): The dirty business of eliminating open defecation: The effect of village sanitation on child height from field experiments in four countries

An additional resource on how to interpret multivariate regression results is available here.

Group project

During the course, we set aside time for each group to work on a group project. This is a chance to apply the lessons from the lectures, case studies and exercises to a worked example. This could be a real world example - i.e. a current or planned randomised evaluation that one or more members of your group will be working on - or it could be something made up. 

Each group will present their project to the presenters and participants of the course on Friday. Presentations should be kept to 15 minutes, allowing for 15 minutes of discussion led by speakers and J-PAL staff. 

Template slides are available here. While each group does not need to follow this format exactly, the presentation should include the following topics:

• Brief project background

• Theory of change

• Evaluation question

• Outcomes

• Evaluation design (randomisation method)

• Data and sample size

• Potential threats and how to manage them

• Dissemination strategy for results

WORKBOOK

For those interested in running their own randomised control trial, here is an additional workbook resource. The workbook is designed to serve as a resource for you to consider each step in the design of a randomised evaluation of a program or policy you work on. While there is no requirement to complete this during the course, the workbook contains additional knowledge and resources as well as exercises to work through over the course of the week or in the future.

The workbook is designed via Google slides and is best accessed via a Google account. If you have a Google account, please follow these steps:

• Click this link

• Immediately you will be asked “Would you like to make a copy of ESP 2023 Workbook?”

• Please click the blue “Make a Copy” button.

• The workbook will then be copied to your personal Google drive where you can rename it and begin to work through it.

If you do not have a Google account, you can create one for free and follow the steps above. If you are not interested in creating a Google account, you can follow this link to download the workbook to a different file format. You can do so by clicking ‘File’ > ‘Download as’ and then choose PDF or PowerPoint. Please note that the workbook has optimal functionality in Google slides and this is the preferred access method.