Reading recommendation: How can experiments play a greater role in public policy? Twelve proposals from an economic model of scaling

How to make sure that the encouraging conclusions obtained from small scale randomized controlled trials can be carried forward to policy-makers without the risks of misguiding them into the wrong direction? This article puts forward 12 simple proposals spanning researchers, policy-makers, funders and stakeholders, which can help to address the scalability issue. It has a double virtue: it helps researchers to improve the quality of their research by pinpointing the situations when experimental results deserve to be communicated with greater care; and, by highlighting and acknowledging a number of research biases, and proposing to correct them, it also contributes to improve the credibility of well-conducted research. Although most examples are drawn from the education and health experimental literature, this article is of course extremely pertinent for REECAP members.

It explains in details what is meant by the “scale-up effect” (changes in net treatment effects, including policy costs, when changing the scale of the experiment) and clarifies the notion of evidence-based policy. It explains clearly the importance of multiple hypothesis testing, of representativeness of the population and of situations (the effect of context-dependency are often under-estimated or even totally overlooked). It draws the attention of researchers to spillovers and general equilibrium effects, which can radically change the outcome of a policy implemented at a large scale.

This article also mobilizes a simplified version of the knowledge creation market model proposed by Al-Ubaydli et al, 2019) to provide insights into the reasons why the organization of research, notably the publication bias, leads us all to the under-reporting of null results or the insufficient replication of experiments. The 12 recommendations help us individually and collectively to be more pertinent in our policy recommendations, and to craft better experimental research protocols. It is a must-read for students as well.

And for those who want to know more:

Al-Ubaydli O, J.A List and D. Sunskind (2019), The science of using science; towards an understanding of the threats to scaling experiments (Working Paper No 25848) - https://bfi.uchicago.edu/wp-content/uploads/BFI_WP_201973-1.pdf

Banerjee A.R. et al, 2017, “From proof of concept to scalable policies: challenges and solutions, with an application”, Journal of Economic Perspective 31(4): 73-102 (free download https://www.aeaweb.org/articles?id=10.1257/jep.31.4.73)

And a short version of the main arguments in the American Economic Review (but we recommend the reading of the long version): Al-Ubaydli 0, List J. and Suskind D., 2017, “What can we learn from experiments? Understanding the threats to the scalability of experimental results”, American Economic Review 107(5): 282-286