Moving Beyond Local Effect Estimates:
The process of translating treatment effect estimates into concrete policy recommendations almost always requires some amount of extrapolation away from the observed moments. One of the most common versions of this involves translating the estimate of the local average treatment effects (LATE) to an estimate of the average treatment effect (ATE) or, more generally, to the policy relevant treatment parameter. I have two papers that study how researchers can use Bayesian approaches to generate the "best guess" of the relevant treatment effect given the observed data and to understand the sources of uncertainty in that guess.
From LATE to ATE: A Bayesian Approach, Journal of Econometrics, forthcoming
Link to R Package and Replication Code: https://github.com/isaacopper/BayesianMTEs
Abstract: We develop a Bayesian model that produces a posterior distribution of the marginal treatment effect (MTE) function. The method provides researchers with a principled way to extrapolate from the observed moments using flexible assumptions, thereby allowing researchers to generate plausible ranges of important and potentially policy-relevant quantities of interest. We then use the model to propose a natural decomposition of the posterior variance into “statistical uncertainty,” i.e., variance that stems from the imprecise estimation of the observed moments, and “extrapolation uncertainty,” i.e., variance that stems from uncertainty in how to extrapolate away from the observed moments. We conclude by showing that under our preferred priors, even in an experiment as large as the Oregon Health Insurance Experiment, the main source of uncertainty in the ATE comes from uncertainty in the true values of the observed moments.
A Global Regression Discontinuity Design: Theory and Application to Grade Retention Policies (with Umut Ozek)
Link to R Package and Replication Code: https://github.com/isaacopper/GlobalRDD
Abstract: We use a marginal treatment effect (MTE) representation of a fuzzy regression discontinuity setting to propose a novel estimation approach. The estimator can be thought of as extrapolating the traditional fuzzy regression discontinuity estimate or as an observational study that adjusts for endogenous selection into treatment using information at the discontinuity. We show in a frequentist framework that it is consistent for the true MTE function under weaker assumptions than existing approaches and then discuss conditions in a Bayesian framework under which it can be considered the posterior mean given the observed conditional moments. We then use this approach to examine the effects of early grade retention. We show that the benefits of early grade retention policies are larger for students with lower baseline achievement and smaller for low-performing students who are exempt from retention. These findings imply that (1) the benefits of early grade retention policies are larger than have been estimated using traditional fuzzy regression discontinuity designs but that (2) retaining additional students would have a limited effect on student outcomes.
Other 'Metrics(ish) Papers:
I also have written a handful of papers that are mainly focused on an empirical question, but which use non-standard methods to do so.
Measuring and Summarizing the Multiple Dimensions of Teacher Effectiveness (with Christine Mulhern) [R&R at AEJ:Policy]
Abstract: There is an emerging consensus that teachers impact multiple student outcomes, but it remains unclear how to measure and summarize the multiple dimensions of teacher effectiveness into simple metrics for research or personnel decisions. We present a multidimensional empirical Bayes framework and illustrate how to use noisy estimates of teacher effectiveness to assess the dimensionality and predictive power of teachers’ true effects. We find that it is possible to efficiently summarize many dimensions of effectiveness and most summary measures lead to similar teacher rankings; however, focusing on any one specific measure alone misses important dimensions of teacher quality.
Leading Indicators of Long-Term Success in Community Schools: Evidence from New York City (with Lauren Covelli and John Engberg) [R&R at Journal of Research on Educational Effectiveness]
Abstract: Community schools are an increasingly popular strategy used to improve the performance of students whose learning may be disrupted by non-academic challenges related to poverty. Community schools partner with community based organizations (CBOs) to provide integrated supports such as health and social services, family education, and extended learning opportunities. With over 300 community schools, the New York City Community Schools Initiative (NYC-CS) is the largest of these programs in the country. Using a novel method that combines multiple rating regression discontinuity design (MRRDD) with machine learning (ML) techniques, we estimate the causal effect of NYC-CS on elementary and middle school student attendance and academic achievement. We find an immediate reduction in chronic absenteeism of 5.6 percentage points, which persists over the following three years. We also find large improvements in math and ELA test scores – an increase of 0.26 and 0.16 standard deviations by the third year after implementation – although these effects took longer to manifest than the effects on attendance. Our findings suggest that improved attendance is a leading indicator of success of this model and may be followed by longer-run improvements in academic achievement, which has important implications for how community school programs should be evaluated.
Dual Methods for Dual Enrollment: Combining approaches to estimate the impact of taking college courses in high school on educational attainment (with Christine Mulhern, Fatih Unlu, Brian Phillips, and Julie Edmunds) [R&R at Education Finance and Policy] (Draft available upon request)
Abstract: Dual enrollment programs are an increasingly popular way for students to earn college credits in high school. We study the impacts of North Carolina’s dual enrollment program using a novel empirical approach that combines a regression discontinuity design (RDD) with a propensity score weighted (PSW) model to generate hybrid estimates that are more precise than the RDD estimates and less biased than the PSW estimates. The hybrid estimates indicate that, on average, dual enrollment participants take 12 more college-level credits and pass approximately 11 more credits than their peers. Overall, dual enrollment participation increases students’ ACT scores, college attendance, and persistence in college, with larger effects for lower achieving students. The effects on college enrollment are largest at the two-year colleges, perhaps because most dual enrollment credits are earned at two-year colleges.
Improving Average Treatment Effect Estimates in Small Scale Randomized Controlled Trials (R & Stata packages available upon request)
Abstract: Researchers often include covariates when they analyze the results of randomized controlled trials (RCTs), valuing the increased precision of the estimates over the potential of inducing small-sample bias when doing so. In this paper, we develop a sufficient condition which ensures that the inclusion of covariates does not cause small-sample bias in the effect estimates. Using this result as a building block, we develop a novel approach that uses machine learning techniques to reduce the variance of the average treatment effect estimates while guaranteeing that the effect estimates remain unbiased. The framework also highlights how researchers can use data from outside the study sample to improve the precision of the treatment effect estimate by using the auxiliary data to better model the relationship between the covariates and the outcomes. We conclude with a simulation, which highlights the value of using the proposed approach.