Research

Selected Publications:

Does Helping John Help Sue? Evidence of Spillovers in Education, American Economic Review, March 2019, 109(3): 1080-1115

Does the impact of teachers extend beyond the students in their classroom? Using the natural transitions of students from multiple elementary schools into a single middle school, this paper provides a new method for isolating and quantifying peer spillover effects of teaching and shows that ignoring these spillovers underestimates a teacher's value by at least 30%. Because the spillovers also affect teacher value-added estimates, I develop a method of moments estimator of teacher value-added and show that accounting for the spillovers does not have a large impact on the ranking of teachers in New York City. I conclude by showing that the spillovers occur within groups of students who share the same race and gender, which suggests that social networks play a critical role in disseminating the effect.

Optimal Allocation of Spots in the Presence of Peer Effects: Evidence a Job Training Program (with Matthew D. Baird and John Engberg), Journal of Labor Economics, April 2023, 41(2): 479-509

We consider the case in which the number of seats in a program is limited, such as a job training program or a supplemental tutoring program, and explore the implications that peer effects have for which individuals should be assigned to the limited seats. In the frequently-studied case in which all applicants are assigned to a group, the average outcome is not changed by shuffling the group assignments if the peer effect is linear in the average composition of peers. However, when there are fewer seats than applicants, the presence of linear-in-means peer effects can dramatically influence the optimal choice of who gets to participate. We illustrate how peer effects impact optimal seat assignment, first under a general social planner utility function and then from both an efficiency and an equity perspective. We next use data from a recent job training RCT to provide evidence of large peer effects in the context of job training for disadvantaged adults. Finally, we combine the two results to show that the program's effectiveness varies greatly depending on whether the assignment choices account for or ignore peer effects.

The effect of natural disasters on human capital in the United States (with R. Jisung Park and Lucas Husted), Nature Human Behavior, June 2023

Although natural disasters are commonplace, they leave in their wake an enormous amount of damage. The physical damage they cause is immediately apparent, but less obvious is the potential magnitude of disruptions to learning and resulting damage to human capital. Using the universe of Presidential Disaster Declarations in the United States, we show that natural disasters impact a region's human capital both via reductions in learning for students who remain in school as well as a reduction in the years of schooling completed. These effects appear to be scarring and persistent. Quantifying these losses using the implied reduction of lifetime earnings suggests that natural disasters reduce a region’s human capital by a similar magnitude as the assessed property damage.

Illustrating the Promise of Community Schools: An Assessment of the Impact of the New York City Community Schools Initiative (with William R. Johnston, John Engberg, Lisa Sontag-Padilla, and Lea Xenakis), RAND Report 3245, 2020

With the launch of the New York City Community Schools Initiative (NYC-CS) in 2014, the New York City Department of Education (NYCDOE) has increased its focus on the implementation of a holistic strategy of education reform to address the social consequences of poverty as a means to improving student outcomes. NYC-CS is a strategy to organize resources in schools and share leadership among stakeholders so that academics, health and wellness, youth development, and family engagement are integrated into the fabric of each school. New York City is implementing this strategy at a scale unmatched nationally. In this study, the authors assessed the impact of the NYC-CS through the 2017–2018 school year. The authors assessed the effects along seven outcome domains and explored the extent to which there is heterogeneity in programmatic impact based on student- and school-level characteristics. The authors leveraged innovative quasi-experimental methodology to determine whether students in the community schools are performing better than they would be had their schools not been designated as Community Schools. The findings of this report will contribute to the emerging evidence base on the efficacy of the community school strategy and will be useful for other school district– and state-level policymakers interested in developing or refining similar interventions that support students' and communities' academic, social, and emotional well-being.

Working Papers:

Screening with Multitasking: Theory and Empirical Evidence from Teacher Tenure Reform (with Michael Dinerstein) [R&R at Journal of Political Economy]

What happens when employers screen their employees but only observe a subset of output? We specify a model with heterogeneous employees and show that their response to the screening affects output in both the probationary period and the post-probationary period. The post-probationary impact is due to their heterogeneous responses affecting which individuals are retained and hence the screening efficiency. We show that the impact of the endogenous response on both the unobserved outcome and screening efficiency depends on whether increased effort on one task increases or decreases the marginal cost of effort on the other task. If the response decreases unobserved output in the probationary period then it increases the screening efficiency, and vice versa. We then assess these predictions empirically by studying a change to teacher tenure policy in New York City, which increased the role that a single measure—test score value-added—played in tenure decisions. We show that in response to the policy teachers increased test score value-added and decreased output that did not enter the tenure decision. The increase in test score value-added was largest for the teachers with more ability to improve students' untargeted outcomes, increasing their likelihood of getting tenure. We estimate that the endogenous response to the policy announcement reduced the screening efficiency gap—defined as the reduction of screening efficiency stemming from the partial observability of output—by 28%, effectively shifting some of the cost of partial observability from the post-tenure period to the pre-tenure period.

From LATE to ATE: A Bayesian Approach [R&R at Journal of Econometrics]

We develop a Bayesian model that produces a posterior distribution of the marginal treatment effect (MTE) function. The method can be used even when the MTEs are not identified -- as is the case in RCTs with imperfect compliance -- thereby allowing researchers to generate plausible ranges for important and potentially policy-relevant (but unidentified) quantities of interest. We then use the model to propose a natural decomposition of the posterior variance into "statistical uncertainty," i.e., uncertainty that is due to the imprecise estimation of the observed moments, and "extrapolation uncertainty," i.e., uncertainty that is due to the non-identifiability of the parameter of interest. We conclude by showing that under our preferred priors, even in an experiment as large as the Oregon Health Insurance Experiment, the main source of uncertainty in the ATE comes from uncertainty in the true values of the observed moments.

A Global Regression Discontinuity Design: Theory and Application to Grade Retention Policies (with Umut Ozek)

We propose a novel estimator for use in a fuzzy regression discontinuity setting. The estimator can be thought of as extrapolating the traditional fuzzy regression discontinuity estimate or as an observational study that adjusts for endogenous selection into treatment using information at the discontinuity. We show that it can be motivated as being the least complex model consistent with the data or as an estimator that is preferable to both a traditional regression discontinuity design and an observational study. We further show theoretically that no other estimators consistently generate better estimates than our proposed estimator. We then use this approach to show that the benefits of early grade retention policies are larger for students with lower baseline achievement and smaller for low-performing students who are exempt from retention. These findings imply that (1) the benefits of early grade retention policies are larger than have been estimated using traditional fuzzy regression discontinuity designs and (2) retaining additional students would have a limited effect on student outcomes.

Measuring and Summarizing the Multiple Dimensions of Teacher Effectiveness (with Christine Mulhern) [R&R at AEJ:Policy]

There is an emerging consensus that teachers impact multiple student outcomes, but it remains unclear how to measure and summarize the multiple dimensions of teacher effectiveness into simple metrics for research or personnel decisions. We present a multidimensional empirical Bayes framework and illustrate how to use noisy estimates of teacher effectiveness to assess the dimensionality and predictive power of teachers’ true effects. We find that it is possible to efficiently summarize many dimensions of effectiveness and most summary measures lead to similar teacher rankings; however, focusing on any one specific measure alone misses important dimensions of teacher quality.

Leading Indicators of Long-Term Success in Community Schools: Evidence from New York City (with Lauren Covelli and John Engberg) [R&R at Journal of Research on Educational Effectiveness]

Community schools are an increasingly popular strategy used to improve the performance of students whose learning may be disrupted by non-academic challenges related to poverty. Community schools partner with community based organizations (CBOs) to provide integrated supports such as health and social services, family education, and extended learning opportunities. With over 300 community schools, the New York City Community Schools Initiative (NYC-CS) is the largest of these programs in the country. Using a novel method that combines multiple rating regression discontinuity design (MRRDD) with machine learning (ML) techniques, we estimate the causal effect of NYC-CS on elementary and middle school student attendance and academic achievement. We find an immediate reduction in chronic absenteeism of 5.6 percentage points, which persists over the following three years. We also find large improvements in math and ELA test scores – an increase of 0.26 and 0.16 standard deviations by the third year after implementation – although these effects took longer to manifest than the effects on attendance. Our findings suggest that improved attendance is a leading indicator of success of this model and may be followed by longer-run improvements in academic achievement, which has important implications for how community school programs should be evaluated.

Dual Methods for Dual Enrollment: Combining approaches to estimate the impact of taking college courses in high school on educational attainment (with Christine Mulhern, Fatih Unlu, Brian Phillips, and Julie Edmunds) [R&R at Education Finance and Policy]

Dual enrollment programs are an increasingly popular way for students to earn college credits in high school. We study the impacts of North Carolina’s dual enrollment program using a novel empirical approach that combines a regression discontinuity design (RDD) with a propensity score weighted (PSW) model to generate hybrid estimates that are more precise than the RDD estimates and less biased than the PSW estimates. The hybrid estimates indicate that, on average, dual enrollment participants take 12 more college-level credits and pass approximately 11 more credits than their peers. Overall, dual enrollment participation increases students’ ACT scores, college attendance, and persistence in college, with larger effects for lower achieving students. The effects on college enrollment are largest at the two-year colleges, perhaps because most dual enrollment credits are earned at two-year colleges.

Resting Working Papers:

Improving Average Treatment Effect Estimates in Small Scale Randomized Controlled Trials

Researchers often include covariates when they analyze the results of randomized controlled trials (RCTs), valuing the increased precision of the estimates over the potential of inducing small-sample bias when doing so. In this paper, we develop a sufficient condition which ensures that the inclusion of covariates does not cause small-sample bias in the effect estimates. Using this result as a building block, we develop a novel approach that uses machine learning techniques to reduce the variance of the average treatment effect estimates while guaranteeing that the effect estimates remain unbiased. The framework also highlights how researchers can use data from outside the study sample to improve the precision of the treatment effect estimate by using the auxiliary data to better model the relationship between the covariates and the outcomes. We conclude with a simulation, which highlights the value of using the proposed approach.

Am I Good Enough? Understanding How Teachers' Assessment of Their Own Practice Depends on the Context (with Kata Mihaly)

We analyze survey responses from teachers who were asked to rate themselves on a formal observation rubric used in a high-stakes evaluation system. Our analysis reveals three findings. First, the data show that teachers generally agree with their principal's assessment of their ability. Second, we find that teachers' self-ratings are positively correlated with their value-added scores; however, this correlation disappears after conditioning on the principals' ratings. Finally, we show that teachers in high-poverty, low-achieving schools tend to rate themselves worse than teachers in schools with low-poverty, high-achieving student bodies, and this finding holds true even when we compare teachers with the same principal ratings or value-added scores. This last finding suggests that teachers internalize factors outside of their control when evaluating their own performance.