Why do we need more RCEs?

This is an opinion article from Neil Heffernan.

I, Neil Heffernan, PI for the ASSISTments TestBed have some strong feeling on why we need more randomized controlled trials (RCEs) in education.

The US Department of Education advocated that K12 schools should apply basic scientific findings from cognitive science to improve educational practices (Pashler et al., 2007). While this seems like a simple solution, there are two problems. First, a great deal of cognitive science research is performed with undergraduates in psychology courses where they must participate in a “subject pool” that is different in many ways from authentic educational settings. Second, even with access to K12 classrooms, the typical research paradigms are inefficient, absorbing student time with one-off pre- and posttests while simultaneously depleting experimental subjects on low-power designs using teacher-level randomization.

Even some of the best scientists working in schools are only able to generate a few experiments per year. For example, handing out different worksheets to different students to assign them to condition is clumsy and maintaining students in the same condition spread over days is very hard. Finally, if they want to implement robust measures of student knowledge like reassessment a weeks later, they would have to orchestrate more classrooms visits. Additionally, these researchers, who make the laudable effort of engaging with these challenges, must spend time developing and maintaining relationships with teachers and schools in addition to creating the experiments.

While efficiency and getting into schools are the hurdles psychology researchers have to overcome, there is a different hurdle for education researchers: it is very expensive to run an RCE at the teacher level. According to the National Research Council report (2002) on education research, “[RCEs] are not frequently conducted in education research” (page 112). Mosteller and Barouch’s book has several chapters that address why there is a scarcity of RCEs (2002, page 6). One reason they suggest is that education researchers care about issues like whole-groups instruction and classroom management that call for the teacher to be the unit of analysis. In order to have adequate statistical power in this type of RCE, many teachers are required and this can be very costly. For instance, the US Department of Education funds Efficacy Trials of promising interventions as well as the even more demanding Effectiveness Trials. The typical efficacy trial costs $3 million and involves 50 schools, while the effectiveness trials cost a median of $6 million. In the math and science realm, they have funded 22 Efficacy trials and five Effectiveness trials. Have these studies been effective? The US Department of Education has set up the What Works Clearinghouse (WWC) to apply rigorous standards to these trials to determine effectiveness. Unfortunately, of the 43 math and science interventions that met the WWC standards for review, only three resulted in “positive effects” and none of the 22 efficacy trials had a “positive effect.” Creating interventions that work is difficult and we think by using more student level randomization we can more quickly and efficiently learn what works in small steps.