Randomized Control Trial FAQ

What is a Randomized Controlled Trial?

Randomized Control Trial (RCT) designs involve comparison of data between treatment and control groups. In this project, treatment groups are sections (lecture, lab, recitation) where course directors and affiliated faculty have enhanced the course through redesign. An RCT randomizes who receives the intervention and who doesn’t, thereby reducing bias, and allowing for an valid estimation of treatment impact.

Why do we have to do the random assignment? Aren’t matched comparisons good enough?

While the What Works Clearinghouse recognizes quasi-experimental designs, and studies using them may meet WWC standards with reservations, this project proposed using a cluster-level RCT to strive for meeting WWC standards without reservations (the highest rating). Random assignment helps protect against the potentially confounding influence of factors that cannot be measured and controlled in the analysis.

What’s a “cluster-level” RCT?

An RCT is considered to be cluster-level when it randomly assigns groups to treatment and control conditions, rather than individuals (i.e., course sections, rather than students).

A cluster-level randomized controlled trial is being used to evaluate the US Dept. of Education-funded STEM Bridges Across Eastern Queens project, in order to meet What Works Clearinghouse standards without reservations (What Works Clearinghouse™Standards Handbook (Version 4.0), 2017).

Why are we not offering the improved sections to all students? Aren’t we denying some students learning opportunities?

The purpose of this evaluative study is to test with scientific rigor whether the treatment sections have better learning outcomes than the control versions. If this is proven early in the project, the case can be made for expanding the treatment to all students. This is why we need to perform the controlled experiment.

I already assigned the treatment instructors to sections for this term, can the data be used in the evaluative study?

No. Only data from sections randomly assigned to treatment and control may be included. The data can be included if sections are randomly assigned before the term starts.

Only half of my instructors are willing to offer the new versions of sections. How does this work with random assignment?

Ideally, only instructors willing to teach the treatment or control (whichever they are assigned) would be offering sections each term. However, as long as sections are randomly assigned, it is fine to match instructors who are willing to offer treatment versions, based on their scheduling availability, to the sections assigned to the treatment condition.

What happens if an instructor assigned to a treatment section actually offers the “business as usual” version of the course?

This is considered “non-compliance” and is detrimental to the overall study. Whenever possible, instructors assigned to offer treatment versions of sections should be supported in doing so.

What happens if an instructor assigned to a control section actually offers the treatment version of the course?

This is “non-compliance” and detrimental to the overall study. Until the treatments are proven effective, the control condition provides necessary data with which to measure the impact of the treatment.

What This Means for Course Directors:

If you are responsible for one of the STEM “landing courses” included in the project, as you prepare to test the enhancements made to the course (whether in the lecture, lab, and/or recitation), you will work with the project’s external evaluator who will randomly assign eligible sections to receive the treatment or control (i.e., business as usual) version of the section. Only randomly assigned sections may be included in the analysis, which is a required part of the funding agreement.

Where can I find more information about the evaluation design and requirements?

Feel free to contact Dr. Kate Winter at kate@katewinterevaluation.com or +1 703-574-3746