Educational A/B Testing at Scale
A Virtual Workshop at Learning @ Scale 2020
August 12, 2020, 12:30pm-3:30pm Eastern Time on Zoom (see link below)
August 12, 2020, 12:30pm-3:30pm Eastern Time on Zoom (see link below)
Proceedings of the workshop are available here.
August 12, 2020 at 12:30pm-3:30pm Eastern Time
12:30 - 12:35PM: Welcome
12:35 - 12:50PM: Paper 1
UpGrade: An Open Source Tool to Support A/B Testing in Educational Software [PDF]
Steve Ritter, April Murphy, Stephen Fancsali, Vivek Fitkariwala, Nirmal Patel and Derek Lomas
12:50 - 1:05PM: Paper 2
Managing group random assignment in UpGrade [PDF]
Steve Ritter, April Murphy and Stephen Fancsali
1:05 - 1:20PM: Paper 3
Optimizing an Educational Game Using UpGrade: Challenges and Opportunities [PDF]
Nirmal Patel, Dhrushit Raval, Vivek Fitkariwala and Derek Lomas
1:20 - 1:35PM: BREAK
1:35 - 1:50PM: Paper 4
Should We Add a Progress Meter? How A/B Testing Can Support Rapid Cycles of Data-Informed Design [PDF]
Derek Lomas, Dhrushit Raval, Steve Ritter, Vera van Dijk, Dion Dumoulin, Pablo Geraedts, Veerle Maljers, Isabel Mens, Vivek Fitkariwala, Nirmal Patel
1:50 - 2:05PM: Paper 5
A Preliminary Taxonomy of A/B: Education Experiments with Different Inferences and Scopes [PDF]
Benjamin Motz, Paulo Carvalho and Emily Fyfe
2:05 - 2:20PM: Paper 6
E-TRIALS: A web-based application for educational experimentation at scale [PDF]
Korinn Ostrow and Ryan Emberling
2:20 - 2:30PM: BREAK
2:30 - 2:50PM: Breakout Groups
2:50 - 3:20PM: Readouts from Breakout Groups
3:20 - 3:30PM: Discussion and Wrap-up
There is no simple path that will take us immediately from the contemporary amateurism of the college to the professional design of learning environments and learning experiences. The most important step is to find a place on campus for a team of individuals who are professionals in the design of learning environments — learning engineers, if you will. - Herbert Simon
The emerging discipline of Learning Engineering is focused on putting into place tools and processes that use the science of learning as a basis for improving educational outcomes. An important part of Learning Engineering focuses on improving the effectiveness of educational software. In many software domains, A/B testing has become a prominent technique to achieve the software’s goals. Many large companies (Amazon, Google, Facebook, etc.) run thousands of AB tests and present at the Annual Conference on Digital Experimentation (CODE), but that venue is too broad to address AB testing issues specific to EdTech platforms. We see a need to address issues with running large-scale A/B tests within the educational context, where the use of A/B testing lags other industries. This workshop will explore ways in which A/B testing in educational contexts differs from other domains and proposals to overcome current challenges so that this approach can become a more useful tool in the learning engineer’s toolbox. Issues to be addressed are expected to include:
managing unit of assignment issues
measurement, including both short and long-term outcomes
practical considerations related to experimenting in school settings, MOOCs, & other contexts
ethical and privacy issues
relating experimental results to learning-science principles
understanding use cases (core, supplemental, in-school, out-of-school, etc.)
accounting for aptitude-treatment interactions
A/B testing within adaptive software
adaptive experimentation
attrition and dropout
stopping criteria
user experience issues
educator involvement and public perceptions of experimentation
balancing practical improvements with generalizable science
We welcome participation from researchers and practitioners who have either practical or theoretical experience related to running A/B tests and/or randomized trials. This may include researchers with backgrounds in learning science, computer science, economics and/or statistics.
Steve Ritter, Carnegie Learning
Neil Heffernan, Worcester Polytechnic Institute
Joseph Jay Williams, University of Toronto
Burr Settles, Duolingo
Phillip Grimaldi, Khan Academy
Derek Lomas, Delft University of Technology
Bob Bodily, Lumen Learning
Ryan Emberling, ASSISTments Foundation
Stephen Fancsali, Carnegie Learning
Phillip Grimaldi, Khan Academy
Derek Lomas, Delft University of Technology
April Murphy, Carnegie Learning
Korinn Ostrow, Worcester Polytechnic Institute
Nirmal Patel, PlayPower Labs
Steve Ritter, Carnegie Learning