A tutorial introduction to optimal experimental design
By Jay Myung and Mark Pitt (Ohio State University)
Abstract: Experimentation is fundamental to the advancement of science, whether one is interested in understanding the cognitive mechanisms governing attentional control in a task-switching experiment or testing the efficacy of a novel drug for treatment of obsessive-compulsive disorders in a clinical trial experiment. The use of adaptive experimentation can potentially improve the quality of inferences by increasing the informativeness of data and the efficiency with which data are collected. Addressing the challenge, scientists in statistics and machine learning have introduced various methods of optimizing experimental design (OED) in which the experimental design is adapted from trial to trial on the fly while the experiment is being conducted. Specifically, in OED, an experiment is run as a sequence of stages, or mini-experiments, in which the values of design variables (e.g., stimulus properties, task parameters, testing schedule) for the next stage are chosen based on the information (e.g., responses) gathered at earlier stages, so as to be maximally informative about the question of interest (i.e., the goal of the experiment). OED has become increasing popular in recent years, largely due to the advent of fast computing, which has made it possible to solve more complex optimization problems, and as such is starting to reach everyday experimental scientists. This tutorial will give an overview of the mathematical and computational foundations of OED, with a particular focus on model discrimination and parameter estimation in a Bayesian inference framework. Example applications in the areas of retention memory and risky choice will also be discussed.