Optimal game design for behavioral research and educational assessment
By Anna Raffery (Carleton College)
Abstract: There is increasing interest in using games for behavioral experiments and educational assessment. Games offer the opportunity for players to make meaningful choices and are naturally engaging. However, while games have the potential to provide richer insights into human cognition, designing games that actually achieve this potential is challenging. In the case of behavioral experiments, participants' choices in the game must be relevant to the question being investigated; in the case of educational assessments, varying levels of understanding must translate to different actions in the game. Even given a basic idea for a game, there are many ways that the game might be structured, such as setting the point values or deciding what items to make available in which locations. We propose applying optimal experiment design methods to choosing a game design, helping to avoid many costly cycles of trial and error.
Our framework combines optimal experiment design with inverse planning, which can make inferences about cognitive processes based on data from games and other freeform environments. Inverse planning requires a generative model of how players act: we use Markov decision processes to model actions, and then make inferences about the parameters of a cognitive model from these actions. Using simulations, we automatically refine the design of the game, searching for designs that maximize expected information gain about the parameters of the cognitive model. We explore the use of this technique in practice through several concept-learning games. Our results show that this method can predict which games will result in better estimates of the parameters of interest. The best games require only half as many players to attain the same level of precision. We are now applying the same technique to improve assessment in an online algebra tutor.
(This is joint work with Thomas Griffiths at the University of California, Berkeley.)