Using ITS Generated Data to Predict Standardized Test Scores
Kelly, K. & Arroyo, I. & Heffernan, N. (2013). Using ITS Generated Data to Predict Standardized Test Scores.In S. D'Mello, R. Calvo, & A. Olney (Eds.) Proceedings of the 6th International Conference on Educational Data Mining (EDM2013). Memphis, TN. pp. 322-323.
Abstract
Much of the literature surrounding intelligent tutoring systems has focused on the effectiveness of theses systems in producing learning games. This study suggests that the data generated by these systems can also be used to accurately predict end-of-year standardized state test scores. Data from 117 middle school students using ASSISTments throughout the 2010-2011 school year were used to predict MCAS (Massachusetts Comprehensive Assessment System) score. A traditional model including only past performance on the test yielded an R2 of 0.38 and an enhanced traditional model that added current class average improved predictions (R2=0.50). These models served as baseline measures for comparing an ITS model. Logistic regression models that include features such as hint percentage, average number of attempts and percent correct overall improved the R2 to 0.57. Using the predicated score to classify students as “advanced”, “proficient”, or “needs improvement” as determined by the Massachusetts Department of Elementary and Secondary Education improved accuracy by 7% over traditional models. The predictive power of the data is as effective with only a few months of use. Results indicate that classifying performance is more effective than predicting actual scores, but both are possible. Decision trees were also used to classify student performance but did not perform as well as the logistic regression, producing Kappa of 0.36. Furthermore, it is interesting to note that the factors that have been determined to be most predictive, such as percent hint usage and percent correct including multiple attempts, are only attainable through the implementation of intelligent tutoring systems in the classroom and for nightly homework. This lends support for the increased use of the systems.
This paper was accepted as a poster presentation. Below find both the short paper and the full version that was originally submitted.