Executive functions (EF) are a set of psychological skills defined as goal-directed cognitive processes. Traditional EF tests are reliable, but they are not able to detect EF in real-time. Learning games have the potential to obtain a real-time, unobtrusive measurement of EF. In this study, we analyzed log data collected from a game designed to train the EF sub-skill of shifting. We engineered theory-based features from log data. Using these features, we built prediction models with students’ accuracy and reaction time during play to predict their standard measure of the EF shifting skill during the post-test and delayed post-test as well as to predict learning gains. Our model that predicts the post score has a correlation of 0.322 and that for the delayed post score is 0.303. The findings suggest that theory-based feature engineering and varying levels of granularity are two promising directions for cognitive skills prediction under the goal of game-based assessment. Also, accuracy, reaction time, and player progression are important features.