Most of this section is about analyzing your data. There is however some discussion of the problem sets since how you design your problem set directly affects the data.
This page is meant to help you create a study, but also to understand how to interpret your data. In this video, Professor Heffernan takes a moment to explain some important concepts.
Professor Heffernan recorded a short video depicting how to create a simple randomized controlled trail, with a pretest experiment, and posttest.
Professor Heffernan recorded a demo from the student's perspective. This is what it would look like for a student participating in randomized controlled experiment designed above. This video also shows what it looks like from the teacher's perspective.
Next Professor Heffernan has recorded a short video explaining an experiment that was performed within a Skill Builder. This website highlights the study. This was a study designed by a researcher at Harvard.
In this video, Professor Heffernan shows the "ASSISTments Certified Problem Sets" folder, where teachers will be able to find your study. The description and Common Core standard that serve as the title will be followed by "EX" to denote an experimental problem set. In this video, Professor Heffernan also discusses a little bit about the ARRS system and how researchers can benefit when teachers have this preference selected.
In this video, Professor Heffernan examines how researchers are notified when they have data. Through an automated system, you will receive an email with two types of data files and a brief summary of how many students have started and/or completed your problem set. This video explains the difference between the two files - depending on your analysis you may prefer one over the other. The first type of file is the problem log file; each row in the file represents one problem for one student. Thus, you will see that some students have more rows than others, as they required more problems to complete their assignment. The second type of file is the student-level file; in this file, each row represents one assignment for one student. It seems likely that most researchers will want to start their analyses with the student-level data, but the problem log file could be used for something like a logistic regression approach. In either case, you will need to designate and label the condition that each student received. Professor Heffernan shows one approach to this task in the video below.
In this next short video, Professor Heffernan opens the student-level file and quickly checks to see if there is a result using a t-test. More complicated studies (those with more than two groups) will likely require ANOVA or MANOVA analyses, as well as analyses of possible covariates. However, this example shows a simple analysis that offers a quick check.
The video below provides the nitty gritty details on how this analysis was performed (in case you are less familiar with Excel). If you are confused about how to figure out what condition each student saw, the video below features Professor Heffernan showing how he went about the process in his dataset. If you would like to practice alongside the video, here is the student-level data.
If you would like to try this while watching the video, here is the Excel file to start from.
The video below features the simple maneuvers that Professor Heffernan used to manipulate the data. It is likely that most researchers will import their data into a program like SPSS rather than using Excel to run calculations - feel free to skip this video if you are comfortable with data manipulation.
Here Professor Heffernan shares what your data will look like and how the data was accessed from the ASSISTments database. Here is the ASSISTments database ER diagram.
In this video, Professor Heffernan opens the student-level file, and finds that 1,000 students participated in the study. He also explains a wide variety of the outcome measures that are offered int he file, including correctness, the student's first action, their response time, etc.
What is a "main problem"? What does it mean to have multiple main problems? What is a scaffolding question? What is the difference between an "ASSISTment" and a "problem?" Professor Heffernan answers these questions and more in the video below.
As a researcher, you may want to build Skill Builder problem sets without having to make 100's of questions. Skill Builders usually consist of about 100 questions generated from only a handful of templates. What is a template and how can it drastically reduce your work in design and analysis? For instance, you might want to use difficulty as a metric in your analyses, perhaps by assuming that the difficulty of an item is linked to its template. This would mean that all problems built from the same template would have the same difficulty. In this video, Professor Heffernan shows how a Skill Builder is really made, and how you can harness the process in the design and analysis of your study.
In the video below, Professor Heffernan reviews some of the gory details. Some documentation can also be found here.
The "tutorial strategies" column in your data may reflect something a little different than the hints, scaffolds, and buggy messages or common wrong answer messages that students receive while working on a problem set. This column is not likely helpful to most researchers, but Professor Heffernan provides an example in the video below.
If you a update during the week, you can use this link to get you data. For instance, if you want to play student, and type in your phone number, you can see yourself in the data if you wait overnight and then go ask for a data dump from this link (ask Heffernan for the password ).