Testing the Testing Effect

The Testing Effect and Retrieval Practice

The testing effect if the finding that learners learn better when using  testing as a practice strategy rather than using rehearsal.  This idea is backed by dozens of scientific studies in the lab and also in the classroom, using variety of learning materials, across settings, subjects and ages (e.g. 1). It was translated into a strategy called Retrieval Practice, in which learners practice by effortfully recalling information , rather than re-reading it.

Research in Classrooms

The experiment described here was designed for an online graduate class in education studies and performed prior to starting a unit on research-informed effective practice. The first goal was to introduce the testing effect, by letting the students experience the long-term consequences of learning the same material using different strategies. A second goal was to introduce behavioral experiments in cognitive sciences, as basis for further discussion of benefits, limitation and possible implication for classroom practice.

The inspiration

The basic inspiration was Karpicke and Roediger's study of the testing effect (2) published in science in 2008. In this classical study participants learned words in Swahili and their English translations. Four different groups studied the 40 words in one session and and came back for a memory test one week later. The learning session included alternating  rounds of Study trials (Word + Translation) and Test trials (word presented, participants requested to type translation). The 4 groups had different amounts of Study vs. Test trials. The results showed that more Test trials at the study session led to much better memory performance after one week (see  more here). Another finding was that when participants were asked to predict their memory performance in 1 week, they were not able to predict accurately. Specifically those who restudied overestimated their performance and those who re-tested  underestimated their performance. This important issue was addressed in another study by the same authors from 2006, that explored the source of inaccurate performance estimation (3) and also in this presentation.

Results

What can be measured? In every point throughout the experiments where the participants were required to recall the translations (turquoise rectangles in the figure above), there is an opportunity to assess their memory performance. Hence, we can follow the average progress (for the Test words) from round to round in the learning  phase (left) and also compare the  average performance for Study vs. Test  after 1 week (right).

Conclusions - cognitive research in the classroom

Participating in this experiment allowed the students to experience the often surprising effects of testing  by themselves. They could (and were asked to) reflect on their own experience, evaluating the contributing factors. In addition, they saw the data averaged across the entire group, and how the patterns closely replicated scientific findings.  This connection between the individual and collective experience support the realization that some factors, like the testing effect, have an impact on learning that goes beyond the individual differences and incidental factors. I believe this realization is powerful for both students and educators, and may be key in their willingness to adopt insights from the cognitive sciences.

Another benefit is that learners can get a better first-hand idea of what a behavioral experiment means, and how it  is conducted. This helps to better appreciate the meaning of other behavioral experiments, to appreciate their value on one hand and to be critical on the other.

Last,  there are many ways to bring behavioral experiments into classrooms of all types. It can be just a short demo,  a compete experiment as in this case, and even a full project where the learners  take part as the experimenters, partnering in the entire process, from designing, to running, analyzing and presenting. The approach is chosen according to the teaching goals, and each has value in bridging the cognitive science with teaching and learning in classrooms. In my experience, with both middle schooners and higher-ed students,  it is a good way to induce motivation to change practices as it combines the intellectual challenge with the personal and often emotional experience. I will always remember a group of middle school students, avid re-readers and highlighters, skeptical at the beginning of a full semester testing-effect research project, asking me enthusiastically upon getting the results of their own experiment "if THEY know all this, why don't they tell US ?"

Wish to try?

If you are interested in trying the same experiment  in your own teaching context, I'm happy to share the files or discuss other possibilities (contact details), if you just want to try it for yourself or let someone else (who doesn't know the results in advance) try, click here. 

Another direction is to use platforms that provide ready-to-use cognitive experiments, Go Cognitive is a great one (it allows downloading the nonperformance data). 

References

1. Agarwal, P. K., & Roediger III, H. L. (2018). Lessons for learning: How cognitive psychology informs classroom practice. Phi Delta Kappan, 100(4), 8-12.‏  [PDF]

2. Karpicke, J. D., & Roediger,  III, H. L. (2008). The critical importance of retrieval for learning. science, 319(5865), 966-968.‏ [PDF]

3. Roediger III, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological science, 17(3), 249-255.‏  [PDF]