Cartoon vs. Lecture Study
Steven Seitz
June 1, 2022

The goal of this study is to test the hypothesis that short-form (~5 minute) animated videos can replace lectures as an effective form of learning for a university-level computer graphics class. This page describes my overall goals and motivation in more detail, along with the top take-aways from the this study -- read it first, if you haven't already.

I created a set of 16 of these videos which I will henceforth refer to as cartoons. These cartoons were roughly based on the first half of the lecture slides from previous offerings of CSE 457 -- the undergraduate computer graphics class taught at the University of Washington School of Computer Science. These slides were originally created by Brian Curless and have been used for the last two decades at UW by multiple instructors (myself, Brian, Zoran Popovic, and Adriana Schulz). While I followed the curriculum for the most part, my style of presentation (and in some cases, content) is significantly different, to include more visuals and animations. Each cartoon was designed to cover the same content as a 50 minute lecture.

I then designed and implemented a user study during the Spring 2022 offering of CSE 457, where students were presented with either cartoons or lectures, and I measured both relative preference, and impact on student performance on assigned materials.

Study Mechanics

Enrollment and participants: The study was submitted and approved by the University of Washington Human Subjects Division through their formal IRB process. The population of study participants consisted of students enrolled in the Spring 2022 section of CSE 457. On the first day of class, students were given a choice to enroll or opt-out of the study via an online form. 67 students enrolled (76%), and 16 opted out (24%). The students were presented with relatively little information about the cartoon format, only that they would be 5 minute short-form animated videos that replaced lectures, so had to make the opt-in/out decision without actually seeing a cartoon in advance. Students were also informed that study participants would need to fill in weekly surveys rating each lecture or cartoon. While we didn't ask why students opted out, possible reasons could include lecture preference (a possible source of bias), avoiding the extra work of surveys, privacy considerations, or general tendency not to participate in studies.

Groups: The 67 participants were divided into two, approximately equal groups, henceforth referred to as Group A and Group B. It was important that participants in different groups did not share materials. To allow for students who like to study together (friends, etc.), I asked all students to send me the list of other students they wished to study with, and adjusted the composition so that each study group was wholly contained in either Group A or Group B. The fact that Groups A and B were no longer random does not matter, as both groups were given both treatments (cartoons and lectures). Group A saw cartoons for the first two weeks, and lectures for the following two weeks. Group B saw lectures for the first two weeks, and cartoons for the following two weeks. When a group had cartoons, they did not come to class. Access to materials was carefully controlled; only participants in Group A were given access (via a Google Drive folder shared only with that group) to the first set of cartoons, and the second set of lecture slides. Similarly, Group B had their own shared drive with the first set of lecture slides and second set of cartoons. Students understood this arrangement and were instructed multiple times not to share materials with other students in different groups. I took attendance each day to verify the right people were in the room. Furthermore, every participant had to fill out a survey for each lecture/cartoon, specifying which format they watched, allowing me to catch cases where a violation occurred (I omitted these from the study results).

Cartoons and Lectures: The cartoons consisted of keynote slides (Apple's version of Powerpoint) and animations, recorded with a voice over. More details and the cartoons themselves can be found here. I used the same set slides for the lectures (converted to PowerPoint, presented with an iPad), adjusted for an interactive format. E.g., I might query the class to inform answers that I write onto the slides with the iPad pen annotations during the lecture, whereas the cartoon would present that same content directly. But otherwise, the content was nearly identical. My annotated lecture slides were shared in pdf format with the participants who had access to that lecture.

Surveys: I conducted two types of surveys -- 1) cartoon/lecture ratings, and 2) relative preference of cartoons vs. lectures. For the first category, I created a survey for each cartoon and each lecture that consisted of the following required questions:

"I watched (cartoon, lecture, neither, both)"

"Overall rating (0 = hated it, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 = as good as best lecture I've seen)"

"My level of enjoyment (1 = very low, 2, 3, 4, 5 = very high)"

"Effectiveness in helping me learn the material (1 = poor, 2, 3, 4, 5 = excellent)"

"Pacing (1 = poor, 2, 3, 4, 5 = excellent)"

"Efficient use of my time (1 = poor, 2, 3, 4, 5 = excellent)"

There was also an optional free-form text response "Comments (optional).

The relative preference survey can be found here.

Results

Timing: My informal experiments prior to the class indicated that each 5 minute cartoon would consume about 50 minutes of lecture time. This turned out to be fairly accurate -- I assigned 3 cartoons per week for each 160 minutes of lecture (two 80 minute class periods). I consistently used the full lecture period. Note that while most cartoons were 5-6 minutes, they ranged from 4.5 to 8.5 minutes. This particular class asked a lot of questions, and I tend to go more slowly than some professors do. So another instructor may find that it's closer to a 1 to 7 or 1 to 8 ratio for the length of cartoons to lectures, which for me was approximately 1 to 10. These numbers are naturally approximate, and may change from class to class.

Student performance: I evaluated the impact of which treatment each participant received (cartoon vs. lecture) on exam, written homework, and programming project scores. There was one (comprehensive) midterm, two written homeworks, and one programming project during the study period. For each problem (e.g., a question on the midterm), I computed the "cartoon" and "lecture" performance on that problem, by averaging the scores separately of students who learned the relevant material by means of cartoons or lectures, respectively. I then computed a total cartoon (lecture) score for the midterm, and similarly for the homeworks and project. The results are as follows:

These differences are not statistically significant between the Cartoon and Lecture groups: t-test P = 0.07 for Midterm, much larger for HW and P2.

Relative preference:

The full (anonymized) set of survey responses can be found here. We present a number of charts below from these survey responses.

Ratings

Each of the cartoons and lectures were rated by the students who saw them. Here are the aggregated results:

Concluding Thoughts

This study demonstrates that, on average, the cartoon format is at least as effective as lectures for teaching the material in a university-level computer graphics class. Given the choice of coming into class to watch live lectures or watching cartoons online, students preferred the latter on average. There was no statistically significant difference between the two formats on class performance -- students did just as well on exams, homeworks, and projects with either treatment.

While the cartoons were preferred to lectures overall, many (~1/3) participants missed the interactive question and answer (Q&A) opportunities of a lecture format; a hybrid approach could yield the best of both worlds, and scored the best in terms of student preference. There are a number of ways cartoons could be supplemented by Q&A, e.g., through online discussion forums, office hours, and/or using lecture time for Q&A. As the class already had a discussion forum and office hours, I believe the last option -- using lecture time for Q&A could give the most significant benefit.

I only had time to produce cartoons for the first half of the class. So I had the students produce the cartoons for the second half. I had students choose teams of 4, and each gave a ranked list of topics they'd like to cover from the second half of the class curriculum. I then assigned topics to teams in a way that all topics were covered. While these student-produced cartoons were not part of the formal study, each one was rated using the same survey as for my cartoons. The quality of the student-produced cartoons was excellent on average -- 6 had a median overall rating of 9/10 and the remaining 13 rated 8/10. These six cartoons were rated higher than the best of my cartoons. One possible source of bias is that this rating contributed a small amount to the team's grade, so students may have been less inclined to give negative ratings. I plan to assign several of these student-produced cartoons along with my own in future offerings of this class, and they are included in my Graphics in 5 minutes cartoon youtube playlist.