Lunch and Learn: Analytics

Post date: Oct 14, 2020 1:36:15 AM

This past summer, I taught Physics for the first time in Canvas. Typically, students struggle with significant figures, especially the rules for adding and subtracting with significant figures because they are weird. This summer was no different, even though I had an amazing group of kiddos. This fall was similar. Below are the analytics for the adding and subtracting significant figures questions for my Fall 2020 class (I've blurred out the actual numbers on the questions to protect quiz security):

So, while I was learning Studio this summer for Lunch and Learn, I created a Studio quiz to give my students practice with adding and subtracting significant figures. I found a Khan Academy video going over these concepts and created a short little quiz over these concepts. It wasn't graded and I didn't tell the students about it. I just put it in the module to see what happened.

I put the "remediation" Studio Quiz BEFORE the graded quiz to try and catch those misconceptions BEFORE the students were quizzed over the concepts. Four students took the Studio quiz:

I've color-coded their results to protect their privacy and also so you can track their scores on the Studio quiz and then on the actual quiz (it's pretty interesting!).

The overall results of the graded quiz are below:

WOW, right? ALL of their scores were ABOVE the mean! In addition, even the kiddos that scored poorly on the Studio quiz scored well on the Graded quiz. I found this so interesting that I did a little more digging! Red Student got 3 of the 4 Addition/Subtraction questions correct! Purple Student got 3 of the 4 Addition/Subtraction questions correct too! So did Green Student. Blue Student, the one that earned a 20% on the Studio Quiz, got 4 of the 4 questions correct on the graded quiz!

This tells me a couple things:

1. The Studio Quiz was worth my time to make - the students that interacted with the quiz earned above average scores on the quiz, regardless of their score.

2. Those that scored low on the Studio Quiz went into the real quiz more prepared than those that didn't even attempt the Studio Quiz - whether the Studio Quiz reminded them that they needed to go back and review that material or what, anecdotally, it worked! It improved their grades on the actual graded quiz, which was the goal of the Studio Quiz.

3. The analytics tools in Canvas are useful - If you have the time, they can provide you with some very useful information.

I didn't look at their test scores because they'd already seen the "re-teaching" item before the first quiz.

Typically, I tend to look at the other Analytics in Canvas - access reports, average scores on assignments, etc. I've taught this course several times before, and even longer in the face-to-face classroom, so I have a pretty good idea of where students tend to struggle in the course. It was interesting to look at the quiz data and see that students are continuing to struggle with these concepts.

However, the really interesting part, and the part I'm really excited about is how well the students that interacted with the "re-teaching" quiz that I placed BEFORE the graded quiz, did on the graded quiz! This is really encouraging. Sometimes I can get bogged down by the fact that I add extra items to my course and students seem to not use them or interact with them. This tells me that I need to focus on those that DO interact with the items I make for my students because these items ARE helpful and for those students, at least anecdotally, they learn and improve from those things. It also makes me want to create Studio pre-quizzes for ALL THE THINGS. I think I'm going to spend more time looking at the analytics in the course to try and pinpoint these areas of weakness and slowly create more Studio Quizzes to try and "catch" misconceptions BEFORE they are quizzed on the material.