Teaching-As-Research Internship

Each Delta Program intern works with a faculty partner to construct and implement a Teaching-As-Research (TAR) internship project that focuses on moving teaching theory into practice in the classroom. The goal of my TAR project was to improve student understanding of key magnetic resonance imaging (MRI) topics in Dr. Beth Meyerand's graduate-level Medical Physics course at the University of Wisconsin-Madison.

A summative report of my project can be found below and the instructional materials used during it can be found here.

The Impacts of Active Learning and Quantitative Literacy on Student Learning Outcomes in a Graduate-Level Medical Physics Course

Abstract

Science, technology, engineering, and mathematics (STEM) courses are routinely lecture-taught despite the fact that numerous studies provide evidence suggesting that incorporating active learning strategies into courses improves student learning to a greater degree than teaching via lecture alone. Such results have led to the implementation of several discussion-based activities in Medical Physics 578 "Diagnostic Imaging with Non-Ionizing Radiation", a graduate-level course at the University of Wisconsin-Madison. Following implementation of this discussion-based format, despite an overall improvement in learning, students appeared to struggle with material from one specific activity, "The Newspaper Exercise". This study sought to improve The Newspaper Exercise by emphasizing graphical literacy and active learning in a revised version of the module. Student learning outcomes using the revised Newspaper Exercise module, implemented in 2015, were compared to student learning outcomes using the original module from the previous four years (2011-2014). Significant improvement in student learning outcomes was observed using both subjective feedback from the instructor and objective data from exam scores. Revisions to the Newspaper Exercise were effective in improving student learning, as compared to previous years; however, student feedback on the concept of active learning was not very positive. Many students felt that additional lecturing would have been a more effective teaching tool. Further research needs to explore why students do not feel confident learning from alternate teaching methods that have been demonstrated to be both efficacious and superior to lecture-based formats.

Introduction

True scientific understanding requires a combination of rigorous mathematics, theory, context, application, and interpretation. Technical science courses have habitually relied on teaching theoretical and mathematical concepts through a rote lecture format that fails to touch on the intricacies necessary for full understanding of the subject. “Far too often, the instructor writes an equation on the blackboard that embodies the essence of a physical law without ever getting the student to see the subtleties that are clear to the instructor…. The result is that students often walk out of a classroom making a distinction between theory and practice when they should really be seeing the connections between theory and practice” (Stith 2001). Scientific advancement relies heavily on the fact that theory and practice are intertwined. Separating the two fosters a disconnect in understanding.

A problem that arises in quantitative fields as a result of the theory-practice disconnect is blind reference to mathematical equations without understanding what they truly mean. Equations are comprised of symbols that enable scientists to encapsulate both conceptual and procedural aspects of scientific phenomena (Tall et al. 2001). It is important to recognize both the inherent mathematical value of equations and also their versatility as tools for explaining and exploring science on a variety of levels.

One strategy being used to tackle this problem and improve overall student understanding is the idea of active learning, which focuses on students playing an active, rather than passive role, in their own education. Research suggests that teaching within an active learning framework produces small but significant gains in science literacy and science process skills as compared to more traditional teaching methods (Gormally et al. 2009). Another study "indicate[d] that active learning increases examination performance by just under half a SD and that lecturing increases failure rates by 55%...these increases in achievement hold across all of the STEM disciplines and occur in all class sizes, course types, and course levels" (Freeman et al 2014). This evidence suggests that active-based learning strategies can be successfully applied in a variety of STEM contexts. In addition to improving student understanding, active learning may also be used as a tool in heterogeneous classroom populations. The integration of active learning into the classroom may be one solution to closing the gender gap seen in STEM; "active learning confers disproportionate benefits for STEM students from disadvantaged backgrounds and for female students in male-dominated fields" (Freeman et al. 2014). This finding was reiterated in a paper by Lorenzo et al. (2006): "Our results show that teaching with certain interactive strategies not only yields significantly increased understanding for both males and females, but also reduces the gender gap. In the most interactively taught courses, the pre-instruction gender gap was gone by the end of the semester."

The first step in quantitative learning for many students revolves around their ability to internalize and comprehend, in their own way, how and why mathematical equations work. The next step is to investigate the relationship that mathematical symbols have with real-world phenomena, which allows students to not only relate scientific theories and laws to reality but to understand their application and use in a variety of contexts. To fully demonstrate understanding of a topic and use that knowledge for further scientific investigation, students must be able to effectively communicate concepts, ideas, methods, experiments, and results to their peers. “Quantitative literacy is the ability to identify, understand, and use quantitative arguments in everyday contexts. Quantitative literacy insists on understanding. This understanding must be flexible enough to enable its owner to apply quantitative ideas in new contexts as well as in familiar contexts.” (Hallett 2003). Quantitative literacy standards should be routinely embedded in lessons and assessments, and instructors should be transparent about their expectations for such literacy (Ewell 2001).

Starting in 2011, Professor Beth Meyerand (University of Wisconsin - Madison) began incorporating active learning techniques and discussion-based activity into teaching for her MRI course. One of her learning modules, the Newspaper Exercise, appeared to generate lively discussion surrounding four specific Medical Physics concepts: radio frequency (RF) pulses, T1, T2, and T2*. However, both test scores and informal evaluations from discussions with students following this unit suggested that students were not achieving the desired level of understanding.

Mastery in the field of Medical Physics, with its rigorous theoretical foundation and clinical applications, epitomizes the need for deep understanding at both a quantitative and qualitative level. It is imperative for students to grasp the theory of MRI, the practical application of this technology, the interplay between the two, and have the skills to communicate their knowledge. New instructional material was created for the Newspaper Exercise module that emphasized active learning and graphical literacy in an effort to promote a holistic approach to learning.

Teaching-As-Research Project Aim: Improve student learning in Medical Physics 578 using new instructional material created and assessed within an active learning framework.

i) Improve students' overall understanding of the 4 specific MRI topics addressed in the Newspaper Exercise: RF pulses, T1, T2, and T2*

ii) Encourage students' active involvement in their own education

iii) Increase quantitative literacy through active-learning exercises that generate conversation and collaboration

Materials and Methods

Medical Physics 578, Diagnostic Imaging with Non-Ionizing Radiation, is a graduate-level course at the University of Wisconsin that covers the physics associated with magnetic resonance imaging (MRI) and diagnostic ultrasound. It is a required class in the Medical Physics graduate program’s core curriculum that meets during two 75-minute sessions each week and is comprised of upper level undergraduates and early-stage graduate students. Most enrolled students are from either the Biomedical Engineering or Medical Physics Department. This small to mid-size course is split into two different content units, each taught by a different professor. The first unit encompasses ultrasound imaging, and the second unit covers MRI. The intervention discussed here was implemented during the MRI portion of the course.

The Newspaper Exercise (NE) is one specific module taught in Medical Physics 578. Modifications, highlighted in Figure 1, were made to the original version of the Newspaper Exercise. The original version was used during 2011, 2012, 2013, and 2014. Modifications made to both the module and the exercise strove to increase student learning and comprehension and improve quantitative literacy within an active learning framework. The revised Newspaper Exercise was presented in class during three consecutive class periods on 26 March, 7 April, and 9 April 2015.

Figure 1: Progression of the multi-day Newspaper Exercise (NE) Learning Module. 2015 additions to the module that differed from previous years are highlighted in yellow.

Definitions

*Newspaper Exercise module: 5 day module that included pre and post Newspaper Exercise presentations, completion of the Newspaper Exercise, a survey, a lecture, and a midterm exam

*Newspaper Exercise: a 3 day group activity that students worked on during Days 1, 2, and 3 of the module that included presentation creation, question generation, a poster presentation, and a Q&A session

The changes made to the original version of the Newspaper Exercise can be seen in Figures 2, and 3, which contain the original and 2015 versions, respectively.

Figure 2: Original version of the Newspaper Exercise (used in 2011 - 2014)

Figure 3: 2015 version of Newspaper Exercise (used during TAR project intervention)

Assessment occurred during the in-class group discussion portion of the exercise on module days 1 and 2, during the poster session on module day 3, during the survey and discussion on module day 4, and during the midterm exam on module day 5. One question on the exam explicitly addressed the concepts introduced in the Newspaper Exercise. The mean scores for the exam and the mean scores for this specific question on the exam were available for 2011 - 2015. Data from 2011, 2012, 2013, and 2014 used the original implementation of the Newspaper Exercise while the 2015 data used the revised version.

Figure 4 below shows an example presentation from the poster session on module day 3.

Figure 4: One of the student presentations during the 2015 poster session. The notes on the board originated from a discussion that two students had after one student asked one of the poster's authors to further explain a concept.

Results

Midterm Exam

The midterm exam that students took at the end of the Newspaper Exercise module had one question written specifically to test students' theoretical and conceptual understanding of radio frequency (RF) pulses, T1, T2, and T2*. Students had this learning objective outlined for them at the beginning of the module. Mean scores, out of 10, for this question and mean scores, out of 100, for the exam were compared to determine the efficacy of revisions made to the Newspaper Exercise in 2015. The individual data was not available and neither were the variance estimates for each year. Analysis was based on the available information, found in Figure 5, which included, for each year 2011 - 2015, the mean score on the question of interest, the mean score on the exam, and the number of students in the class.

Figure 5: Results from the midterm exam, each of which contained a question specifically designed to address information learning during the Newspaper Exercise. Individual data was not available for analysis. Years 2011 - 2014 used the original version of the Newspaper Exercise, while 2015 was the year in which the revised implementation was used.

A t-test was used to compare mean exam scores before and after the 2015 Newspaper Exercise revision. The 4 data points pre-revision were used to generate variance estimates for the mean exam and mean question scores. Because there was no way to estimate a post-revision variance, the post-revision variance was assumed to be equal to the pre-revision variance. Analyses were done in R and were weighted based on the number of students in the class for each year. Results were comparable for both weighted and unweighted results.

Significant improvement (p < 0.0001) in mean midterm exam score following the 2015 implementation of the Newspaper Exercise was observed when compared to the mean exam score using the original version of the Newspaper Exercise .

Similarly, significant improvement (p < 0.0001) was seen for the mean score of the relevant question on the midterm exam following the 2015 implementation of the Newspaper Exercise when compared to the original.

Mean exam scores and mean question scores were highly correlated with a Pearson correlation of 0.90 (Figure 6). This is unsurprising as the question score contributes to the exam score. However, it also supports the hypothesis that emphasizing active learning and quantitative literacy in a single exercise could have larger ramifications when it comes to learning other subject material.

Pearson correlation of mean exam scores with year was 0.498 (Figure 7), while Pearson correlation of mean question scores with year was 0.474 (Figure 8), suggesting only moderate correlation of student performance by year. This is encouraging, as it does not suggest a significant time trend associated with exam scores.

Figure 6: High correlation (0.90) of exam and question scores. This makes sense considering

the fact that the question score comprises somewhere around 10% of the exam score. It

also supports the hypothesis that embedding active learning and emphasizing quantitative literacy

in a single activity can have larger ramifications on learning than on that subject material alone.

Figure 7: Mean exam scores plotted by year. Note that there is not a clear linear trend by

year. Sometimes improvements in test scores can be explained by the time a teacher has

been teaching. Time does not appear to account for the increase in mean test score seen in 2015.

Figure 8: Mean scores for the relevant question plotted by year. Again, note that there

is not a clear linear trend over time, supporting the belief that time does not account for

the increase seen on this test question in 2015.

Survey

A survey soliciting feedback on the Newspaper Exercise was given on Day 4 of the Newspaper Exercise module. 32 students responded to this survey, which included all of the 28 students who sat for the midterm exam, as well as 4 students auditing the course who did not sit for the midterm exam. Based on this survey, students were uncomfortable with the idea of engaging in student-centered learning prior to receiving the information in a traditional lecture format. 31% of students specifically stated their desire for additional lecture time (Figure 9). Many wanted to receive a lecture before being asked to complete the exercise, while some (9%) specifically stated that they would have preferred a lecture to the exercise. Some students did admit that they ultimately thought the exercise was a good learning tool, but most of these students reported feeling uncertain, skeptical, or frustrated "at first". Additionally, 15% of students felt that either the exercise or their learning from the exercise felt "incomplete." 25% of the class had very negative impressions of the exercise, while 75% had neutral or positive impressions, as categorized following coding of qualitative data (Figure 10). Students also reported typical accountability issues associated with group work.

Figure 9: 32 students responded to the post - Newspaper Exercise survey soliciting feedback on their impressions of the learning module.

Figure 10: Overall impressions of the newspaper were largely neutral or positive. However,

those students with negative impressions were the most vocal about their displeasure.

Faculty Observations

Having students write down questions for further clarification on Day 2, before the poster session on Day 3, appeared to generate a substantially higher level of involvement during the poster session, as students had a specific goal to work toward-answering the individualized questions they had identified as unclear or troublesome during their preparation of the material. Dr. Meyerand noted that students had more questions and that they were of better quality during and after the poster session.

Discussion

Focusing on active learning and quantitative literacy led to increased exam scores and higher levels of engagement in Medical Physics 578 during 2015. Student understanding improved as measured by mean exam question score, mean exam score, and faculty opinion. However, students' perceptions of discussion-based activities was not as positive as their test scores might suggest. Some of them admitted that by the end of the module it seemed like a good activity, but many of them said they would have preferred more lecturing in whole class. This is an opinion that is probably most pervasive in STEM classrooms where students' historical experience is with a lecture-based format. The greatest struggle for this implementation, and for future ones like it, is getting students expecting traditional lectures to believe in the efficacy of active learning techniques. Talking about learning gains supported by the literature was not enough. It is possible that encouraging student reflection following active learning exercises would help students discover for themselves that the exercises were effective. A few student surveys indicated that, looking back, they were surprised to find that they did actually learn something.

The single most effective addition to the 2015 Newspaper Exercise was the day 2 instruction asking students to determine the topics that were still unclear to them, formulate questions, and then seek to answer those questions on day 3 during the poster session. This simple instruction elicited student-centered active engagement that paid dividends in comprehension, understanding, and literacy, as witnessed during the discussions during and after the Newspaper Exercise.

It is also important to note three differences between the 2015 course and the preceding years.

1. In the past, the MRI class was comprised of roughly 50/50 split between biomedical engineering students and medical physics students. In 2015, the majority of the class was comprised of medical physics graduate students. Given the nature of the two majors, it is conceivable that engineers would respond more enthusiastically to active learning and group work than physics students. This should not have affected class ability. However, it may have affected class perception of the teaching style.

2. The class was larger than usual in 2015, which contributed to some logistical difficulties. Previous activities for this course were designed for classes 1/2 to 3/4 the size. This didn't appear to affect learning, but it did contribute to student dissatisfaction related to group work.

3. 2015 was the first year in which the "ultrasound" imaging class and the"MR" imaging class were combined, meaning students only received half a semester of MR imaging instruction, instead of a whole semester. Again, this did not appear to adversely affect learning.

With burgeoning evidence supporting active learning, now is the time to introduce discussion-based activities into the classroom. This type of exercise asks students to play an active role in their education and also provides them with the opportunity to build skills integral to scientific communications and collaborations.

The 2016 version of the Newspaper Exercise, revised based on 2015 results can be found below (Figure 11).

Figure 11: 2016 version of the Newspaper Exercise (revised based on 2015 implementation)

Sources

Ewell, P.T. (2001). Numeracy, Mathematics, and General Education. In Lynn Arthur Steen (Ed.), Mathematics and democracy: the case for quantitative literacy (pp. 37-48). United States of America: National Council on Education and the Disciplines & The Woodrow Wilson National Fellowship Foundation.

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415. http://doi.org/10.1073/pnas.1319030111

Gormally, Cara; Brickman, Peggy; Hallar, Brittan; and Armstrong, Norris (2009) "Effects of Inquiry-based Learning on Students’ Science Literacy Skills and Confidence," International Journal for the Scholarship of Teaching and Learning: Vol. 3: No. 2, Article 16.

Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64. http://doi.org/10.1119/1.18809

Hallett, D.H. (2003). The role of mathematics courses in the development of quantitative literacy. In Why Numeracy Matters (pp. 91-98). United States of America: National Council on Education.

Lorenzo, M., Crouch, C. H., & Mazur, E. (2006). Reducing the gender gap in the physics classroom. American Journal of Physics, 74(2), 118. http://doi.org/10.1119/1.2162549

Pilzer, S. (2001). PEER INSTRUCTION IN PHYSICS AND MATHEMATICS. PRIMUS, 11(2), 185–192. http://doi.org/10.1080/10511970108965987

Prince, M. (2004). Does Active Learning Work? A Review of the Research. Journal of Engineering Education, 93(3), 223–231. http://doi.org/10.1002/j.2168-9830.2004.tb00809.x

Redish, E. F. (1998). Student expectations in introductory physics. American Journal of Physics, 66(3), 212. http://doi.org/10.1119/1.18847

Singh, C., Mason, A., Sabella, M., Henderson, C., & Singh, C. (2009). Physics Graduate Students’ Attitudes and Approaches to Problem Solving (pp. 273–276). http://doi.org/10.1063/1.3266734

Springer, L., Stanne, M. E., & Donovan, S. S. (1999). Effects of Small-Group Learning on Undergraduates in Science, Mathematics, Engineering, and Technology: A Meta-Analysis. Review of Educational Research, 69(1), 21–51. http://doi.org/10.3102/00346543069001021

Stith, J. H. (2001). Connecting Theory and Practice. In Lynn Arthur Steen (Ed.), Mathematics and democracy: the case for quantitative literacy (pp. 73-78). United States of America: National Council on Education and the Disciplines & The Woodrow Wilson National Fellowship Foundation.

Tall, David, Gray, Edward Martin, Bin Ali, Maselan, Crowley, Lillie, DeMarois, Phil, McGowen, Mercedes, Pitta, Demetra, Pinto, Marcia, Thomas, Michael and Yusof, Yudariah. (2001) Symbols and the bifurcation between procedural and conceptual thinking. Canadian Journal of Science, Mathematics and Technology Education, Vol.1 . pp. 81-104. ISSN 1492-6156