Ensuring the growth and sustainability of our OER initiative requires that we continuously evaluate and assess where we are and the impact of our program. During the initial stages of program development, we will measure our success based on whether we achieve key milestones associated with our identified SMART goals and strategic priorities. The key milestones are outlined below. As we move toward greater awareness, engagement, and adoption of OER, we will use the various metrics from The COUP Framework to measure the impact of OER on Cost, Outcomes, Use, and Perceptions (COUP).
SMART Goal: Develop, administer, and analyze a student survey that examines the effects of textbook and materials cost on education, purchasing behaviors, course completion, and student success. Develop and administer the survey by Spring 2021. Analyze the data in Summer 2021.
Milestones: Develop survey, establish needed partnerships, analyze and share results
SMART Goal: Develop, implement, and analyze the results of a textbook listening tour to gain a better understanding of how faculty select course materials; how they perceive and engage with textbooks; and their understanding, perception, and use of open educational resources. Develop open ended questions by Summer 2021, commence the tour during the Fall 2021 and Spring 2022 semesters, analyze the results in Summer 2022 and report results by the Fall of 2022.
Milestones: Develop questions and pilot tour, interview departments, prepare report and share results
SMART Goal: Identify and reach out to model programs in the Fall of 2020 to borrow or adapt successful program strategies, identify best practices, establish benchmarks for success, and build external partnerships.
Milestones: Identify model programs, reach out to model programs, incorporate best practices and resources gained
SMART Goal: Apply for the OpenStax Institutional Partner program in July 2021 to aid us in expanding our current OER initiative. The partnership would run from 2021-2022.
Milestones: Collect necessary data on enrollment and current OER impact, gather letters of support, complete application
SMART Goal: Develop digital storytelling content (e.g., Student Money Diaries, videos, podcasts, photos, infographics, and/or social media) to capture student stories around the cost of course materials and their effects. Develop the content during the Fall 2020 and Spring 2021 semesters and share it out for Open Education Week in March 2021. Plan additional events for Open Education Week 2021 (e.g., student-faculty panel, interactive whiteboards, Why OER? Webinar) to complement the digital stories.
Milestones: Establish needed partnerships, develop digital stories, develop events, share stories and events for Open Education Week 2021
SMART Goal: Form an OER Working Group and identify its charge by Fall of 2021.
Milestones: determine structure of the working group, send invitations to potential members, convene group and establish charge
SMART Goal: Design and deliver faculty workshops around OER and OER-enabled pedagogy. Design workshops by the end of Fall 2021 and deliver workshops in the Spring and Fall of 2022.
Milestones: Form needed partnerships, secure budget and resources, deliver and assess workshops
SMART Goal: Investigate the possibility of including open education in the guidelines for promotion and tenure by Spring 2023.
Milestones: Build key relationships, secure meeting with Chair of Appointments and Promotion committee
This section identifies specific metrics for evaluating and assessing the impact of OER on cost savings, student learning and success outcomes, OER usage, and faculty and student perceptions of OER.
Cost. The success of our OER initiative will be measured, in part, by the extent to which we achieve equitable access to courses by making them more affordable. This involves creating significant cost savings to students. Other financial and cost metrics include changes to bookstore and tuition revenues. We are using the following metrics to measure success in terms of cost:
Cost savings to students. This will include single term textbook savings, savings by academic departments, and total savings. To calculate savings, we will gather information about the cost of previous (new) textbooks and the number of students enrolled in the course. We will multiply the cost per textbook by the number of students in the sections that used OER. Since OER are free, the amount of money calculated is the amount of money saved (see OER Research Guidebook)
Number of students enrolled in OER courses
Number and percentage of OER sections
Number and percentage of high enrollment courses with an OER option
Number of students benefitting from savings
Changes in campus bookstore revenue
Change in tuition revenue due to changes in drop rates
Changes in tuition revenue due to changes in enrollment intensity
Changes in tuition revenue due to changes in persistence
Tracking changes in bookstore and tuition revenue are long-term goals since collecting data around these metrics first requires building cross campus partnerships with particular campus offices.
Outcomes. Student success is a central focus of our campus strategic plan. The metrics listed below are designed to capture the effects of OER on different indicators of student success.
Changes in the percentage of students receiving a C or better. This requires us to compare the final grades for every student in the course sections using OER instead of commercial textbooks to the final grades of students using commercial textbooks in other sections of the same class. To control for differences among teachers, the course and instructor will remain consistent (i.e., same course with same instructor at different points in time) (see OER Research Guidebook)
Changes in rates of completion
Changes in drop rates
Changes in enrollment intensity
Changes in semester-to-semester persistence
Changes in graduation rates
Usage. We are going to measure two forms of usage. The first is how student use of resources differs between OER and commercial materials. The second is the extent to which faculty and students are engaging in the 5R permissions made possible by OER (i.e., reusing, remixing, revising, redistributing, and retaining).
Earliest access to OER materials by students (when do students actually access available OER?) (See Lumen Learning's OER Champion Playbook)
Student engagement with OER materials. We will track downloads, pages views, session times, and other metrics using Moodle's engagement analytics)
Deleting material from OER
Inserting other open material within the OER
Moving material around within the OER
Editing material in the OER
Perceptions. How do students and faculty perceive OER after adoption and use? What do faculty and students see as the major benefits of OER? What do they see as the major challenges? We will need to determine how best to collect data for the metrics below (e.g., surveys, interviews, focus groups, and/or faculty OER reviews):
Faculty perceptions of the quality and comprehensiveness of OER
Faculty perceptions of the benefits and challenges of OER
Student perceptions of the quality and comprehensiveness of OER
Student perceptions of the benefits and challenges of OER