Evaluation

Design and Conduct Summative Evaluation                                                                                                                

Evaluation is the process of determining the effectiveness of the instruction. Summative evaluation is the review of the finished instructional product. Other types of evaluation take place in earlier stages of the Dick and Carey model.

According to Dick & Carey, the evaluation phase of their instructional design model includes summative assessment of the completed instructional product and formative evaluation of effectiveness of the overall design. 

Formative evaluation provides data for revising and improving instructional materials so as to make it as effective as possible for larger numbers of learners. Formative evaluation has typically included any of the following forms:
  • One on One: One evaluator sitting with one learner to interview
  • Small Group
  • Field Trial 

The purpose of summative evaluation is study the effectiveness of the instructional system as a whole and is conducted after the system has passed through its formative stage. Summative evaluation can vary in terms of its scale as well as the time period covered. 

Owing to the constraints of completing all phases of the design model within the span of slightly less than an academic semester, the evaluation phase of the project combined both summative and formative evaluation in one tool, a Google-based survey embedded within the site that housed the instructional content. The timeframe for evaluation responses has, by necessity, been limited for the purposes of this analysis to the two-week period of the instructional product's implementation. As a self-directed assessment tool, its inclusion within the instructional wiki was intended to maximize learner convenience and support learner willingness to complete it. Both utilization of the instructional material and completion of the assessment survey were optional activities. While the information presented here reflects the conclusion of the time allocated for implementation and subsequent evaluation within the parameters of the associated graduate course, the intent is to continue to monitor learner use and assessment of the instructional product as the tutorial developed is one component in a larger, ongoing staff development initiative. Where feasible, future revisions to the content may be made as needed, based on future user feedback.

Implementation Summary

The Creating Accessible Instructional Materials tutorial was 'launched' via the Upstate TLT News online newsletter on November 10.  The information about and invitation to complete the tutorial appeared as follows:
Screen snip of tutorial announcement
The tutorial was added to universal design and accessibility learning materials for a teaching online course running concurrently for 22 faculty members. Members of the learning technologies department advisory committee were invited to complete the tutorial to provide evaluation feedback. A reminder message was sent to the campus on November 18, as well as reminders to the various other groups invited to use the tutorial. 

Additional evaluation feedback was sought and provided from the campus director of disability services. She completed all parts of the tutorial, completed the survey, and provided suggestions for improvement for future iterations from the perspective of an SME. The director also graciously offered to consult her professional network for further review. She sent the tutorial link to her Project ShIFT colleagues. (A description of this project from the director: Project ShIFT [Shaping Inclusion through Foundational Transformation] is a Department of Education grant-funded initiative in which 25 disability services directors were selected to participate in a national 3-year development program designed to change the focus of our services from disability-centered to access-centered. Particular focus was on shifting access to a campus-wide responsibility, including educating faculty about how to design accessible courses that do not require accommodative intervention [reducing the need for specialized accommodation, and thus normalizing the disability experience]. Several members have shared results of ShIFT at national conferences and have worked on publications developed through the Project). Members of Project ShIFT completed the survey as a part of their review. The survey response data includes their input. 




As of November 22, 12 completed surveys have been submitted. Overall, the formative evaluation items resulted in ratings of the tutorial site that were more positive than not. Some of the suggested items for improvement may merit consideration moving forward (ex. adding content area specific suggestions), whereas others such as the Mac software issue may remain less of a priority due to the very small number of Mac users among the learner population. The compiled summative responses are as shown here and in the open-end text responses that follow:

Self-assessment Responses from "What did you learn?" survey:

survey results

Learner responses to “Write your personal definition of 'substantially equivalent ease of use' as it pertains to your specific course context and delivery mode”:

·      Just as faculty strive to meet the needs of varying learning styles, so should faculty ensure that course materials are provided in such a way that all students have equal access.

·      Substantially equivalent ease of use as it pertains to my specific courses.  My courses are online, which does require the student to have computer access as well as visual ability.  I have many different types of activities within my courses-power points, articles, tests, discussion board questions including responses to classmates and projects.   I have had a student who was learning disabled.  For him, I allowed extra time on test completion.   Otherwise, the required assignments were the same.  Students are given a week to complete assignments, so completion of weekly work wasn't an issue with this student.

·      As an educator in the field of literacy and English as a second language, equal access to curriculum for me has long been more than simply providing each student with a teacher and a book. We are always looking for ways to make course content comprehensible while maintaining the integrity of higher order thinking and 21st Century Skills. In short, my definition is that each instructor has the responsibility to understand their students' needs (physical and/or cognitive) and to make the curriculum accessible via myriad pathways of instructional design and delivery.

·      My delivery mode is primarily in a face to face context through the use of PowerPoint presentations.  Substantially equivalent ease of use will require that I use the templates provided in PowerPoint along with the creation of Alt Text for hyperlinks and images which convey course content.  In addition, I will need to add explanatory material in the notes section.   This will allow students who need to use auditory as the primary sense to better understand and receive the material.

·      Everyone regardless of disability deserves the ability to access materials for the course.

·      In Composition, students need to receive clear guidelines for assignments, effective models of writng, and the needed tools to put them on a path to success. I have created graphic organizers, bulleted lists of tips and reminders, step-by-step explanations of as well as specific tools and strategies for completing these assignments. Using various forms of presentation, including PPT, WORD, Screencasts, Prezi, etc., I can make the assignment expectations clear and give students various research processes using the above mentioned tools. In addition, I can make my classes more engaging with the iclusion of visuals and other participatory activities, making sure that every student has the opportunity to participate.

·      Faculty being mindful when creating instructional materials of ALL students.  By creating headings, using alt text for pictures, etc. they make the materials accessible to all students. 

·      Providing options and creating curriculum that is accessible by the majority of learners.

·      as a disability services director, my definition is that it is something that may not be exactly the same as what the majority of the students are using, but it provides equal access in terms of resources, instruction, and information that is given to other students.

Participant responses to the formative items from the assessment survey are summarized below:

Feedback on the tutorial:

tutorial feedback results

Learner responses to “most meaningful” aspects of tutorial:

The webaim screenshot documents for Word and PPT

Power point information

The actual screen shots with step by step narration!

Learning how  Alt Text is used to provide greater accessibility

Learning different ways to accommodate students 

Explanation of Alt Tags

The knowledge gained.  

Videos, clear examples, concise information

helpful links to explore more information 

The focus on helping professors learn to make technology accessible for all students

Learner responses to “suggestions for improvement”:

Clearer outline of content.

nothing

I think highlighting ( including in the screen cast rather than as an additional resource) how the end product is seen and heard by students using a screen reader would be helpful

demos on keynote, pages or other mac programs.  I don't use word or powerpoint.

Add some content specific suggestions

One of the tutorials is outdated and refers to the older versions of ADA and IDEA.  

videos in first section seemed geared to elementary school in language choice and student representation(children in classroom)  Not engaging for postsecondary.  I don't want to see children when talking about college students. 

Use videos that show college students.  Use of elementary school classrooms challenges me to think about how these concepts apply in higher education.

I would not use the term "differently abled", as it is a bit outdated.  Captions for the screencasts, Would like to see it look a little more like it is geared to higher education



Source: 

Lee, H. & Lee, S. (n.d.). Dick and Carey Model. Retrieved from http://www.umich.edu/~ed626/Dick_Carey/dc.html.


Comments