Evaluations

April 4, 2021

Formative Evaluations

Part of the responsibility of the Instructional Designer (ID) is the evaluate their ID project and its overall effectiveness. To do this, an ID will use an evaluation method, either formative or summative, depending on the phase of their project and the goal of the evaluation. This blog will explore formative evaluations, and briefly highlight another of form of evaluation which is used later in the process.

Formative vs. Summative

There are two types of evaluations that IDs will use: formative and summative. A formative evaluation is used when the ID is creating training. A formative evaluation is “about collecting and analyzing data to make decisions about the quality of training and how to improve it” (OPWL Instructors, 2020, pg. 146). Using formative evaluations will link all phases and activities of the ID project together. The other type of evaluation is a summative evaluation. A summative evaluation occurs after the training has been implemented to determine the overall effectiveness. to the right, is a table that describes and compares the two types of evaluations.

Adapted from Brenneman (1989); Tessmer (1994); Carliner (2003); Dick, Carey, & Carey (2009); and Morrison, Ross, Kalman, & Kemp (2013) cited in OPWL Instructors, 2020, pg. 146

When to use Formative Evaluations?

There are different methods for implementing formative evaluations, which may include reviews, trials, or tests. Although formative evaluations are typically used when the ID transitions from the Design and Develop phase to the Implement and Evaluate phase, they can start earlier. It is the responsibility of the ID team to consider the size of the project, complexity, and risk before they identify when and what type of evaluation they should use. Having a subject matter expert (SME) to review the task analysis or having a small group of learners test a specific portion, is considered a type of formative evaluation. No matter the how the evaluation is carried out, the goal is always “to collect and analyze data in ways that let IDs determine how to create and revise the training—as they are creating it” (OPWL Instructors, 2020, pg. 147). Doing this will help ensure time, resources, and money are being used appropriately and the training will be effective.

Two types of evaluation techniques that are typically used are Formative Reviews or Formative Tests.

Formative Reviews occur when IDs ask individuals to “review training deliverables and materials to help ensure that they will work as expected during training and in the workplace” (OPWL Instructors, 2020, pg. 148). These reviews include ID reviews, client reviews, editorial reviews, SME reviews, trainer reviews, and user reviews.
Formative tests occur when IDs ask individuals to “test draft versions of the training deliverables to help ensure that they will work as expected during training and in the workplace” (OPWL Instructors, 2020, pg. 148). These tests include One-on-one tests, Usability tests, and Pilot tests.

The ID needs to keep in mind that conducting formative evaluations, means you are asking for feedback. Sometimes that feedback is hard to hear, so the ID team should be prepared to receive comments on their work. It is the responsibility of the ID team to listen to the feedback, then identify how the feedback can be used to help them improve their instruction.

Image By Tumisu from Pixabay

In my professional experience, I have used formative evaluations, both reviews and tests, on every project. Typically, I have started the reviews in the analysis phase, with SME review of deliverables, then moved on to testing. I was not necessarily aware they were called formative evaluations, but the goal was to collect information so that I could revise each deliverable to ensure I was delivering the most effective project. Implementing formative evaluations into the process enables me to make changes along the way so that there wasn’t significant rework that needed to be done after implementation. Another benefit of using formative evaluations, is gaining SME and client buy-in. By having the SME and client review and provide feedback, it has helped me gain their trust and help them feel they part of the process, increasing their buy-in. I find this to be a valuable take-away from formative evaluation process.

Reference

OPWL 537 Course Instructors (2020). Instructional Design Course Handbook. Boise, ID: Boise State department of Organizational Performance and Workplace Learning.

Page updated

Report abuse