EDTEC 544 - Prototype Evaluation

Map Reading Skills for Delivery Drivers

EDTEC 544 Instructional Design Assignments - Spring, 2007

Initiating Memo

Initial Analysis

System Specification

Design Concept/Description of Product

Prototype Evaluation

Design Notebook

EDTEC Assignments Home


New delivery drivers lack sufficient map reading skills to do job effectively on the first day. The web-based instruction is designed to teach drivers practical maps reading skills so that drivers can determine the best route and arrange efficient delivery line, thereby improving the overall performance of their jobs.

Instructional Objectives for Design Concept

This prototype evaluation document is only for Module 1. The objectives for this module are: Given various maps, the student will be able to accurately complete the following tasks:

·         Point out the cardinal/ordinal directions

·         Use the street index to locate a given street

·         Explain the meaning of scale

·         Use the legend to locate a landmark

 Design Concept

The instructional product is designed as an asynchronous web-base course for individual self-paced learning. Module 1, Map Basics, provide all basic knowledge for map reading that they will need when learning other four modules. The figure below shows major topics covered in Module 1 and how learners will learn using the resources provided.

Figure 1: Topics covered, activities, and resourced provided in Module 1.

1.      Module 1 content is divided into bite size chunks so that learners will feel respectably easy to read through without stress. Intentionally, the module uses a non hierarchical structure to inspire learner engagement, e.g., when learners place cursors over the screen, topic titles will be enlarged one by one and a brief introduction for the enlarged topic will be presented.

2.      Multimedia is used to leverage different storage area of learners’ working memory. Each topic of Module 1 is introduced by a video that lecture the overview and objectives. Video clips are also used to present worked examples and scenario-base test problems.

Formative Evaluation Questions

The following questions are used to evaluate the prototype:

1.       Do the topics cover basic skills for reading maps?

2.       Are the navigation system logically replaced?

3.       Are the content descriptions easy to understand?

4.       Are the practice exercises considered useful and relevant by drivers?

How Prototype was Built

The prototype was developed using web page templates that come with Macromedia Dreamweaver. The designer replaced graphics and texts with relevant materials and structured a working prototype for evaluation. The prototype is created to communicate the appearance (key screen layouts) and interaction (overall navigation system) of Module 1 with learners.


Figure 2: Sample prototype screen shots that include navigation system and page layouts.


Time and costs for Prototype Development

The following techniques were used to minimize time and costs for prototype development:

1.      Use existing web page templates. These templates have already included basic navigation bars and menus. The designer only need to do simple cut and paste to replaces graphics and texts as appropriate to Module 1. After changing source and target information for each menu and button, a working example is ready for users to browse between key screens as they would do in a real product.

2.      Rather than spend a respectable time developing video, lectures are portrayed using short video clips acquired from Time Square live cam that were available over the Internet.

3.      Feedback loops for evaluation were not programmed. Instead, the designer described how feedback would be shown and what actions can be taken to testers during prototype evaluation process.

4.      Glossary was not created completely. Only few terminologies were listed to show how glossary would be layout and how learners can shift between alphabetical order and topic order in real product

Methods Used to Test the Prototype

§         Two learners, postmen, were invited to test the prototype.  

§         The designer gave a short introduction about the product. The purpose and process for the prototype evaluation were explained in detail. 

§         Formative evaluation questions were print out and handed to the testers.

§         Two testers tested the prototype separately.

§         During testing process, the tester operated computer independently. By thinking aloud, testers spoke out his confusion and praise. The designer took note.

§         On completing the testing, the tester was asked to write down their feedback for formative questions.

§         Changes were made after the first prototype test, and the process was repeated again using the same test process.

First Test: Observations/Findings

The testers gave following comments after the first round of testing:

1.          “I think the topics have covered basic knowledge a driver needs for reading maps. I liked the idea for presenting small amount of knowledge each time however I would like it more if the font size for body text could be increased a little.”

2.          "I have no problem to find the content I need by names of the menu."

3.          “The content description included in the prototype is understandable; however, there are four topics in the module but only a small amount of content were presented here, I would like to see 1-2 more topics.”

4.          “I like how the topic titles are enlarged when mouse over and how I am allowed to control my own learning direction and speed.”

5.          “The tests items are useful and very close to my real work however, feedback are also important. It accounts for whether or not I can remember and use what I learned from the product. I think at least one sample that demonstrates the feedback loop should be included in prototype.” 

Modifications to the Prototype Based on the First Test

Base on tester comments from the first round of prototype evaluation, following modifications were made:

1.          Font size for body text was increased from 10 pt to 12 pt.

2.          Add additional learning content to prototype.

3.          Create a simple feedback loop for one test item using ConceptTutor to show how and what feedback would be presented after answer was submitted.

Recommended Modifications of the Design Concept Based on the First and Second Tests.

The following design recommendations are suggested from prototype evaluation:

1.          The small knowledge chunk strategy has proved to be welcomed by prospective learners and should be carried on in coming design and development; however, it is important to group content into different categories and put logically related chunks closely together. This will minimize necessary learning cycle.

2.          Scenario-based test and worked examples are recommended to use as many as possible because learners expect that they can use the skill in their work right after they learn the product. The product must ensure sufficient opportunities for simulating practice.

Lessons Learned

Following lessons are learned from prototype evaluation:

1.          Interface design is more important for learners with basic literacy than for learners with higher academic education.

2.          The non hierarchical structure used in this prototype is good for product that contains only few topics. For a complex learning knowledge, the non hierarchical structure might cause confusion from learners.

3.          Learner with basic literacy prefer to learn through audio and visual than through reading text.

4.          Prototype building could be costly and time consuming. Be sure to discuss how specific the prototype should be with clients before giving final quotation.