Evaluation Report

Evaluation Purpose:

Hannah and Kristen from NYSW mentioned that they wanted teachers to feel more confident in their skills and less pressure that they need to be perfect. They want their teachers to rely less on the NYSW team since it's ok to make mistakes through trial and error. Since the teachers are not performing surgery where they need to demonstrate perfection (@Maaike), we will not focus on recall. We focused our evaluation on comfortability: do the teachers feel more comfortable knowing that they have a more clear and concise guide with all the answers when they need it? Our goal is to help our learners feel more confident in their own abilities, so that they are able to take charge of their systems and troubleshoot on their own.

Methodology:

We designed a customer feedback survey to collect data from 14 classmates in our Design Process for Learning class because we were not able to share this survey with current NYSW teachers. We wanted to collect their feedback in order to adjust and improve the effectiveness of our current manual design. Two images of seeding instructions were provided to the participants. Graphic A is from the old manual and Graphic B is from our new guide. Participants were asked to rate each graphic's clarity and their comfort level to perform seedling task after reading each graphic. We also want to collect participants opinion and suggestions on what works in the new guide and what doesn't.

Data Analysis:

We created two affinity maps on Mural to visualize data from the Google Form results. To display the overall clarity and comfort score of these two different guides, we used pie charts to represent participants' comfort level before using the guide and after. Proceed to Conclusion and Recommendations below for data analysis.

After reviewing the exported data from the Google Form survey, we met on Slack video on Friday, April 16th to create the following Affinity maps on Mural. From each piece of data, we copied and pasted the comments or numbers onto a single sticky note. From these notes, we arranged these to find some pattern from the data.

Participants, aka our classmates, were asked to compare rate the clarity of two different seedling instructions and their comfortably using the guide to perform task on a scale from 1-10. Below, we created pie charts from the data.

Clarity: Seeding Instructions

Comfortability: Plant Growth Phases

From the data collected, there was an improvement for clarity in Guide B and comfort in Graphic B compared to Guide A and Graphic A. Although participants showed less preference for the graphics in Guide A, we realized that we did not properly define the values of the scale. We labeled 1 as "poor" and 10 as "excellent." However, the numbers between 2 and 9 were subjective. For example, why did someone choose 8 instead of 9?

Below are some suggestions we collected from our peers:

• The gif animation of the Graphic B plant growth timeline was distracting. They recommended to remove it or slow it down.

• Color coding keywords can be used throughout the guide to signify similar topics.

• Continue reducing text on the video pages. Instead, display a download button to print the instructions or open a new webpage tab for seeding steps.

• For Graphic B plant growth timeline, use larger arrows. Instead of a wavy line, straighten these lines to make the progress line more clear.

• For Graphic B plant growth timeline, consider making multiple options of this graphic so that teachers can see how all of the most commonly-used plants look from germination to maturity.

Brown, A. H., & Green, T. D. (2015). Chapter 10: Determining the success of the instructional design product and process. In The Essentials of Instructional Design: Connecting Fundamental Principles with Process and Practice. New York, NY: Routledge.

Rohrer, C. (2014, October 12). When to use which user-experience research methods. Nielsen Norman Group. Retrieved from https://www.nngroup.com/articles/which-ux-research-methods/