Overview
We chose to run both CTA and TAT tests. The CTA seemed most applicable because we could easily record and share with our teammates, and it also allowed for report accuracy regardless of lapse between testing and reporting since it is recorded. The Timed Assembly Tests were performed because they allowed us to measure quantitatively the sections that were more complex, or needed work. Some tests were performed without pictures, as this allowed us to test the effectiveness of our written instructions. We performed three iterations of testing, with revisions to the instructions between each version.
Formal instructions:
Welcome to the builders United usability testing. As a tester, you will not receive an image of what the final result is to look like. Please follow directions as explicitly as possible and do not worry about making mistakes – these actions will help us in our revision process. Please think out loud, no thought is irrelevant. We are especially looking for instances where you get confused or lost, but explaining when instructions are good or clear is helpful, too. Lastly, please note when you move from one assembly to the next. Enjoy!
First Iteration
Instructions Version 1
During these tests, the subjects were given the required pieces and told to complete the build to the best of their abilities. CTA subjects were encouraged to think aloud, and TAT subjects were encouraged to complete the set as efficiently as possible. This allowed us to see which parts of our instructions were hard to follow and needed to be changed
.
Test 1a
CTA | Full Instructions
Subject: Poppy | Age: 17 | LEGO Skills: 1/5
Results:
Subject was confused by step 10 and ended up placing the central wheels too far back, resulting in problems at step 16 when placing the back wheels.
Subject misread step 29 and placed the arm too far forward
Subject was confused by step 31 as the rover's left was her right.
Test 1b
TAT | Text Only
Subject: Cor | Age: 16 | LEGO Skills: 5/5
Results:
Base 0:26 - Subject used wrong orange slope
Wheels 0:59 - Subject briefly forgot which way was back and placed center wheels wrong, but managed to correct the error
Camera 0:45 - Subject was confused by step 19 as "either" can mean "both"
Arm 0:58 - Subject used the wrong orange slope at step 30. Subject was confused at step 31 as the rover's left was his right
Total 3:46
Test 1c
TAT | Text Only
Subject: Katie | Age: 19 | LEGO Skills: 4/5
Results:
Base 0:48 - No issues
Wheels 2:06 - Subject was confused by center wheel placement
Camera 1:14 - Subject was confused by step 19 as "either" can mean "both"
Arm 1:26 - No issues
Total 5:34
Test 1d
CTA | Full Instructions
Subject: Tawny | Age: 47 | LEGO Skills: 2/5
Results:
Subject took it slow, and did not encounter any problems. She thought the instructions were fun and easy to follow.
Test 1e
CTA | Full Instructions
Subject: River | Age: 9 | LEGO Skills: 5/5
Results:
Subject raced through the build, relying almost entirly on the pictures and encountering no issues.
Test 1f
TAT | Full Instructions
Subject: Evelyn | Age: 23 | LEGO Skills: 1/5
Results:
Evelyn is part of our group and is familiar with our build. However, attempting the instructions herself provided valuable insights as to possibly confusing wording.
Also note: this report came in after the iteration 1 revisions, so issues it discovered were not fixed until iteration 2
Base 1:33 - Step 1 "on the end" implies one end is different. At step 4, the "slanted orange piece" is ambiguous. Which part of the rover is the "front" should be more clear
Wheels 3:50 - Step 16 axle placement doesn't specify no overhang
Camera 1:01 - No issues
Arm 2:21 - Saying "rover's left" is confusing.
Total 8:45
Iteration 1 findings
The subjects that constructed the set without pictures encountered many more problems despite being very skilled, suggesting the text components for many of the steps were lacking. Many of the steps had confusing or ambiguous wording.
Iteration 1 changes
(s4) Added clarity for orange sloped "tile" (previously it was just "piece")
(s5) Added further orientational text to clarify which side is the back
(s10) Rewrote to mention colors for extra clarity
(s19) Replaced "either" with "one of" to avoid confusion
(s31) Added clarity for arm direction
Second Iteration
Instructions Version 2
These tests were conducted in a similar fashion to iteration 1, with the subjects given the pieces and asked to recreate the build.
Test 2a
CTA | Text Only
Subject: Paul | Age: 16 | LEGO Skills: 4/5
Results:
Subject was unsure if the back axle placed at step 16 should overhang or not
Test 2a
TAT | Full Instructions
Subject: Maggie | Age: 14 | LEGO Skills: 1/5
Results:
Subject payed very little attention to the instructions throughout the build, frequently missing important details and having to go back.
Base 2:48 - No major issues
Wheels 6:12 - Subject didn't overhang the front wheels and spent a lot of time trying to figure out what was wrong.
Camera 0:40 - No major issues
Arm 2:01 - No major issues
Total 11:31
Test 2a
CTA | Text Only
Subject: Elliot | Age: 13 | LEGO Skills: 3/5
Results:
Despite the revisions, Subject confused the two orange slopes.
At step 28, Subject placed the grey brick underneath the rest of the arm module instead of above it.
Test 2a
CTA | Text Only
Subject: Mariel | Age: 17 | LEGO Skills: 1/5
Results:
Subject confused the two orange pieces, but managed to realize their mistake at step 30 and fix it.
Iteration 2 findings
Despite our changes in revision 1, our test subjects repeatedly confused the two orange pieces. Additionally, there were several other minor problems, such as the back axle placement and arm placement.
Iteration 2 changes
Added Parts List
(s2) Changed "the end" to "one end" to avoid confusion
Renamed colors throughout for consistency
(s4, s30) Slightly reworded "orange piece" steps for better clarity
(s16) Added clarification for back axel placement
(s28) Rewrote entire step for clarity
(s31) Added emphasis to "rover's left" to avoid confusion
Third Iteration
Instructions Version 3
These tests were conducted in a similar fashion to the previous two iterations, however, this time the subjects were given too many pieces and had to find the correct ones using the parts list.
Test 3a
CTA (with timed Parts List) | Text Only
Subject: Cleopatra | Age: 15 | LEGO Skills: 3/5
Results:
Gathering the correct pieces took only 2:40 and went very smoothly. However, Subject selected a green 2x4 brick instead of the light green 2x6 plate.
Test 2a
TAT | Text Only
Subject: Evelyn | Age: 23 | LEGO Skills: 1/5
Results:
Evelyn ran through the instructions one last time to see if she could find any more issues.
Base 1:02 -Orange slope piece could use more clarity
Wheels 2:25 - Step 15 "Repeat steps" doesn't explicitly specify using a new axle
Camera 0:59 - Orientation and exact placement of camera is unclear
Arm 1:42 - No major issues
Total 6:08
Iteration 3 findings
This iteration went more smoothly, but still showed room for improvement. The orange pieces were still not distinguished enough without images, and some wordings in the Parts List were ambiguous.
Iteration 2 changes
Added even more clarity for the orange pieces. It should be very hard to confuse them now
Changed Parts List description of green plate slightly
(s15) Added clarity for axle pieces
Added troubleshooting section
Added a sectioned specifying exact camera module placements
Final Product
Instructions Version 4
This extensive testing and revision allowed us to create a set of instructions that is easy to follow, but difficult to misinterpret. Additionally, we added a trouble shooting section addressing some common problems or questions.