To measure the usability of our design, we have chosen to use both the Concurrent Think Aloud (CTA) and Times Assembly Test (TAT) due to their insights into the subjects’ thought processes while assembling the design.
We administered the CTA test by providing the LEGOs associated with the design, similarly to how a boxed set would be, the printed instructions, and an audio recording device, as well as instructions to voice all of their though processes, which will be recorded, along with the step that they are having an issue with.
For the TAT tests, we gave them the pieces necessary for the construction, as they would be provided in a standard LEGO kit, printed instructions, and started timing them as soon as they are given the materials. We started intervals at key points in construction, such as the base module, middle module, top module, and the conjoining of all parts into the final design, noting the steps and mistakes made, comparing amount of mistakes to time taken.
Before testing, we believe that the amount of pieces may end up causing issues with ease of creation. We have around 120 pieces in the construction of the build, so there’s a lot of room for error when making the build. There are also a few pieces on the instructions that the parts list is contrary to the piece within the instructions. The arrows on the instructions, while simplifying construction, may cause confusion in placements of the pieces.
Number of mistakes: 7
Base assembly step 2: Subject mistook color of thin LEGO piece
Base assembly step 6: Struggled putting the flower pot component and getting it to stay
Core Assembly part list: Subject was confused when there was no 2x8 present in the bricks, but there was on in the instructions
Core assembly step 15: Struggled getting the 2 x 6 pieces to stay in place
Core assembly step 17: Pushing the piece in resulted in some pieces collapsing in and detaching
Roof assembly step 8: Misplaced the angled pieces
Combining step 2: Misplaced the roof piece between the core
Combining step 4: Struggled placing base between legs of the core.
Subject comments: The arrows were a little unclear on exact positions, but this was mitigated by looking at the next picture and their exact placement, and the base piece had some issues in the final combination as well as not staying in place after being put in place.
The CTA test concluded that the instructions were a little unclear or confusing at times, though looking at the next step usually seemed to resolve this issue. With no words, the builder expressed they only had photos to rely on, which were easier to understand than wordy explanations due to just being able to look at them, allowing for quick comprehension and execution of the instructions. We fixed the error in the parts list for the core to prevent future confusion.
Overall time spent making design
· Time: 16:49.81
· Mistakes: 7
Time spent on core
· Time: 10:01.93
· Mistakes: 2
Time spent on roof
· Time: 2:25.02
· Mistakes: 1
Time spent on base
· Time: 1:13.89
· Mistakes: 2
Time spent on Gazebo
· Time: 1:38.79
· Mistakes: 0
Time spent combining:
· Time: 1:30.18
· Mistakes: 2
Comments: They stated that the final step of combining all of the modules was great except for the last part, connecting the base, which was very finnicky to get to work. Otherwise, the instructions were very easy to follow as well as concise in their presentation, though the base faced troubles of getting the flower-pot-like item to stay in place.
The TAT test concluded that by far, the section that consistently takes the longest is the core. Considering the sheer amount of pieces in the core, it’s entirely understandable that this module would take significantly longer than the others. The more experienced LEGO builder also, unsurprisingly took less time than the less experienced LEGO builder. (More to be added)
Redesign Decisions: One of the pieces on the parts list was misrepresented, so that piece was fixed and therefore is no longer an issue.