In order to evaluate the product, a user test study will be conducted using the prototype as a stand-in for the physical product. A virtual mock-up of the phone and watch application will be provided as prototypes of the interface. The test will be conducted with the co-designer. Due to the nature of his short-term memory loss, it will be important to establish whether the product is intuitive and familiar enough for the user to be able to manoeuvre the interfaces effectively with no outside interference. Although the product met the requirements list created through the co-designing process, the test aims to investigate/check whether there are elements to be removed or improved. The group wants to discover to what extent the product contributes to the co-designers daily life compared to his current appointment making process and notifications.
Does the application help the co-designer make an appointment more efficiently?
Does the application have any affordances that are not intuitive or familiar?
Does the watch help the co-designer by notifying him when an appointment is?
Application
Is there a difference between using it the first and the second time?
How long does it currently take for him to make an appointment?
Is he able to use the application for planning an appointment without help from others?
Are there functions that need to be adjusted?
How much time will be taken in each step?
Watch
Whether it is possible for the co-designer to accurately understand the meaning of notifications.
Does he wear the watch when he is on his own?
What is his response to a notification on the watch?
The user study test will be conducted according to the plan below. The co-designer will be informed about the test plan beforehand and he will be asked for consent to participate in the test. If he agrees, two group members will take on the role of researcher and visit his place with the prototypes.
The whole process of the test will be video recorded to be used in later analysis. The test will consist of multiple tasks. While the co-designer executes the given tasks, the researchers will measure the time with a stopwatch in every process, and observe his facial expression, movement, and behaviour directly as well as his use process, this data can then be analysed to figure out difficulties and inconveniences that will arise while he uses the product.
After completing the experiment, the co-designer will be asked to fill out a questionnaire so that his opinions about the product will become clear. To avoid the co-designer forgetting before it’s time for the questionnaire, after each task he will be asked for his initial thoughts in one or two sentences.
Location: co-designer’s house
Participant: co-designer
Equipment: prototypes (watch, phone application, smartwatch), camera, stopwatch, questionnaire
Task 1 will be repeated with two different applications, which are the current application that the co-designer uses and the application designed by us. This way a comparison can be made between the different applications to express the efficiency and find the difference in the progress of use.
1.1 Ask to make an appointment.
Could you make a dinner appointment for April the 25th at 6 pm? The location is a restaurant of your choosing in the city, you are going there by bike and you will meet Jim and Emma there.
Input the basic information, input the location, calculate the time it takes through Google Maps, modify the proposed time, and set the departure time.
1.2 Change the appointment.
Could you change the time to 7 pm? You also decided to go to Jim and Emma’s place instead. This time you can go there on foot.
The time and location of the appointment are changed so he needs to adjust some information on the agenda.
1.3 Check the calendar for an appointment
Do you have an appointment on April the 30th at 14 o’clock?
Things to observe
Time taken in each step and overall time
Movement of his finger and hesitation
Progress during the repetition of using the same application
Question round 1
How easy was it to find an upcoming appointment?
Scale of 1 through 5 (very easy - very difficult)
How easy was it to make a new appointment?
Scale of 1 through 5 (very easy - very difficult)
How easy was it to search for the correct location?
Scale of 1 through 5 (very easy - very difficult)
How easy was it to change the departure time?
Scale of 1 through 5 (very easy - very difficult)
What did you like the most about the application?
What did you dislike about the application?
Is there anything missing in the application?
Is there anything that is unnecessary in the application?
For task 2 an apple watch will be used, with the screen showing the designed watch interface.
2.1 Explain the interface of the watch
Could you explain to us what you see on the watch? How do you think it works?
If he is able to tell us how it works without us explaining beforehand, it indicates if it is intuitive.
2.2 Check at what time the next appointment is
Could you tell us at what time your next appointment is?
2.3 Reaction to the alarm
When you get a notification, could you play out what you would want to do?
An alarm will sound and the screen on the watch shows that he has an appointment.
By not explaining what the notification is exactly, a more genuine reaction can be observed, because he will forget every day what type of notification he gets from the watch.
Things to observe
How often does he check the watch?
His reaction to the alarm. Does he check his phone? Does he know what he needs to do? Does he look uncertain?
Question round 2
How easy was it to see when your next appointment is?
Scale of 1 through 5 (very easy - very difficult)
How easy was it to use in general?
Scale of 1 through 5 (very easy - very difficult)
What did you like the most about the interface?
What did you dislike about the interface?
Is there anything missing on the interface?
Are there more functions that you would like to have on the watch?
Is there anything that you would like to remove from the interface?
For task 3 the model will be given to him, to try out if he can implement wearing it into his routine. This will be done by asking and relying on his honest answer.
3.1 Wearing it
Through WhatsApp, the following question will be asked for four days in a row, in the morning:
Are you currently wearing the watch?
What is the battery level of your watch?
When did you last charge your watch?
Question round 3
Is it comfortable to wear?
Does it suit your style?
Can you see yourself wearing this on a daily basis?