Formal Usability Study

See the Final Refinement Phase Checklist for all due dates for this assignment.

Formal usability studies are designed to answer concrete questions about specific aspects of your design (or about some other aspect of human-computer interaction). In order to be able to gather rigorous data, you need to design an experiment that holds most things constant while varying only the element (the "factor") that you wish to test. You will then select something to measure (the "response variable") to see how changes in the factor affect that response. A suitable experiment might compare two variations on one of your design decisions or two different interfaces (but not too different or the results might not be valid).

Although there will not be time to conduct a full study, this portion of the assignment asks you to create a hypothetical formal study that you could run to evaluate your interface. You will run a pilot version of this study with two of your users.

1. Identify some specific question that would be helpful in your design and for which a formal study might be appropriate. Some possible dimensions include:
   • quantitative usability metrics such as response time or lost-ness
   • observed behaviors, such as taps and swipes (which may best be captured through a recording viewed by a teammate who was not present at the interview)
   • preferential usage or efficiency with alternative designs (A/B testing), in which case you should explicitly set up your experiment to compare two versions of your system that differ in very specific ways such as layout, specific graphic elements, use of color, font choice, ....
   • learning curves
2. Pick something that seems significant in the context of your particular design, of course. If there is a particular design decision that your team has been wrestling with, this is the time to collect quantitative data on it!
3. Write up your experiment design and post it on your site before class by the time designated on the Final Refinement Phase Checklist. Your writeup can be a page or less, but it should clearly spell out the logic of your experimental design:
   • Explain the hypothesis that your formal study would allow you to test.
   • Describe the factor(s) (independent variable(s)) that you will vary:
     1. What are the different experimental conditions that you will try?
     2. What will vary and what will stay constant?
     3. For each factor that you vary, what are the different conditions that you will test?
   • What dependent (response) variable(s) will you observe?
   • If your hypothesis is correct, what pattern would you expect to observe in the response variables?
   • Also explain how many subjects you would ideally test, how many iterations each would participate in, and if each participant tries multiple conditions, how these conditions would be ordered and distributed over participants. Note: if your team has no prior experience in the design of controlled experiments, please indicate that in this section.
4. Pilot your test with at least two users. You may combine this with informal usability testing or you may do formal and informal testing separately.
5. After your usability study is complete, and no later than the date provided on the Final Refinement Phase Checklist, you should add the following to the page documenting your experiment design:
   • A general overview of what you observed from both informal and formal usability testing in this round.
   • A specific report of the response variables on your formal usability pilot, i.e., the thing you said you would be measuring.
   • An interpretation of those response variables: if those results held with a larger user population, what would it mean for your system?
   • An explanation of what you intend to do in the next week (or what else you would do if you had more time) in response to what you've seen so far. This is the most important part of the write-up, since you need to think about how you would fix your system as a result of what you observed.