Usability Testing

Usability testing (UT) involves watching and listening to "real" people use an application in realistic scenarios. In contrast to Usability Assessments, UT removes subjective opinion (both experts and client management) by focusing on how people actually behave when interacting with an application.

The success of UT depends on how well the test participants are chosen so that they resemble people who will use the live system, and how well the test scenarios are designed so as to create realistic situations. UT is most realistic when the fully functioning system is working, but UT also has value when tested using a "Wizard of Oz" prototype system before the real system is completed. This WOZ testing is particularly valuable when there are questions about how the target audience will interact.

A typical usability test requires two days of testing, with six participants each day taking up to an hour each. There are several costs involved: (a) Testing is often best accomplished locally to the client company so that test participants are either existing customers of the company, or at least resemble such customers in demographic characteristics, such as regional accents, education levels, and other factors. Local testing may therefore involve travel expenses for our staff. (b) In most cases, participants must be motivated to devote their time and to travel to the testing site, and the best motivation is cash or gift certificates. (c) Client personnel may be needed to help set up the test system, and we highly recommend that client personnel be observers of all testing so that they can verify the validity of the test procedures by direct observation. (d) Testing must be conducted in a quiet environment without distraction of the participants or of the client observers. This may require the rental of office or office-like space. (e) Testing should be recorded, at least with audio, and preferably with video too, both for later analysis and also to show client management who are not present as observers. There may be expenses involved for recording equipment.

The basic deliverable from UT is a written list of specific recommendations. Typically there will be recommendations for changes to the design of the application, and for tuning of recognition grammars. There may also be broader recommendations for changes to client procedures for serving customers so that the total customer experience of the client company is a positive and profitable one.