Objective

“Perform a health check on the existing ABC Company IVR application call flow, looking for adherence to industry best practices for User Interface design and opportunities for a better caller experience with the final goal to increase efficiencies in the IVR.

We will perform a 2 day on-site meeting to discuss with the business owners, technology owners, and call center people to do a discovery of the pain points around the existing IVR call flow. At the completion of the 2 day meeting, the team will analyze and review the information gathered and supply recommendations to meet the objective.”

Two resources spent a total of three days on site. Consultant focused primarily on the XYZ IVR application. Following the on-site visits, team spent another week analyzing collected data, and writing up the results.

Background & Application Description

ABC Company is an electricity generator, services provider and distributor in San Francisco, California. There are two entities within ABC Company that provide power and customer service, although both are serviced by the same call center. One is the deregulated entity and the other the traditional public service type utility that people get by default. In addition to power, the call center also handles inquiries for municipal gas, water and sewer utilities.

The applications discussed here are designed to figure out what callers are calling about, in order to route them to the appropriate customer service agent. An additional function of the IVR application is to identify the caller, in order to collect their account information and enable a ‘screen pop’ on the CSRs desk of the caller’s account information.

There are a couple different applications deployed for different types of calls, and different phone numbers, including a general number, a number primarily advertised to residential customers, a number advertised to business customers, a number advertised to the deregulated provider, but the applications are all very similar, to the extent that they ask the caller to select from a menu of call reasons, then, depending upon the reason, they may attempt to authenticate the caller, or, play back some basic information, such as account balance or phone numbers to dial.

Methodology

Prior to coming on-site, the following was requested:

- Test phone numbers and accounts numbers;

- Call flow diagrams;

- Requirements documents;

- Technical Design specifications;

- Grammar files;

- IVR logs;

- Wrap codes;

- Whole call recordings;

- Call traffic statistics;

Some of this information, but not all, was delivered before coming on site.

On-site Agenda

Various interviews with ABC Company staff, including:

- Technical Account Manager;

- ABC Company Project Manager

- ABC Company Call Center Director

- ABC Company Call Center Manager

- ABC Company Call Center Technical Lead

In addition, we had conversations with a number of call center agents and agent supervisors.

Architecture

The call center architecture is shown in the attached diagram entitled ABC Company Production Architecture.vsd. There are 48 ports of IVR in HA mode along with 48 ports of Nuance OSR 3.0 speech recognition and 24 ports of Realspeak v2.4 Text to Speech. A summary of the hardware characteristics deployed in the various components is given in ABC Company Hardware.xls. The IVR applications were built using their VXML Studio development environment.

Call Flows

The Call Flow documentation we received is not quite current with the existing applications. We received ABC COMPANY Master Call Flow End state_revised v2.vsd as well as ABC COMPANY Frontline - Open Hours IVR Flows 2008.vsd and ABC COMPANY Frontline - Closed Hours IVR Flows 2008.vsd. To help get a handle on what the dialed applications sounded like and behaved, we created a new call flow ourselves, entitled ABC Company_Call_Flow_v1.0.vsd. One advantage of the format of the new call flow we produced is that the prompt list, along with prompt names and grammar names, can be automatically produced directly from the Visio diagram. We recommend that any call flows of new applications developed be kept in synch with the code base.

Requirements Documentation

The requirements document we received is entitled ABC Company Frontline - IVR and Routing Business Requirements Document 2008.doc.

Technical Design Specification

The Technical Design Specification we received is entitled ABC COMPANY_TDS_v1.1.1.doc.

Phone Numbers

The following phone numbers were provided to access the applications:

- 877-123-2222 (Sales)

- 877-123-3333 (Repairs)

- 877-123-1111 (Business)

- 877-123-4444 (Consumer)

- 888-123-5555 (Marketing)

Test Account Numbers

A listing of test account numbers is given in ABC COMPANY - Testing Resources List - 2008.xls.

IVR Logs

There were no IVR logs made available to us for analysis. We were informed that there were none available.

Grammars

The grammars provided are in the attached zip file named ABC Company Grammars.zip. There is no easy way to tell which grammars are active at which points in the application from the documentation, but from inspection of the grammars and interaction with the application live, we were able to infer this information. We recommend that future revisions of the documentation indicate which grammars are active.

Interviews

All interviewee’s indicated they felt there were recognition issues with the applications. ABC Company has a caller satisfaction program in place, and the feedback has been consistent across all applications (i.e., business or residential) that caller’s are dissatisfied with the IVR. We were not given access to these results.

Another ‘pain point’ described by ABC Company was the inability to quickly record and deploy emergency messages. Sometimes this ability is called dynamic announcements or Message of the Day. Currently, in order to deploy a new prompt, a compile-release-test cycle is used, which takes up to 10 days.

A third point made in the interviews concerned the lack of log files for the IVR application.

Whole Call recordings

A set of of live customer calls was recorded 12/10 and delivered on 12/12.

Recommendations

Hammer Stress Testing

It appears that Hammer stress testing was never run on the application suite. While current call volumes are reportedly low, we recommend stress testing in order to determine the limits of growth of the current environment.

Recognition Tuning

There has never been a round of recognition tuning of this application. In fact, due to the absence of logs, recognition accuracy is largely unknown. There are a number of log analysis tools available, including Nuance OSI and a native toolkit. These tools should be deployed and the results analyzed. Since it is highly likely that the application should be re-written anyway, however, it is probably prudent to make the changes suggested below and then perform a round of recognition tuning.

Documentation

The call flows should be brought up to date with the existing applications. They should include prompt names and prompt text, as well as grammar names active and when barge-in is enabled and disabled. Also, the Requirements document and Technical Design Specs are really aimed at the overall call center infrastructure, and would not be sufficient to hand-off to a development team and have them build the applications. It is our understanding from discussions that the application specification was ‘reverse engineered’ from an existing application. While it may not be practical at this point to re-create this documentation, it might help ABC Company arrive at a common vocabulary/documentation set for future development going forward.

Use of Speech vs dtmf, dtmf Back-Off

There are no functions in the application that require speech input in order to operate. That is, all the functions could be supported by DTMF input only. In addition, while many of the menus in the application have DTMF enabled, they do not have prompts that tell callers that DTMF is available or how to use it. Finally, there is no DTMF fallback. That is, after a misrecognition event, it is common in VUI design to prompt the caller to use DTMF instead of speech, but none of the menus have this feature. Providing DTMF fallback is recommendation number 5 in the GetHuman standard. (http://www.get2human.com/gethumanStandard.htm)

In a recent survey of users and technology providers, one study[1] asked users “How often would you prefer to use a speech recognition system rather than a touchtone system?”. The responses from users were as follows:

- Most of the time – 8%

- Depends on the time of day – 7%

- Depends on how much time I have – 10%

- Depends on where I am – 7%

- Depends on reason I’m calling – 23%

- As little as possible – 43%

DTMF replacement has been the buzz-word behind speech recognition since its inception as a viable technology. The perceived benefit is that it offers a more natural interaction to the caller as well as the ability to gather richer, alpha-numeric data, as opposed to what has traditionally been the more stilted, numeric input-only interactions offered by touchtone applications. These benefits appear to have escaped consumers, as evidenced by the 45% that would prefer to use speech recognition over touchtone applications as little as possible.

Providers of speech recognition technology are sometimes loathe to use DTMF, and sometimes exhibit the tendency to act like “If the only tool you have is a hammer, everything looks like a nail.”

To be sure, we do heartily recommend speech recognition as a great technology that can make caller interactions faster and easier most of the time. However, we also must recognize that sometimes it is more appropriate to use DTMF if it is the best solution for a given problem.

One strategy we recommend is to have a fall-back strategy as follows.

First prompt, speech or DTMF input active: “Which would you like, X, Y or Z?”

Second prompt on first error, speech or DTMF active: “Say X or press 1. Say Y or press 2. Say Z or press 3”

Third prompt on second error, DTMF input “For X, press 1. For Y, press 2. For Z, press 3.”

Another strategy to consider is backing off to DTMF completely in some cases. That is, after some number of errors with the speech recognizer, that the application fall back to a DTMF input only mode. This is a useful strategy in cases of a poor audio connection (e.g., cell phone drop outs or high background noise) or in cases where the caller’s speech is poorly understood, perhaps because of a heavy accent or speech pathology.

For frequent callers who represent a significant number of users and who are intimately familiar with the application it may be faster to punch in the familiar numbers on the phone pad as opposed to speaking the entire string of digits and will preserve their routine productivity.

Language

<City> immigrant, non-native English speaking population is surprisingly large and growing. In 2001, about 1 in 6 persons were classified as allophones, that is, people who had neither English nor French as a native language.

If other languages are not supported with separate inbound lines, or supported by a language selection, then the least that can be done to accommodate non-native speakers is to allow them to use DTMF.

Barge-in is Disabled on Very Long Legalese Prompts

There are two messages that are rather long and written as legalese. These prompts cannot be interrupted:

You can choose any retailer listed. Electricity delivery to your home or business isn't affected by your choice of retailer.

Thank you, before I transfer you to an agent, please listen to the following important message. ABC Company requires personal information, which is protected under privacy law and the electricity industry's code of conduct. We use this information to bill and collect payments, to service your account and to develop and market our products and services. The ABC Company personal information commitment on ABC Company.com explains how we collect, use and disclose this information. To limit the use or disclosure of your personal information to the minimum required to serve you, please advise our agent. Please note ABC Company Energy serves customers on the regulated rate tariff and default rate and obtains personal information in that regard. ABC Company Energy will not use this information for competitive sales and marketing purposes without active consent or as permitted in the Code of Conduct. Thank you. Your call is now being transferred.

The second of these prompts takes almost a minute to play.

Legal disclaimers in IVR applications are often considered important from the company’s legal team’s perspective, but are often annoying to the callers, and may not really be required for all callers in all situations and lines of business. It has been our experience that messages can often be shortened, re-written to be more understandable and user-friendly, and sometimes omitted entirely, after further review by the legal team and the IVR team.

It is important to try to reduce the legalese for a number of reasons. First, it can be confusing to callers, and can sound intimidating. This reduces caller satisfaction. Second, it increases time to task. Time to task is a measure of effective VUI design:

When customers call an airline, bank, or other business, they generally have a task in mind or a problem to solve. Time-to-task measures the amount of time it takes for a caller to begin the task he called about. Lengthy instructions, references to a Web site, untargeted marketing messages, or other irrelevant information at the top of a call delay callers from their tasks.[2]

Time to Task is calculated as the time it takes from answering the call to the time the caller starts performing the desired task. Increasing time-to-task results in decreased caller satisfaction.

Even if the message needs to be played before transferring to an agent, it might be shortened significantly. A recent article[3] discussed some of the legal issues surrounding these messages. Experts there advise that the simple message “this call may be recorded” is sufficient. GOOG 411 (800 466 4664), the telephone search application run by Google uses the drastically short: “Goog 411. Calls recorded. What city and state?”

Another suggestion is to give callers a way to skip through the message if they do not wish to hear it. For example: “To skip the following message concerning information privacy, press 1 now.”

No N-best skip lists in Account number and House Number

(detail removed)

No Hesitation Prefixes in Grammars

(detail removed)

No Politeness Suffixes in Grammars

(detail removed)

(other 15 sections with specific recommendations removed)

References

Bruce, e BallentinIts Better to Be a Good Machine than a Bad Person, www.beagoodmachine.com

Mike Cohen, James Giangola, Jennifer Balogh, Voice User Interface Design, Addison Wesley, 2004.

Patrick Commarford et al., “A Comparison of Broad Versus Deep Auditory Menu Structures,” Human Factors: The Journal of the Human Factors and Ergonomics Society, Volume 50, Number 1, February 2008, pp. 77-89(13)

Dimension Data/Cisco, Alignment Index for Speech Self-Service, 2007.

Larson et al., “Ten Criteria for Measuring Effective Voice User Interfaces,” Speech Technology Magazine, Vol 12, 2005.

Jenny McKienzie, “Do Users Really Care About Modality?,” presentation at SpeechTech 2005.

Clifford Nass and Scott Brave, Wired for Speech: How Voice Activates and Advances the Human-Computer Relationship, MIT Press, 2005.

Scott Reeves and Clifford Nass, The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places, CSLI/Cambridge University Press, 1996.

Phil Shinn, “N-Best Skip Lists: A Practical Guide,” Proceedings of the Applied Voice Input Output Society, Jan. 2006.

The Voice User Interface Designers Group: http://groups.yahoo.com/groups/vuids

[1] Dimension Data and Cisco Systems, 2007

[2] Larson et al, 2005

[3] Klee, 2008