Critical Decision Method

The Critical Decision Method is a type of Cognitive Task Analysis that focuses specifically on understanding challenges and unusual decisions that individuals overcome when making major decisions.

This is a multi-trial retrospective interview approach, in which three 'sweeps' are completed for an incident identified by the participant from personal experience. These sweeps can focus on 'timeline' verification to structure the account into more meaningful segments, progressive deepening to develop a contextually detailed and rich account of the incident, and 'what-if' queries to identify potential errors or alternative solutions, as well as expert-novice differences in approaching the incident.

Below in an example NDM method in action:

• Critical Decision Method

In the first sweep, I asked about timeline for the task. I asked about how long the task normally takes versus how long it takes if the robots get destroyed and if there is a certain amount of time that would lead to restarting the task entirely (a.k.a. how much time is too much time). Since his explanations provided good detail about the incident, the progressive deepening sweep appeared to happen concurrently. Next, we continued into the 'what-if' sweep. In this section I asked more generally about what if the robots were destroyed, what if more than one robot was destroyed, and the differences between his decision-making for these scenarios versus a novice player's. This sweep allowed me to gain more valuable insight into his thought process and critical decision making when these incidents occur and escalate to be more severe. Additionally, I was able to dig into his perception of how a novice player may react in these situations and how their decision making may differ, depending on knowledge and experience.

Things to remember:

• While this example highlights how progressive deepening was completed concurrently with the other sweeps, not all individuals may offer up so much detail without being probed. Be prepared to go into more detail for parts of the incident that remain unclear.

• Like most unstructured interviews, be prepared for the conversation to go to unplanned places. Dig deeper into valuable insights, especially those that offer differences between expert and novice decision-making.

Pros:

• Very useful method to gain in-depth knowledge about a specific incident and how experts react to them in a conversational situation.

• Results from this method can inform training techniques and taxonomies of information to pass on to novices in the area.

Cons:

• This is a retrospective approach asking participants to think back to a time when they experienced this incident personally. As such, they may not have a completely accurate account of what they did, depending on how long ago the incident was.

• It is helpful to be knowledgeable in the task to be able to ask appropriate questions. The CTA should allow for a base knowledge understanding, if you are a novice in the area of interest.

References

1. Hoffman, R.R., Crandell, B., & Shadbolt, N. (1998). Use of critical decision method to elicit expert knowledge: A case study in the methodology of expert task analysis. Human Factors, 40, 254-276

2. Klein, G. A., Calderwood, R., & Macgregor, D. (1989). Critical decision method for eliciting knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19, 462-472.

3. Lipshitz, R., Klein, G., Orasanu, J., & Salas, E. (2001). Taking stock of naturalistic decision making. Journal of Behavioral Decision Making, 14, 331-352. https://doi.org/10.1002/bdm.381