Analysis of Competing Hypotheses

The analysis of competing hypotheses (ACH) provides an unbiased methodology for evaluating multiple competing hypotheses for observed data. It was developed by Richards (Dick) J. Heuer, Jr., ACH is used by analysts in various fields who make judgments that entail a high risk of error in reasoning. Abductive reasoning is an earlier concept with similarities to ACH.

Heuer outlines the ACH process in considerable depth in his book, Psychology of Intelligence Analysis.[1] It consists of the following steps:

  1. Hypothesis – The first step of the process is to identify all potential hypotheses, preferably using a group of analysts with different perspectives to brainstorm the possibilities. The process discourages the analyst from choosing one "likely" hypothesis and using evidence to prove its accuracy. Cognitive bias is minimized when all possible hypotheses are considered.[1]
  2. Evidence – The analyst then lists evidence and arguments (including assumptions and logical deductions) for and against each hypothesis.[1]
  3. Diagnostics – Using a matrix, the analyst applies evidence against each hypothesis in an attempt to disprove as many theories as possible. Some evidence will have greater "diagnosticity" than other evidence — that is, some will be more helpful in judging the relative likelihood of alternative hypotheses. This step is the most important, according to Heuer. Instead of looking at one hypothesis and all the evidence ("working down" the matrix), the analyst is encouraged to consider one piece of evidence at a time, and examine it against all possible hypotheses ("working across" the matrix).[1]
  4. Refinement – The analyst reviews the findings, identifies any gaps, and collects any additional evidence needed to refute as many of the remaining hypotheses as possible.[1]
  5. Inconsistency – The analyst then seeks to draw tentative conclusions about the relative likelihood of each hypothesis. Less consistency implies a lower likelihood. The least consistent hypotheses are eliminated. While the matrix generates a definitive mathematical total for each hypothesis, the analyst must use their judgment to make the final conclusion. The result of the ACH analysis itself must not overrule analysts' own judgments.
  6. Sensitivity – The analyst tests the conclusions using sensitivity analysis, which weighs how the conclusion would be affected if key evidence or arguments were wrong, misleading, or subject to different interpretations. The validity of key evidence and the consistency of important arguments are double-checked to assure the soundness of the conclusion's linchpins and drivers.[1]
  7. Conclusions and evaluation – Finally, the analyst provides the decisionmaker with his or her conclusions, as well as a summary of alternatives that were considered and why they were rejected. The analyst also identifies milestones in the process that can serve as indicators in future analyses.[1]

Other approaches

Work by Pope and Jøsang uses subjective logic, a formal mathematical methodology that explicitly deals with uncertainty.[14] This methodology forms the basis of the Sheba technology that is used in Veriluma's intelligence assessment software.

A few online and downloadable tools help automate the ACH process. These programs leave a visual trail of evidence and allow the analyst to weigh evidence. PARC ACH 2.0 was developed by Palo Alto Research Center (PARC). Another useful program is the Decision Command software created by Dr. Willard Zangwill. SSS Research, Inc. is an analytic research firm that created DECIDE. DECIDE not only allows analysts to manipulate ACH, but it provides multiple visualization products. Competing Hypotheses is an open source ACH implementation. Source: http://en.wikipedia.org/wiki/Analysis_of_competing_hypotheses

https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/art11.html

  • Chapter 8

Analysis of Competing Hypotheses

Analysis of competing hypotheses, sometimes abbreviated ACH, is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.

ACH is an eight-step procedure grounded in basic insights from cognitive psychology, decision analysis, and the scientific method. It is a surprisingly effective, proven process that helps analysts avoid common analytic pitfalls. Because of its thoroughness, it is particularly appropriate for controversial issues when analysts want to leave an audit trail to show what they considered and how they arrived at their judgment.

*******************

When working on difficult intelligence issues, analysts are, in effect, choosing among several alternative hypotheses. Which of several possible explanations is the correct one? Which of several possible outcomes is the most likely one? As previously noted, this book uses the term "hypothesis" in its broadest sense as a potential explanation or conclusion that is to be tested by collecting and presenting evidence.

Analysis of competing hypotheses (ACH) requires an analyst to explicitly identify all the reasonable alternatives and have them compete against each other for the analyst's favor, rather than evaluating their plausibility one at a time.

The way most analysts go about their business is to pick out what they suspect intuitively is the most likely answer, then look at the available information from the point of view of whether or not it supports this answer. If the evidence seems to support the favorite hypothesis, analysts pat themselves on the back ("See, I knew it all along!") and look no further. If it does not, they either reject the evidence as misleading or develop another hypothesis and go through the same procedure again. Decision analysts call this a satisficing strategy. (See Chapter 4, Strategies for Analytical Judgment.) Satisficing means picking the first solution that seems satisfactory, rather than going through all the possibilities to identify the very best solution. There may be several seemingly satisfactory solutions, but there is only one best solution.

Chapter 4 discussed the weaknesses in this approach.

Summary and Conclusion

Three key elements distinguish analysis of competing hypotheses from conventional intuitive analysis.

  • Analysis starts with a full set of alternative possibilities, rather than with a most likely alternative for which the analyst seeks confirmation. This ensures that alternative hypotheses receive equal treatment and a fair shake.
  • Analysis identifies and emphasizes the few items of evidence or assumptions that have the greatest diagnostic value in judging the relative likelihood of the alternative hypotheses. In conventional intuitive analysis, the fact that key evidence may also be consistent with alternative hypotheses is rarely considered explicitly and often ignored.
  • Analysis of competing hypotheses involves seeking evidence to refute hypotheses. The most probable hypothesis is usually the one with the least evidence against it, not the one with the most evidence for it. Conventional analysis generally entails looking for evidence to confirm a favored hypothesis.