If you believe Signal Detection theory, a reasonable measure of the distance between two distributions is d', which is essentially the zscore difference between the signal and noise distribution. Theoretically, d' as a measure of sensitivity requires satisfaction of assumptions that are never tested, and often may not be testable. For example, in the figure to the left, what if the signal and noise distribution have different variances. In this case, d' is misleading or could be very wrong. A bigger practical problems is as follows: d' = Z(Hit Rate)  Z(False Alarm Rate), where Z is the inverse cumulative of the normal distribution. But what happens when HR or FAR is 0 or 1? The value is either inf or +inf! And d' is undefined. This wouldn't be a problem if these values never happened, but a false alarm rate of 0 is probably the most frequently obtained false alarm rate, and it is very common to have hit rates of 0 or 1. What is a person to do? Pollack and Norman (1964) proposed a 'nonparametric' measure based on the following logic. First, let's you had direct access to the underlying distributions. For equalvariance distributions, these are are monotonically nondecreasing functions. Assume that the Receiver operating characteristic (ROC) function, formed by plotting the cumulative hit rate versus false alarm rate. Below are two examples of ROC functions from Mueller and Weidemann (2008) and Weidemann and Mueller (2008), using either (simulated) confidence ratings or (real) response time distributions to examine the shape of the underlying distributions. Pollack and Norman (1965) based their measure on the assumption that a 'proper' ROC function should be monotonically nondecreasing. Consequently, a single point specifies a family of possible ROC functions through that point. They discussed several possible ROC measures, as shown in the Figure below. For example, the area covered by polyogon I is the smallest proper ROC curve. They also argued that the largest proper ROC is either I+A1 or I+A2. Thus, based on this logic, they proposed a measure called A', which is the average between I+A1 and I+A2 (I+A1 + I + A2)/2. These are the proper ROC curves with steepest and shallowest slopes. There are a number of problems with this. First, it, like d', ignores the possibility of differences in variance between the two distributions (which can lead to 'improper' ROC curves). But more importantly, the meaning of A', if it is considered at all, is typically stated as something like "An average between the smallest and largest proper ROC curves that can pass through a point". Given the above figure, the smallest is I, but the largest is either I+A1, or I+A2, or maybe something else. Thus, the stated interpretation of A' is not really what A' measures at all. Smith (1995) pointed this out and gave the formula people should use if they mean what they really mean by A'. This was either I+A1 or I+A2, depending on where the point lay. But it turned out he made an errorthe largestarea ROC curve through a point in the upper left quadrant actually looks like the figure below. When p is in the upper right quadrant, the largest area is I+A2. When it is in the lower left quadrant, it is I+A1. But when it is in the upper left quadrant, it is neither, rather it is a polygon where the top slanted line intersects neither (0,1) or (1,1). This correction was published by Zhang and Mueller (2005), and called A. If you want to use A', you probably should use this formula instead (and cite Zhang & Mueller while you are at it). The proof is in the paper, but the formula to measure this is: Also in the paper, there is a bias measure associated with A, which is related to the slope of the ROC function at the point of the observation. The formula is as follows, with b=1 being unbiased. You can use log(b) as a symmetric bias measure. Computing A The R code for computing these values is as follows: A <function(h,f) { if(f<=.5 & h>=.5) { a < .75 + (hf)/4  f*(1h) } else if(f<=h & h<=.5) { a < .75 + (hf)/4  f/(4*h) } else { a < .75 + (hf)/4  (1h)/(4 * (1f)) } return(a) } b < function(h,f) { if(f<=.5 & h>=.5) { b <(54*h)/(1+4*f) } else if(f<=h & h<=.5) { b <(h^2+h)/(h^2+f) } else { b < ((1f)^2 + (1h))/((1f)^2 + (1f)) } return(b) } Also, I've made a spreadsheet that lets you input your hit rate and false alarm rate. Click here to download your own copy from google docs, or here is a direct download of an xlsx file. All of this is discussed in greater detail in Zhang & Mueller, 2005 (see below for citation). Should you use A'? The answer is probably no. If you are doing it because you really the average of the leastslope and greatestslope proper ROC curves through a point, then the answer is yes. If you are doing it because this is what everyone else does, then that is pseudoscience. If you are doing it because you want the nonparametric measure equivalent to the average areas of minimum and maximum ROC curves through a point, you should use A, as we describe it here. If you are using it because you have 0/1 hit rate or false alarm rate, then you should probably just use the raw hit rate, false alarm rate, or probability correctevery measure out in the tails is so sensitive to small perturbations (especially bias measures) that you cannot really trust anything.
Shane Mueller shanem@mtu.edu References: Pollack, L., & Norman, D.A. (1964). Nonparametric analysis of recognition experiments. Psychonomic Science 1, 125–126. Smith,W.D. (1995). Clarification of sensitivity measure A'. Journal of Mathematical Psychology 39, 82–89.
