A' (A-prime) as sensitivity in signal detection (and why to use A instead)



If you believe Signal Detection theory, a reasonable measure of the distance between two distributions is d', which is essentially the z-score difference between the signal and noise distribution. Theoretically, d' as a measure of sensitivity requires satisfaction of assumptions that are never tested, and often may not be testable.  For example, in the figure to the left, what if the signal and noise distribution have different variances.  In this case, d' is misleading or could be very wrong.

A bigger practical problems is as follows:
d' = Z(Hit Rate) - Z(False Alarm Rate), where Z is the inverse cumulative of the normal distribution.  But what happens when HR or FAR is 0 or 1?  The value is either -inf or +inf!  And d' is undefined.

This wouldn't be  a problem if these values never happened, but a false alarm rate of 0 is probably the most frequently obtained false alarm rate, and it is very common to have hit rates of 0 or 1.  What is a person to
do?


Pollack and Norman (1964) proposed a 'non-parametric' measure based on the following logic.  First, let's you had direct access to the underlying distributions.  For equal-variance distributions, these are are monotonically non-decreasing functions.
  Assume that the Receiver operating characteristic (ROC) function, formed by plotting the cumulative hit rate versus false alarm rate.  Below are two examples of ROC functions from Mueller and Weidemann (2008) and Weidemann and Mueller (2008), using either (simulated) confidence ratings or (real) response time distributions to examine the shape of the underlying distributions.

Confidence ROC function
























Pollack and Norman (1965) based their measure on the assumption that a 'proper' ROC function should be monotonically non-decreasing. Consequently, a single point specifies a family of possible ROC functions through that point.  They discussed several possible ROC measures, as shown in the Figure below.  For example, the area covered by polyogon I is the smallest proper ROC curve.  They also argued that the  largest proper ROC is either I+A1 or I+A2.  Thus, based on this logic, they proposed a measure called A', which is the average between I+A1 and I+A2 (I+A1 + I + A2)/2.  These are the proper ROC curves with steepest and shallowest slopes.

T
here are a number of problems with this.  First, it, like d', ignores the possibility of differenc
es in variance between the two distributions (which can lead to 'improper' ROC curves).  But more importantly, the meaning of A', if it is considered at all, is typically stated as something like "An average between the smallest and largest proper ROC curves that can pass through a point".  Given the above figure, the smallest is I, but the largest is either I+A1, or I+A2, or maybe something else. Thus, the stated interpretation of A' is not really what A' measures at all.

Smith (1995) pointed this out and gave the formula people should use if they mean what they really mean by A'.  This was either I+A1 or I+A2, depending on where the point lay.  But it turned out he made an error--the largest-area ROC curve through a point in the upper left quadrant actually looks like the figure below.  When p is in the upper right quadrant, the largest area is I+A2.  When it is in the lower left quadrant, it is I+A1. But when it is in the upper left quadrant, it is neither, rather it is a polygon where the top slanted line intersects neither (0,1) or (1,1).



This correction was published by Zhang and Mueller (2005), and called A. If you want to use A', you probably should use this formula instead (and cite Zhang & Mueller while you are at it). The proof is in the paper, but the formula to measure this is:



Also in the paper, there is a bias measure associated with A, which is related to the slope of the ROC function at the point of the observation.  The formula is as follows, with b=1 being unbiased.  You can use log(b) as a symmetric bias measure.



Computing A

Computing A is pretty easy.  In the frame below, you can enter your hit rate and false alarm rate (as values between 0 and 1) and it will compute A, b, and d-prime.  Also, it creates a figure that shows the least-area and max-area proper ROC curves through your observed point, to better understand the range of possible curves.  The page is available standalone at http://mueller-linux.bio.mtu.edu:3838/aprime/

Zhang and Mueller Sensitivity Calculator



This uses R  and shiny to for the web app.  The R code for computing these values is as follows:


A <-function(h,f)
{
  if(f<=.5 & h>=.5)
  {
    a <- .75 + (h-f)/4 - f*(1-h)
  } else if(f<=h & h<=.5)
  {
    a <- .75 + (h-f)/4 - f/(4*h)
  } else {
    a <- .75 + (h-f)/4 - (1-h)/(4 * (1-f))
  }
  return(a)
  }

b <- function(h,f)
{
  if(f<=.5 & h>=.5)
  {
    b <-(5-4*h)/(1+4*f)
  } else if(f<=h & h<=.5)
  {
    b <-(h^2+h)/(h^2+f)
  } else {
    b <- ((1-f)^2 + (1-h))/((1-f)^2 + (1-f))
  }
  return(b)
}


Also, I've made a
spreadsheet that lets you input your hit rate and false alarm rate. Click here to download your own copy.

A-sdt


All of this is discussed in greater detail in Zhang & Mueller, 2005 (see below for citation). Should you use A'?  The answer is probably no.  If you are doing it because you really the average of the least-slope and greatest-slope proper ROC curves through a point, then the answer is yes.  If you are doing it because this is what everyone else does, then that is pseudoscience.  If you are doing it because you want the non-parametric measure equivalent to the average areas of minimum and maximum ROC curves through a point, you should use A, as we describe it here.  If you are using it because you have 0/1 hit rate or false alarm rate, then you should probably just use the raw hit rate, false alarm rate, or probability correct--every measure out in the tails is so sensitive to small perturbations (especially bias measures) that you cannot really trust anything.



Shane Mueller
shanem@mtu.edu

References:

Pollack, L., & Norman, D.A. (1964). Non-parametric analysis of recognition experiments. Psychonomic Science 1, 125–126.
Smith,W.D. (1995). Clarification of sensitivity measure A'. Journal of Mathematical Psychology 39, 82–89.