Bill Vesely gave a talk recently at a NASA meeting in Williamsburg, Virginia, in which he emphasized that using the binomial model for estimating reliability with binary (success/fail) data assumes that the testing protocol is 100% effective at detecting a failure. This is often unreasonable.
In our case, we are likewise assuming, for instance, that we are completely confident about whether a patient has the disease in the protocols that determine a test’s sensitivity and specificity. This would seem to be a dubious assumption. How can it be relaxed? What happens to the test performance statistics when it is relaxed?
See Bill Vesely’s slides from his talk in Williamsburg in the PDF file attached below in which he describes NASA’s solution.