This website is under development
Risk analysts and others often want to compare process rates that give rise to yes-no data. For instance, analysts might ask whether one medical treatment is better or worse than another, or whether one diagnostic test is more accurate than another at detecting disease. Such comparisons can be made with sample data that identify the number of successes and failures for each of the two processes being compared. The sample data is in the form of counts of successes versus failures. Successes might be the patients with healthy outcomes after treatment versus those who had less than healthy outcomes. Of course the statistical methods to make these comparisons are very general and they could be applied in a host of settings with yes-no outcomes. For instance, in education, the question might be whether more students pass an exam under one syllabus than under another. In manufacturing quality control, the question might be whether a machine or process is producing widgets with a higher failure rate than another.
Various statistical schools of thought have devised schemes for making such comparisons, and it is remarkable that there has been no consensus among statisticians about even this most fundamental kind of comparison. The differences among the various approaches becomes especially pronounced when the data are poor. Poor data includes data sets with really small samples sizes, observations that may not be independent of one another, and data that may include cases in which neither success nor failure was clearly observed.
This website is a roundup of various discussions on this topic, and downloaded software.
Richard Lowry's VassarStats page "The Confidence Interval for the Difference Between Two Independent Proportions"
Wikipedia page for Laplace's rule of succession: choose i with larger pi, where p1=(k1+1)/(n1+2) and p2=(k2+1)/(n2+2)
A Bayesian view of Amazon Resellers
John Cook's blog post on the standad Bayesian approach
Which rating is better, mathematically speaking? | Probabilities of probabilities, part 1
3Blue1Brown's video exposition of John Cook's blog, Laplace's rule of succession and the underpinnings of the Bayesian approach
Inferences from multinomial data: learning about a bag of marbles
Peter Walley's controversial paper introducing the imprecise beta/Dirichlet model, see especially the application for ECMO babies
Comparing binomial rates with poor data [this website]
https://sites.google.com/site/comparingrateswithpoordata/
Beyond the statistician's bag of marbles
How binary sampling data informs us when we can’t make the usual assumptions
Introduction and review papers on risk analysis based on confidence structures
Download the process comparison code "megan" (rename to have extension .EXE)
Software for Microsoft Windows that compares success rates when data sets are really small
Download zip file of "megan"
Zip file of the software
Using expressions like "k out of n" to convey uncertainty about a probability
Applied Biomathematics Uncertainty
Roundup of public websites related to uncertainty from research at Applied Biomathematics
Applied Biomathematics' NIH ARRA links
Roundup of websites for Applied Biomathematics' NIH-funded project on medical counseling software