Post date: Aug 14, 2015 12:13:16 AM
<<Scott wrote to Min-ge Xie>>
Min-ge::
It has been a long time since we have spoken. I hope you are well. I wanted to run something by you to see what you think.
It seems that the "modern" definition of CDs which I guess is due to Schweder & Hjort is essentially saying if we plot the (random) fraction of a CD that is less than the true value the CD is estimating, the plot should be U(0,1). Around here, we've been calling these "Singh plots" in honor of your colleague. I haven't found a particular passage in which Kesar Singh suggests making these plots, but they seem to be implied.
These plots give us a convenient visual means to check the statistical performance of a CD. If a confidence distribution is performing according to its definition, then its Singh plot will look like the standard unit uniform distribution. This will be a cumulative distribution from (0,0) to (1,1). If an implementation of a confidence distribution has a Singh plot which is the standard uniform, then it seems that we can infer that all of the confidence intervals it encodes will have their respective nominal coverages. Of course, the plot is specific for a given set of true values for the estimated parameter and any nuisance parameters.
The analog of a Singh plot for c-boxes is an imprecise structure that straddles the uniform distribution. Using the red R code (below), I made a Singh graph for the binomial rate c-box (depicted below and in the attached PNG file). The standard uniform distribution is shown as a red line. The wiggly curves characterize the performance of c-boxes with sample size of 8 binomial deviates randomly sampled from a true binomial rate of p=0.2 with N=20 trials. The blue curve represents the performance of the right edge of the c-box. It is the distribution of the proportion of how much of the c-box’s right edge is smaller than the true binomial rate. The black curve represents the performance of the leftedge of the c-box. It’s the distribution of the proportion of how much of the c-box’s left edge is smaller than the true binomial rate. I guess the blue curve should never be below the red line, and the black curve should never be above the red line. One hopes that they will be as tight as possible around U(0,1). In this case, because the data are discrete, the estimation cannot be perfect, but the saw-tooth structure is perhaps as good as one can get.
Comparable R code for similarly demonstrating the performance of a precise CD for the normal mean appears in blue below.
Do you think I've understood these plots correctly? Do you agree that it is reasonable to name them after Professor Singh?
Best regards,
Scott
Scott Ferson
Applied Biomathematics
# demonstrate how the c-box for the binomial rate generalizes a confidence distribution (CD)
many <- 5000
lots <- 10000
singh <- function(n,truetheta,cd,rsamples,plotting=TRUE) {
u <- w <- array(0, lots)
for (i in 1:lots) {
d <- cd(rsamples(n,truetheta),truetheta[[-1]]) < truetheta[[1]]
u[i] <- sum(d[1:many]) / many
w[i] <- sum(d[(many+1):(2*many)]) / many
}
plot(c(0,1),c(0,1),col='white',lwd=4, type='l', xlab='',ylab='')
lines(sort(u),1:lots/lots,type = 's')
lines(sort(w),1:lots/lots,type = 's',col='blue')
return(invisible(list(u=u,w=w)))
}
r.binomN <- function(n,p) rbinom(n,p[[2]],p[[1]])
cd.binomN <- function(x,N) {n <- length(x); k <- sum(x); return(c(beta(k, n*N-k+1), beta(k+1, n*N-k)))}
beta <- function(v,w) if (v==0) rep(0,many) else if (w==0) rep(1,many) else rbeta(many, v, w)
singh(n = 8, truetheta = c(0.2,20), cd.binomN, r.binomN)
# Gosset's scaled t-distribution is a confidence distribution for the normal mean
cd = function(z) {n = length(z); mean(z) + sd(z) * rt(many,n-1) / sqrt(n)}
some = 5000
many = 5000
n = 8
mu = runif(1,-100,100)
sg = runif(1,0,100)
cat(n,' random samples from normal(',mu,',',sg,')\n',sep='')
u = array(0, some)
for(i in 1:some) {
x = rnorm( n , mu , sg )
u[i] = sum( cd(x) < mu )/many
}
plot( c(0,1) , c(0,1) , type='l' , col='red',lwd=4)
lines(sort(u), 1:some/some,type = 's',lwd=2)
<<Min-ge Xie reponded>>
Scott,
Good to hear from you. Based on what I know, the `modern' definition was formulated by Kesar Singh and Tore Scheweder, independently, around the same time between 2000 - 2001. So calling it a "Singh plot" or "Singh-Schewder plot" is fitting. (If you like to know more details, I can provide you my account of the early developments of the two groups based on what I know here at Rutgers and what I have learned from Tore.)
You are correct that such a U(0,1) plot is implied. We used it a lot, e.g., I used such a plot to 'validate' our Theorem 4.1 in Singh et al (2005) before we actually found a mathematical proof. But I cannot recall at the moment whether such a plot has appeared in any publications other than Liu et al (2014, http://www.tandfonline.com/doi/abs/10.1080/01621459.2014.946318?journalCode=uasa20) or my 2011 NSF grant proposal (http://www.nsf.gov/awardsearch/showAward?AWD_ID=1107012&HistoricalAwards). Also, I won't be surprised, if you can find such a U(0,1) plot (or some variation) in Singh's earlier publications (in 1990s or earlier) on bootstrap p-values (co-authored with Regina Liu or Bob Berk).
I like your c-plot addition to the U(0,1) plot. It is pretty nice. Thank you for sharing.
Best regards,
Minge
<snip>
*N*=20 trials. The blue curve represents the performance of the *right *edge
of the c-box. It is the distribution of the proportion of how much of the
c-box’s right edge is smaller than the true binomial rate. The black curve
represents the performance of the *left *edge of the c-box. It’s the
<snip>