Helping or Hurting?
A chart similar to this is presented in the NEJM report on the Phase 3 trials comparing the immunotherapies Opdivo (N) and Yervoy (I) to chemotherapy (Ch). The trial looked at the effect of these drugs on survival of patients with non-small cell lung cancer (NSCLC). The report concludes: "First-line treatment with nivolumab plus ipilimumab resulted in a longer duration of overall survival than did chemotherapy in patients with NSCLC, independent of the PD-L1 expression level. "
The problem is that this is not what the data says. In fact the data says that for some people the combination of the two immunotherapies results in SHORTER duration of overall survival than chemotherapy. We know this because the survival curves cross. See that the two lines switch places around 7 months?
I don't have access to the original trial data so the chart above is created with a synthetic data set that has more or less the same probabilities at each 3 month mark. Note that my synthetic patients do no leave the trial other than through death. With this synthetic data we can bound the distribution of the treatment effect. We can bound distribution of the difference in survival between the immunotherapy combination and chemo. These are the Kolmogorov bounds.
The figure shows that at least 5% of the synthetic patients live longer on chemotherapy than they do on the immunotherapy combination. To see this look at the 0 days and the lower bound line. See how that line crosses 0 at a point above 0? That crossing at 0, tells us that survival benefit of the combination is negative for at least some synthetic patients. How many synthetic patients? We don't know. But we do it is between 5% and about 90%. The upper bound line cross 0 at around 0.9.
Do some people live longer on the combination? Sure. We know that at least 10% of synthetic patients live longer on the immunotherapy combination and up to 95% may live longer on the combination.
How much longer to people live? That's trickier. At least a few percentage of people may live longer than 12 months on the combination, but we can't rule out that no one lives 12 months longer. Note that the lighter lines represent the Kolmogorov bounds from bootstrapped pseudo-synthetic samples. We can learn that no more than about 70% of synthetic patients live longer than 12 months on the combination. We can also say that no more than 60% of patients live 12 months or more on chemo relative to the combination, but it could be as few as zero.
# Nivo.R
# Data as a csv: https://drive.google.com/file/d/1LgR6rSpDwVAT2MBPgcAN1QdcO_WkeQBT/view?usp=sharing
# This is just my guess of the probabilities for the two trial arms at each quarter.
# Set working director to file location or create path variable.
x <- read.csv("ch227.csv", as.is = TRUE)
# Create synthetic data
N <- round(runif(round(400*(1 - x[1,2])), 1,
30*x[1,1]))
Ch <- round(runif(round(400*(1 - x[1,3])), 1,
30*x[1,1]))
for (i in 2:14) {
N <- c(N, round(runif(round(400*(x[i-1,2] - x[i,2])),
30*x[i-1,1], 30*x[i,1])))
Ch <- c(Ch, round(runif(round(400*(x[i-1,3] - x[i,3])),
30*x[i-1,1], 30*x[i,1])))
}
N <- c(N, rep(40*30, 400 - length(N)))
Ch <- c(Ch, rep(40*30, 400 - length(Ch)))
# Create the probabilities for each arm.
F_n <- F_c <- matrix(NA,39,1)
for (i in 1:39) {
F_n[i] <- mean(N < i*30)
F_c[i] <- mean(Ch < i*30)
}
# Create the first plot.
plot(1:39, 1 - F_n, ylim=c(0,1), type="l", lwd=3,
col="green", axes=FALSE,
ylab="Patients Who Survived (%)",
xlab="months")
lines(1:39, 1 - F_c, lwd=3, lty=2, col="gray")
abline(v=c(12,24), lty=2)
axis(1, at=c(3,6,9,12,15,18,21,24,27,30,33,36,39),
labels = NULL)
axis(2)
text(35,0.42,"N+I")
text(35,0.15, "Ch")
# Kolmogorov bounds
FL <- function(b, y1, y0) {
f <- function(x) -(mean(y1 < x) - mean(y0 < x - b))
# note the negative sign as we are maximizing (Remember to put it back!)
a <- optimize(f, c(min(y1,y0),max(y1,y0)))
return(max(-a$objective,0))
}
FU <- function(b, y1, y0) {
f <- function(x) mean(y1 < x) - mean(y0 < x - b)
a <- optimize(f, c(min(y1,y0), max(y1,y0)))
return(1 + min(a$objective,0))
}
Kolmogorov <- function(x, y1, y0) {
# x is an ordered vector
x <- sort(x)
temp <- matrix(NA,length(x),2)
temp[1,1] <- FL(x[1], y1, y0)
temp[1,2] <- FU(x[1], y1, y0)
for (i in 2:length(x)) {
temp[i,1] <- max(c(temp[i-1,1], FL(x[i], y1, y0)))
temp[i,2] <- max(c(temp[i-1,2], FU(x[i], y1, y0)))
}
return(temp)
}
# create second plot
K <- 200
min_diff <- min(Ch) - max(N)
max_diff <- max(Ch) - min(N)
del_diff <- (max_diff - min_diff)/K
y_K <- min_diff + c(1:K)*del_diff
a <- Kolmogorov(y_K, N, Ch)
plot(y_K, a[,1], type="l", ylim=c(0,1), xlim=c(-1000,1000), lty=2, lwd=3, axes = FALSE,
xlab="Survival Difference on N+I (vs Ch) in Days",
ylab="Probability")
lines(y_K, a[,2], col="red", lty=3,lwd=3)
J = 100
for (j in 1:J) {
index_j <- round(runif(400, 1, 400))
a_j <- Kolmogorov(y_K, N[index_j], Ch[index_j])
lines(y_K, a_j[,1], lty=2, col="gray")
lines(y_K, a_j[,2], lty=3, col="pink")
}
lines(y_K, a[,1], col="black", lty=2,lwd=3)
lines(y_K, a[,2], col="red", lty=3,lwd=3)
abline(v=c(0,30*12,-30*12),lty=2,lwd=3)
axis(2)
axis(1, at=c(-36*30,-24*30,-12*30,0,12*30,24*30,36*30))
text(-700,0.7,"Upper Bound", col="red")
text(700,0.4, "Lower Bound")