Natural Bounds

The idea for bounding a value based only on mathematical properties is due to Manski (1990). The term "natural bounds" is due Judea Pearl.

Consider that we are interested in the distribution of potential outcomes for patients in the Emilia trial. But we only have the distribution of outcomes from each of the trial arms. In the discussion of Frechet-Hoeffding bounds, we made an assumption that the observed distribution was a reasonable estimate of the potential outcomes distribution. Here we are not willing to make that assumption, either because we don't know about the randomization procedure used or because we have concerns about movement out of the trial by patients.

Although we don't know the potential outcome distribution for patients in the Kadcyla arm we can use the information provided in the chart above to bound the distribution.

We don't know the probability that ALL patients would live to 24 months on Kadcyla but we do know that 65% of patients in the Kadcyla arm lived to 24 months. While patients were randomly assigned either Kadcyla or the standard chemo, a large number patients left the trial and it is not clear if this was random.

Of particular concern in the participant flow chart from the Emilia trial (above) is the row labeled "Subject's Decision." This was an "open-label" trial, meaning that patient knew which drug they were being treated with. Note that more patients left the standard chemo arm than the Kadcyla arm.

The issue with these patients is that we don't know if or when they passed away. In particular, we don't know if they are still alive at the 24 month mark. If we assume that the other patients "left at random" so their unobserved outcome is washed out in the comparison, we can bound the potential outcomes.

41 of 495 is 8%.

So for 92% of patients we know that 65% were still alive at 24 months. For the remaining 8% of patients there are two possibilities. They could have all passed away prior to 24 months. If that is the case, then the survival probability at 24 months is 0.92 x 65 + 0.08 x 0 = 60%. Alternatively, they may have all survived to the 24 month mark, in which case the survival probability is 0.92 x 65 + 0.08 x 100 = 68%.

We can compare potential outcomes between the two trial arms, even though we are worried that patients didn't leave the trial at random. For the Kadcyla arm, the survival probability lies between 60% and 68%. For the standard chemo arm the survival probability lies between 46% and 57%.

Even though we we are concerned that exit from the trial was not at random, we still have that the survival probability at 24 months is higher with Kadcyla than on the standard chemo therapy.

Moreover, we can combine the natural bounds with the Frechet-Hoeffding bounds and determine that at least 3% of patients are better off taking Kadcyla than the standard chemo offered in the trial.

Note that the analysis above does not account for the uncertainty in the estimates due to sampling.

Math for the standard chemo natural bounds

55 of 496 is 11%

Lower bound = 0.89 x 52 + 0.11 x 0 = 46%

Upper bound = 0.89 x 52 + 0.11 x 100 = 57%