Enter the statisticians

Professor John Copas is Professor of Statistics at Warwick University, and an acknowledged expert in criminological statistics. With a colleague (Copas and Jackson, 2004) he suggested a “bound for publication” hypothesis, which goes something like this:
  1. Most evaluation studies are of small samples.
  2. Such studies are notoriously prone to fluke effects.
  3. Clinicians believe in treatment, and will conclude that an unsuccessful result means they are at fault themselves.
  4. Unsuccessful studies will not be reported, as clinicians don’t want to (as they see it) advertise their own incompetence.
  5. Thus, a spurious “treatment effect” will be produced because of the suppression of those studies that failed to show it; this will mainly involve small studies.
Copas has demonstrated this effect to occur in other fields, such as medicine; this has resulted in some claims for medical treatments being heavily toned down. In a personal communication, he gave the opinion that few if any studies were of sufficient quality to be entered into a meta-analysis which is usually used to combine the results of a group of evaluation studies.

The bound for publication hypothesis gives us a testable experimental hypothesis:

  1. If Copas and Jackson are right, the smallest samples should be responsible for the largest “treatment effects”.
  2. If treatment is genuinely effective, then the larger the sample, the larger the effect.

This hypothesis has not been explicitly tested, but in 2005 a meta-analysis by Lösel and Schmucker provided data which enable it to be examined. Their research is described in the meta-analysis section, and the data from it examined in the bound for publication bias section.