Projects‎ > ‎

Multi-lab replications

The goal of this project, in which psi labs and main stream labs will collaborate, is to find the true replication rate for the best experiment that we can come up with. Is there a psi effect that can truly be replicated?
This will be an European academic project - we will invite senior scientists of major European universities, to collaborate.Those who would like to become part of this long term research effort may contact Dick Bierman,

The program is a cautious build-up of knowledge so that we have the largest probability of obtaining a convincing result in the final replication project. 

It is based upon 4 notions:

  • N1. Academic experiments have generally used unselected university students. It has been argued that these are not the best choice. 
  • N2. Academic experiments have always used unselected experimenters. If one claims that experimenter effects might be the reason for low replication rates, the logical step is to introduce experimenter selection.
  • N3. Our QRP analyses (see Bierman, Spottiswoode & Bijl, submitted) strongly suggest that the real effect size is much smaller than the values that have been reported. (between 0.0 - 0.06 rather than 0.15-0.25). We could argue using experimental results of psi-research with artistic populations that the 0.06 is typical for unselected subjects and can become 0.20 when selection of subjects and experimenters is properly done.
  • N4. If 3. holds true then academic experiments have never had enough power. That could explain the elusive results.
Taking this into account, the long term research program we propose looks like this:

1. Development of a QRP-preventing shell to embed a key psi task in. (has been finished)
This is an essential part of the project. It will guarantee data-integrity, and will eliminate concerns about most QRPs. One can embed any (psi) task in the shell and that will cause all data to be stored in real time on an unhackable server. Each bit of data will be labeled as being relevant for explorative or confirmatory hypotheses. Nobody can change those data.The shell also helps with pre-registration of experiments.

2. Development of an efficient selection instrument for subjects and experimenters (~ 1 year)
This has started this semester at the University of Groningen. We are running several psi experiments (Bem retro-priming), and are constructing a questionnaire to see if we can come up with constructs that predict psi performance. Parallel efforts would be a welcome addition.

3. One pilot experiment using 1 & 2 and a fixed protocol that does not allow experimenter freedom. (~1.5 year)
We might run two pilot experiments, the second one in an unfriendly main stream lab. No public pre-registration, no main stream publication.

Then CONDITIONALLY: If  the pilot experiment(s) yields an effect size of ~ 0.20 because we have done a good selection of subjects and experimenters,  we will continue with 4. otherwise we will have to return to 2,  developing better selection instruments.

4. a friendly collaborative replication effort with 6-8 european psi labs using a strict protocol and the results of 1 & 2 ( 1  year)
Local supervisors could be: Chris Roe, Renaud Evrard, Harald Walach, Eva Lobach, Etzel Cardena, Caroline Watt, Jacob Jolij and Patrizio Tressoldi. If indeed we can produce an es of 0.20, and the sample size is modest (50-100). Most probably the psi task will be Bem’s 'precognition of erotic pictures'. This task lasts about 15 minutes. So the time invested by the local collaborator is limited.

Then CONDITIONALLY:  IF 4. is succesful (= 80% replication rate), we repeat the friendly collaboration but this time with pre-registration and intended publication in a psi journal:

5. pre-registration + 4.(6 months)

then CONDITIONALLY: IF 5 is succesful (= 80% replication rate) then

6. Adversarial collaborative replication effort with 5 psi labs and 5 main stream labs and publication in a major main stream journal (one year)

To give you an idea how the protocol for experimenter selection might work:

a. senior supervisor is unselected (may be a die hard skeptic)
b. supervisor selects assistant according to a standard selection instrument (resulting in someone open minded and believing in the potential success)
c. the assistant selects the (student) experimenters on the basis of another standard selection instrument (resulting in open minded experimenters with good social skills)

The senior supervisor never interacts with the experimenters that are handling the subjects.

We expect that we can go to the pilot experiment (3) near the end of 2016 and possibly run the first friendly collaboration project (4) in 2017.
We might eventually run 6. in 2019. So this is really long term thinking. But we sincerely think that drawing main stream into our research is the way to go.