A collection of pieces from professional researchers and writers in psi-related fields
Measuring the prevalence of questionable research practices
with incentives for truth-telling
An article by Leslie John, George Loewentstein, and Drazen Prelec (in press) provides evidence that questionable research practices are rife in psychology. To be published in Psychological Science, the article is a truly ironic reflection on the sometimes virulent opposition of "straight" psychologists to professional parapsychology, where a high level of scientific integrity is the norm.
Thanks to David Luke for the reference.
practices, including testing increasing numbers of participants until a
result is found, are the "steroids of scientific competition,
artificially enhancing performance". That's according to Leslie John and
her colleagues who've found evidence that such practices are worryingly
widespread among US psychologists. The results are currently in press
at the journal Psychological Science and they arrive at a time when the
psychological community is still reeling from the the fraud of a leading
social psychologist in the Netherlands. Psychology is not alone.
Previous studies have raised similar concerns about the integrity of
John's team quizzed 6,000 academic psychologists in the USA via an
anonymous electronic survey about their use of 10 questionable research
practices including: failing to report all dependent measures;
collecting more data after checking if the results are significant;
selectively reporting studies that "worked"; and falsifying data.
As well as declaring their own use of questionable research
practices and their defensibility, the participants were also asked to
estimate the proportion of other psychologists engaged in those
practices, and the proportion of those psychologists who would likely
admit to this in a survey.
For the first time in this context, the survey also incorporated an
incentive for truth-telling. Some survey respondents were told,
truthfully, that a larger charity donation would be made by the
researchers if they answered honestly (based on a comparison of a
participant's self-confessed research practices, the average rate of
confession, and averaged estimates of such practices by others). Just
over two thousand psychologists completed the survey. Comparing
psychologists who received the truth incentive vs. those that didn't
showed that it led to higher admission rates.
Averaging across the psychologists' reports of their own and others'
behaviour, the alarming results suggest that one in ten psychologists
has falsified research data, while the majority has: selectively
reported studies that "worked" (67 per cent), not reported all dependent
measures (74 per cent), continued collecting data to reach a
significant result (71 per cent), reported unexpected findings as
expected (54 per cent), and excluded data post-hoc (58 per cent).
Participants who admitted to more questionable practices tended to claim
that they were more defensible. Thirty-five per cent of respondents
said they had doubts about the integrity of their own research. Breaking
the results down by sub-discipline, relatively higher rates of
questionable practice were found among cognitive, neuroscience and
social psychologists, with fewer transgressions among clinical
John and her colleagues said that many of the iffy methods they'd
investigated were in a "grey-zone" of acceptable practice. "The inherent
ambiguity in the defensibility of research practices may lead
researchers to, however inadvertently, use this ambiguity to delude
themselves that their own dubious research practices are 'defensible'."
It's revealing that a follow-up survey that asked psychologists about
the defensibility of the questionable practices, but without asking
about their own engagement in those practices, led to far lower
John's team think the findings of their survey could help explain
the "decline effect" in psychology and other sciences - that is, the
tendency for effect sizes to decline with replications of previous
results. Perhaps this is because the original, large effect size was
obtained via questionable practices.
The current study also complements a recent paper published in
Psychological Science by Joseph Simons and colleagues that used
simulations and a real experiment to show how toying with dependent
variables, sample sizes and other factors (the kind of practices
explored in the current study) can massively increase the risk of a
false-positive finding - that is, claiming a positive effect where there
"[Questionable research practices] ... threaten research integrity
and produce unrealistically elegant results that may be difficult to
match without engaging in such practices oneself," John and her
colleagues concluded. "This can lead to a 'race to the bottom', with
questionable research begetting even more questionable research."
Leslie John, George
Loewentstein, and Drazen Prelec (In Press). Measuring the prevalence of
questionable research practices with incentives for truth-telling.
The Denier Movements
Critique Evolution, Climate Change,
And Nonlocal Consciousness
By Stephan A. Schwartz
In our culture right now we have several “denier” movements actively engaged in trying to impede the free development of science: the creationists, the climate change deniers, and the consciousness-deniers — those who cannot, or will not, consider consciousness as anything other than materialist processes. For all their lack of substance, these movements are powerful forces in the culture, with powerfully detrimental effects.
Click here to read the full article.
edited comments from a discussion of disingenuous or dishonest
skeptics, on helping media folks who hope to get a balanced view of psi
research by interviewing (such) skeptics.
The way to deal
with professional deniers is to have to hand instances of duplicity and
perfidy, and proof of the crude baldness of this community. Rupert, I
like your letter, particularly the
reference to "Communicating
Skepticism to the Public." Is it possible to obtain a copy of this
manual? It makes the point in their own words, which is always nice. In
any case, having specifics to cite is always important, and should be
accompanied by suggesting to any media person looking for balance
something like this: "When you interview (the denier) ask them
specifically what is wrong with (the researcher's) experiment, and ask
yourself, does this seem reasonable?" I also make a point of citing the
Utts-Hyman exchange, and Daryl's story about Carl Sagan (both to be
found in Opening to the Infinite), as a way of illustrating the
difference between real skepticism and professional denier calumniation.
weekend I said to a video interviewer from Europe, who asked me about
skeptics, that he should ask a skeptic how many research papers he had
published on the subject to hand, and what qualifications he had for
making his assertions. I told him that on both counts the answer would
almost certainly be either an evasion or none. Then I asked the
interviewer, would you hire a videographer who knew of cameras only by
reading other people's accounts of them? I know
the syllogism is not
complete, but he got the point because it addressed his own world,
which one should always try to do. I have about 3-4 media interactions
a month, and I never shy away from deniers, often introducing the topic
of skepticism myself sometime, perhaps during a break, in a casual way.
These kinds of interactions should always be handled with a
light touch. The previous week I had another video interview and,
during lunch, I told a true story, miming some of the parts, about
Randi at a PA conference intruding upon a conversation by sitting down
at a table with Marcello Truzzi, Arthur Templeton, and myself, and my
looking over to see the winking red light of his tape recorder when the
pocket of his jacket gaped as he leaned forward to try to steer the
conversation to our views on Hal Puthoff. I described how I had pulled
it out of his pocket passed it around commenting on its technology,
noting it was running, and directly asking Randi, "Surely
not recording gentlemen at table in their casual conversation?" And how
Randi had stammered, "Of course not, I don't know how that happened."
Then rewinding the tape, setting it on record to overdub everything he
had accumulated on the tape, saying, testing one, two, three, and
holding it front of Arthur and Marcello, who got what was happening
instantly, for them to test it, playing that back then and, when I was
sure nothing of our or any other conversation remained, turning it off
and stuffing it back into Randi's pocket. Whereupon he immediately
stood up and walked away. The journalist and his crew, who may have
played such games themselves, thought the whole thing very funny -- the
The following letter is a response by Dr. Larry Dossey to an article by Dr. Val Jones, who is a skeptic or critic of Complementary and Alternative Medicine. It demonstrates the ill-thought nature of comments presented by ideological skeptics.
The Jones article is titled “The Decades Top 5 Threats to Science in Medicine.” (sic)
(With permission from Dr. Dossey)
To: Dr. Val Jones
I suggest that we should demand rigorous scientific proof of
the efficacy and safety of ALL the therapies that we physicians employ. This includes conventional as well as
complementary/alternative medicine (CAM).
Your article implies that conventional medicine is grounded in
evidence-based research and that CAM
is not. This is grossly
overstated, and suggests that a double standard is being applied to these
Of the 2,404 conventional therapies recently assessed by the
Clinical Evidence initiative of the British Medical Journal, only 15% were
rated as clearly “beneficial,” and 22% were rated as “likely to be beneficial.”
Sixteen percent were judged to be either ineffective or harmful, unlikely to be
beneficial, or a trade-off between benefit and harm. The largest category, 47%,
was rated as “unknown effectiveness.” See: http://clinicalevidence.bmj.com/ceweb/about/knowledge.jsp.
Epidemiologist Barbara Starfield, of the Johns Hopkins
School of Medicine, reported in 2000 that around 225,000 deaths occur annually
in American hospitals due to the adverse effects of medications, infections,
and errors, making hospital care the third leading cause of death in the United
States, behind heart disease and cancer. [Barbara Starfield, “Is U.S. health
really the best in the world?” Journal of the American Medical Association. 2000; 284 (4): 483-485]. These findings have become part of the
national conversation in the United States, following the Institute of
Medicine’s startling 2000 report, “To Err is Human” [L. T. Kohn et al, To Err is Human: Building a Safer Health System. Washington, DC: National Academies Press; 2000].
These dismal findings are not new. Richard Smith, editor of the British Medical Journal, observed in
1999, “[O]nly about 15% of medical interventions are supported by solid
scientific evidence….This partly because only 1% of the articles in medical
journals are scientifically sound and partly because many treatments have never
been assessed at all”
[“Where is the wisdom…! British Medical Journal.
David A. Grimes, of UC-San Francisco School of Medicine,
writing in the Journal of the American
Medical Association in 1993, said, “Much, if not most, of contemporary
medical practice still lacks a scientific foundation” [“Technology
follies. Journal of the American Medical Association; 1993; 269(23):
As early as 1978, the Congressional Office of Technology
Assessment found that only an estimated 10 to 20% of the techniques that
physicians use are empirically proven [Assessing the Efficacy and Safety of Medical
Technologies. Washington, DC: Congressional Office
of Technology Assessment; 1978:7].
Dr. Brian Berman, of the University of Maryland School of
Medicine, examined a random subset of 159 out of 326 completed Cochrane reviews
of conventional medical practices. Overall, the “positive” and “possibly
positive” reviews totaled 38.4%, while the “no evidence of effect” and “negative
effect” totaled an alarming 61.6% [J. Ezzo, et al, “Reviewing the reviews: How strong is the evidence? How clear
the conclusions?” International J.
Technol. Access in Health Care.
2001; 17(4): 457-466].
Overwhelming evidence reveals that conventional medicine is,
on the whole, woefully unscientific. It’s fashionable and easy to deny this, but the facts say
otherwise. So, by all means, Dr.
Jones, be critical of CAM – but do not fall into a double standard. Let us ruthlessly apply science to ALL we
do as physicians. Let us challenge
ALL areas of medicine to a higher standard. On that, I’m pretty sure we can agree.
Keep up the good work.
Larry Dossey, MD
Woo Woo Is a Step Ahead of (Bad) Science
by Deepak Copra. MD
Sunday December 27, 2009
used to annoy me to be called the king of woo woo. For those who aren't
familiar with the term, "woo woo" is a derogatory reference to almost
any form of unconventional thinking, aimed by professional skeptics who
are self-appointed vigilantes dedicated to the suppression of
curiosity. I get labeled much worse things as regularly as clockwork
whenever I disagree with big fry like Richard Dawkins or smaller fry
like Michael Shermer, the Scientific American columnist and editor of
Skeptic magazine. The latest barrage of name-calling occurred after the
two of us had a spirited exchange on Larry King Live last week.
. Maybe you saw it. I was the one rolling my eyes as Shermer spoke.
Sorry about that, a spontaneous reflex of the involuntary nervous
Afterwards, however, I had an unpredictable reaction. I realized that I
would much rather expound woo woo than the kind of bad science Shermer
stands behind. He has made skepticism his personal brand, more or less,
sitting by the side of the road to denigrate "those people who believe
in spirituality, ghosts, and so on," as he says on a YouTube video. No
matter that this broad brush would tar not just the Pope, Mahatma
Gandhi, St. Teresa of Avila, Buddha, and countless scientists who
happen to recognize a reality that transcends space and time. All are
deemed irrational by the skeptical crowd. You would think that skeptics
as a class have made significant contributions to science or the
quality of life in their own right. Uh oh. No, they haven't. Their
principal job is to reinforce the great ideas of yesterday while
suppressing the great ideas of tomorrow.
Let me clear the slate with Shermer and forget the several times he has
wiggled out of a public debate he was supposedly eager to have with me.
I will ignore his recent blog in which his rebuttal of my position was
relegated to a long letter from someone who obviously didn't possess
English as a first language (would Shermer like to write a defense of
his position in Hindi? It would read just as ludicrously if Hindi isn't
his first language).
With the slate clear, I'd like to see if Shermer will accept the offer
to debate me at length on such profound questions as the following:
• Is there evidence for creativity and intelligence in the cosmos?
• What is consciousness?
• Do we have a core identity beyond our biology, mind, and ego?
• Is there life after death? Does this identity outlive the molecules through which it expresses itself?
The rules will be simple. He can argue from any basis he chooses, and I
will confine myself entirely to science. For we have reached the state
where Shermer's tired, out-of-date, utterly mediocre science is far in
arrears of the best, most open scientific thinkers -- actually, we
reached that point sixty years ago when eminent physicists like
Einstein, Wolfgang Pauli, Werner Heisenberg and Erwin Schrodinger
applied quantum theory to deep spiritual questions. The arrogance of
skeptics is both high-handed and rusty. It is high-handed because they
lump brilliant speculative thinkers into one black box known as woo
woo. It is rusty because Shermer doesn't even bother to keep up with
the latest findings in neuroscience, medicine, genetics, physics, and
evolutionary biology. All of these fields have opened fascinating new
ground for speculation and imagination. But the king of pooh-pooh is
too busy chasing down imaginary woo woo.
Skeptics feel that they have won the high ground in matters concerning
consciousness, mind, the origins of life, evolutionary theory, and
brain science. This is far from the case. What they cling to is
nineteenth- century materialism, packaged with a screeching hysteria
about God and religion that is so passé it has become quaint. To
suggest that Darwinian theory is incomplete and full of unproven
hypotheses, causes Shermer, who takes Darwin as purely as a
fundamentalist takes scripture, to see God everywhere in the enemy camp.
How silly. Shermer is a former Christian fundamentalist who is now a
fundamentalist about materialism; fundamentalists must have an absolute
to believe in. Thus he forces himself into a corner, declaring that all
spirituality is bogus, that the sense of self is an illusion, that the
soul is ipso facto a fraud, that mind has no existence except in the
brain, that intelligence emerged only when evolution, guided by random
mutations, developed the cerebral cortex, that nothing invisible can be
real compared to solid objects, and that any thought which ventures
beyond the five senses for evidence must be dismissed without question.
I won't go into detail about the absurdity of such rigid thinking.
However, the impulse behind dogmatic materialism seems intended to
flatten one's opponents so thoroughly that through scorn and arrogance
they must admit defeat, conceding that science is the complete
refutation of all preceding religion, spirituality, psychology, myth,
and philosophy -- in other words, any mode of gaining knowledge that
arch materialism doesn't countenance.
I've baited this post with a few barbs to see if Shermer can be goaded
into an actual public debate. I have avoided his and his follower's
underhanded methods, whereby an opponent is attacked ad hominem as an
idiot, moron, and other choice epithets that in his world are the
mainstays of rational argument. And the point of such a debate? To
further public knowledge about the actual frontiers of science, which
has always depended on wonder, awe, imagination, and speculation. Petty
science of the Shermer brand scorns such things, but the greatest
discoveries have been anchored on them.
If you are tempted to think that I have taken the weaker side and that
materialism long ago won this debate, let me end with a piece of
utterly nonsensical woo woo:
"Nobody understands how decisions are made or how imagination is set
free. What consciousness consists of, or how it should be defined, is
equally puzzling. Despite the marvelous success of neuroscience in the
past century, we seem as far from understanding cognitive processes as
we were a century ago."
That isn't a quote from "one of those people who believe in
spirituality, ghosts, and so on." It's from Sir John Maddox, former
editor-in-chief of the renowned scientific journal Nature, writing in
1999. I can't wait for Shermer to call him an idiot and a moron. Don't
worry, he won't. He'll find an artful way of slithering to higher
ground where all the other skeptics are huddled.