Cases and Comments

A collection of pieces from professional researchers and writers in psi-related fields

(Please note: under the Cases and Comments heading there are several other topics with their own pages.)




Measuring the prevalence of questionable research practices
with incentives for truth-telling


An article by Leslie John, George Loewentstein, and Drazen Prelec (in press) provides evidence that questionable research practices are rife in psychology. To be published in Psychological Science, the article is a truly ironic reflection on the sometimes virulent opposition of "straight" psychologists to professional parapsychology, where a high level of scientific integrity is the norm. Thanks to David Luke for the reference.

----------------------------------------

Questionable research practices, including testing increasing numbers of participants until a result is found, are the "steroids of scientific competition, artificially enhancing performance". That's according to Leslie John and her colleagues who've found evidence that such practices are worryingly widespread among US psychologists. The results are currently in press at the journal Psychological Science and they arrive at a time when the psychological community is still reeling from the the fraud of a leading social psychologist in the Netherlands. Psychology is not alone. Previous studies have raised similar concerns about the integrity of medical research.

John's team quizzed 6,000 academic psychologists in the USA via an anonymous electronic survey about their use of 10 questionable research practices including: failing to report all dependent measures; collecting more data after checking if the results are significant; selectively reporting studies that "worked"; and falsifying data.

As well as declaring their own use of questionable research practices and their defensibility, the participants were also asked to estimate the proportion of other psychologists engaged in those practices, and the proportion of those psychologists who would likely admit to this in a survey.

For the first time in this context, the survey also incorporated an incentive for truth-telling. Some survey respondents were told, truthfully, that a larger charity donation would be made by the researchers if they answered honestly (based on a comparison of a participant's self-confessed research practices, the average rate of confession, and averaged estimates of such practices by others). Just over two thousand psychologists completed the survey. Comparing psychologists who received the truth incentive vs. those that didn't showed that it led to higher admission rates.

Averaging across the psychologists' reports of their own and others' behaviour, the alarming results suggest that one in ten psychologists has falsified research data, while the majority has: selectively reported studies that "worked" (67 per cent), not reported all dependent measures (74 per cent), continued collecting data to reach a significant result (71 per cent), reported unexpected findings as expected (54 per cent), and excluded data post-hoc (58 per cent). Participants who admitted to more questionable practices tended to claim that they were more defensible. Thirty-five per cent of respondents said they had doubts about the integrity of their own research. Breaking the results down by sub-discipline, relatively higher rates of questionable practice were found among cognitive, neuroscience and social psychologists, with fewer transgressions among clinical psychologists.

John and her colleagues said that many of the iffy methods they'd investigated were in a "grey-zone" of acceptable practice. "The inherent ambiguity in the defensibility of research practices may lead researchers to, however inadvertently, use this ambiguity to delude themselves that their own dubious research practices are 'defensible'." It's revealing that a follow-up survey that asked psychologists about the defensibility of the questionable practices, but without asking about their own engagement in those practices, led to far lower defensibility ratings.

John's team think the findings of their survey could help explain the "decline effect" in psychology and other sciences - that is, the tendency for effect sizes to decline with replications of previous results. Perhaps this is because the original, large effect size was obtained via questionable practices.

The current study also complements a recent paper published in Psychological Science by Joseph Simons and colleagues that used simulations and a real experiment to show how toying with dependent variables, sample sizes and other factors (the kind of practices explored in the current study) can massively increase the risk of a false-positive finding - that is, claiming a positive effect where there is none.

"[Questionable research practices] ... threaten research integrity and produce unrealistically elegant results that may be difficult to match without engaging in such practices oneself," John and her colleagues concluded. "This can lead to a 'race to the bottom', with questionable research begetting even more questionable research."
_________________________________

Reference: Leslie John, George Loewentstein, and Drazen Prelec (In Press). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science







The Denier Movements

Critique Evolution, Climate Change,

And Nonlocal Consciousness

By Stephan A. Schwartz


In our culture right now we have several “denier” movements actively engaged in trying to impede the free development of science: the creationists, the climate change deniers, and the consciousness-deniers — those who cannot, or will not, consider consciousness as anything other than materialist processes. For all their lack of substance, these movements are powerful forces in the culture, with powerfully detrimental effects.

Click here to read the full article.


Stephan Schwartz

[Lightly edited comments from a discussion of disingenuous or dishonest skeptics, on helping media folks who hope to get a balanced view of psi research by interviewing (such) skeptics.]

The way to deal with professional deniers is to have to hand instances of duplicity and perfidy, and proof of the crude baldness of this community. Rupert, I like your letter, particularly the reference to "Communicating Skepticism to the Public." Is it possible to obtain a copy of this manual? It makes the point in their own words, which is always nice. In any case, having specifics to cite is always important, and should be accompanied by suggesting to any media person looking for balance something like this: "When you interview (the denier) ask them specifically what is wrong with (the researcher's) experiment, and ask yourself, does this seem reasonable?" I also make a point of citing the Utts-Hyman exchange, and Daryl's story about Carl Sagan (both to be found in Opening to the Infinite), as a way of illustrating the difference between real skepticism and professional denier calumniation.

This weekend I said to a video interviewer from Europe, who asked me about skeptics, that he should ask a skeptic how many research papers he had published on the subject to hand, and what qualifications he had for making his assertions. I told him that on both counts the answer would almost certainly be either an evasion or none. Then I asked the interviewer, would you hire a videographer who knew of cameras only by reading other people's accounts of them? I know the syllogism is not complete, but he got the point because it addressed his own world, which one should always try to do. I have about 3-4 media interactions a month, and I never shy away from deniers, often introducing the topic of skepticism myself sometime, perhaps during a break, in a casual way.

These kinds of interactions should always be handled with a light touch. The previous week I had another video interview and, during lunch, I told a true story, miming some of the parts, about Randi at a PA conference intruding upon a conversation by sitting down at a table with Marcello Truzzi, Arthur Templeton, and myself, and my looking over to see the winking red light of his tape recorder when the pocket of his jacket gaped as he leaned forward to try to steer the conversation to our views on Hal Puthoff. I described how I had pulled it out of his pocket passed it around commenting on its technology, noting it was running, and directly asking Randi, "Surely
you are not recording gentlemen at table in their casual conversation?" And how Randi had stammered, "Of course not, I don't know how that happened." Then rewinding the tape, setting it on record to overdub everything he had accumulated on the tape, saying, testing one, two, three, and holding it front of Arthur and Marcello, who got what was happening instantly, for them to test it, playing that back then and, when I was sure nothing of our or any other conversation remained, turning it off and stuffing it back into Randi's pocket. Whereupon he immediately stood up and walked away. The journalist and his crew, who may have played such games themselves, thought the whole thing very funny -- the biter bitten.



Larry Dossey

The following letter is a response by Dr. Larry Dossey to an article by  Dr. Val Jones, who is a skeptic or critic of Complementary and Alternative Medicine. It demonstrates the ill-thought nature of comments presented by ideological skeptics.

The Jones article is titled “The Decades Top 5 Threats to Science in Medicine.” (sic)

See: http://getbetterhealth.com/the-decades-top-5-threats-to-science-in-medicine/2009.12.31.

------

(With permission from Dr. Dossey)

To:  Dr. Val Jones 

I suggest that we should demand rigorous scientific proof of the efficacy and safety of ALL the therapies that we physicians employ.  This includes conventional as well as complementary/alternative medicine (CAM). Your article implies that conventional medicine is grounded in evidence-based research and that CAM is not. This is grossly overstated, and suggests that a double standard is being applied to these fields. Consider:

Of the 2,404 conventional therapies recently assessed by the Clinical Evidence initiative of the British Medical Journal, only 15% were rated as clearly “beneficial,” and 22% were rated as “likely to be beneficial.” Sixteen percent were judged to be either ineffective or harmful, unlikely to be beneficial, or a trade-off between benefit and harm. The largest category, 47%, was rated as “unknown effectiveness.” See: http://clinicalevidence.bmj.com/ceweb/about/knowledge.jsp.

Epidemiologist Barbara Starfield, of the Johns Hopkins School of Medicine, reported in 2000 that around 225,000 deaths occur annually in American hospitals due to the adverse effects of medications, infections, and errors, making hospital care the third leading cause of death in the United States, behind heart disease and cancer. [Barbara Starfield, “Is U.S. health really the best in the world?”  Journal of the American Medical Association.  2000; 284 (4): 483-485].  These findings have become part of the national conversation in the United States, following the Institute of Medicine’s startling 2000 report, “To Err is Human” [L. T. Kohn et al, To Err is Human:  Building a Safer Health System. Washington, DC: National Academies Press; 2000].

These dismal findings are not new. Richard Smith, editor of the British Medical Journal, observed in 1999, “[O]nly about 15% of medical interventions are supported by solid scientific evidence….This is partly because only 1% of the articles in medical journals are scientifically sound and partly because many treatments have never been assessed at all” [“Where is the wisdom…!  British Medical Journal. 1991;303:798-799]. 

David A. Grimes, of UC-San Francisco School of Medicine, writing in the Journal of the American Medical Association in 1993, said, “Much, if not most, of contemporary medical practice still lacks a scientific foundation” [“Technology follies.  Journal of the American Medical Association; 1993; 269(23): 3030-3033]. 

As early as 1978, the Congressional Office of Technology Assessment found that only an estimated 10 to 20% of the techniques that physicians use are empirically proven  [Assessing the Efficacy and Safety of Medical Technologies.  Washington, DC:     Congressional Office of Technology Assessment; 1978:7].

Dr. Brian Berman, of the University of Maryland School of Medicine, examined a random subset of 159 out of 326 completed Cochrane reviews of conventional medical practices. Overall, the “positive” and “possibly positive” reviews totaled 38.4%, while the “no evidence of effect” and “negative effect” totaled an alarming 61.6% [J. Ezzo, et al, “Reviewing the reviews:  How strong is the evidence? How clear the conclusions?” International J. Technol. Access in Health Care.  2001; 17(4): 457-466].

Overwhelming evidence reveals that conventional medicine is, on the whole, woefully unscientific.  It’s fashionable and easy to deny this, but the facts say otherwise.  So, by all means, Dr. Jones, be critical of CAM – but do not fall into a double standard.  Let us ruthlessly apply science to ALL we do as physicians.  Let us challenge ALL areas of medicine to a higher standard.  On that, I’m pretty sure we can agree.

Keep up the good work.

Sincerely yours,

Larry Dossey, MD



Woo Woo Is a Step Ahead of (Bad) Science

by Deepak Copra. MD

Sunday December 27, 2009

Categories: Consciousness

It used to annoy me to be called the king of woo woo. For those who aren't familiar with the term, "woo woo" is a derogatory reference to almost any form of unconventional thinking, aimed by professional skeptics who are self-appointed vigilantes dedicated to the suppression of curiosity. I get labeled much worse things as regularly as clockwork whenever I disagree with big fry like Richard Dawkins or smaller fry like Michael Shermer, the Scientific American columnist and editor of Skeptic magazine. The latest barrage of name-calling occurred after the two of us had a spirited exchange on Larry King Live last week. . Maybe you saw it. I was the one rolling my eyes as Shermer spoke. Sorry about that, a spontaneous reflex of the involuntary nervous system.

Afterwards, however, I had an unpredictable reaction. I realized that I would much rather expound woo woo than the kind of bad science Shermer stands behind. He has made skepticism his personal brand, more or less, sitting by the side of the road to denigrate "those people who believe in spirituality, ghosts, and so on," as he says on a YouTube video. No matter that this broad brush would tar not just the Pope, Mahatma Gandhi, St. Teresa of Avila, Buddha, and countless scientists who happen to recognize a reality that transcends space and time. All are deemed irrational by the skeptical crowd. You would think that skeptics as a class have made significant contributions to science or the quality of life in their own right. Uh oh. No, they haven't. Their principal job is to reinforce the great ideas of yesterday while suppressing the great ideas of tomorrow.

Let me clear the slate with Shermer and forget the several times he has wiggled out of a public debate he was supposedly eager to have with me. I will ignore his recent blog in which his rebuttal of my position was relegated to a long letter from someone who obviously didn't possess English as a first language (would Shermer like to write a defense of his position in Hindi? It would read just as ludicrously if Hindi isn't his first language).

With the slate clear, I'd like to see if Shermer will accept the offer to debate me at length on such profound questions as the following:

• Is there evidence for creativity and intelligence in the cosmos?
• What is consciousness?
• Do we have a core identity beyond our biology, mind, and ego?
• Is there life after death? Does this identity outlive the molecules through which it expresses itself?

The rules will be simple. He can argue from any basis he chooses, and I will confine myself entirely to science. For we have reached the state where Shermer's tired, out-of-date, utterly mediocre science is far in arrears of the best, most open scientific thinkers -- actually, we reached that point sixty years ago when eminent physicists like Einstein, Wolfgang Pauli, Werner Heisenberg and Erwin Schrodinger applied quantum theory to deep spiritual questions. The arrogance of skeptics is both high-handed and rusty. It is high-handed because they lump brilliant speculative thinkers into one black box known as woo woo. It is rusty because Shermer doesn't even bother to keep up with the latest findings in neuroscience, medicine, genetics, physics, and evolutionary biology. All of these fields have opened fascinating new ground for speculation and imagination. But the king of pooh-pooh is too busy chasing down imaginary woo woo.

Skeptics feel that they have won the high ground in matters concerning consciousness, mind, the origins of life, evolutionary theory, and brain science. This is far from the case. What they cling to is nineteenth- century materialism, packaged with a screeching hysteria about God and religion that is so passé it has become quaint. To suggest that Darwinian theory is incomplete and full of unproven hypotheses, causes Shermer, who takes Darwin as purely as a fundamentalist takes scripture, to see God everywhere in the enemy camp.

How silly. Shermer is a former Christian fundamentalist who is now a fundamentalist about materialism; fundamentalists must have an absolute to believe in. Thus he forces himself into a corner, declaring that all spirituality is bogus, that the sense of self is an illusion, that the soul is ipso facto a fraud, that mind has no existence except in the brain, that intelligence emerged only when evolution, guided by random mutations, developed the cerebral cortex, that nothing invisible can be real compared to solid objects, and that any thought which ventures beyond the five senses for evidence must be dismissed without question.

I won't go into detail about the absurdity of such rigid thinking. However, the impulse behind dogmatic materialism seems intended to flatten one's opponents so thoroughly that through scorn and arrogance they must admit defeat, conceding that science is the complete refutation of all preceding religion, spirituality, psychology, myth, and philosophy -- in other words, any mode of gaining knowledge that arch materialism doesn't countenance.

I've baited this post with a few barbs to see if Shermer can be goaded into an actual public debate. I have avoided his and his follower's underhanded methods, whereby an opponent is attacked ad hominem as an idiot, moron, and other choice epithets that in his world are the mainstays of rational argument. And the point of such a debate? To further public knowledge about the actual frontiers of science, which has always depended on wonder, awe, imagination, and speculation. Petty science of the Shermer brand scorns such things, but the greatest discoveries have been anchored on them.

If you are tempted to think that I have taken the weaker side and that materialism long ago won this debate, let me end with a piece of utterly nonsensical woo woo:

"Nobody understands how decisions are made or how imagination is set free. What consciousness consists of, or how it should be defined, is equally puzzling. Despite the marvelous success of neuroscience in the past century, we seem as far from understanding cognitive processes as we were a century ago."

That isn't a quote from "one of those people who believe in spirituality, ghosts, and so on." It's from Sir John Maddox, former editor-in-chief of the renowned scientific journal Nature, writing in 1999. I can't wait for Shermer to call him an idiot and a moron. Don't worry, he won't. He'll find an artful way of slithering to higher ground where all the other skeptics are huddled.