Why Won't You Skeptics Accept My Study?
The COPD-Acupuncture Study
In 2012, a study from Japan was published in Archives of Internal Medicine entitled A Randomized, Placebo-Controlled Trial of Acupuncture in Patients With Chronic Obstructive Pulmonary Disease (COPD) . Its lead author, (Masao Suzuki, LAc, PhD, Department of Clinical Acupuncture and Moxibustion, Meiji University of Integrative Medicine) concluded:
“We demonstrated clinically relevant improvements in DOE (dyspnea on exertion) (Borg scale), nutrition status (including BMI), airflow obstruction, exercise capacity, and health-related quality of life after 3 months of acupuncture treatment. Randomized trials with larger sample sizes and longer-term interventions with follow-up evaluations are necessary to confirm the usefulness of acupuncture in COPD treatment.”
The patients in the study all had chronic lung disease and were all treated with conventional inhalers. One group was given traditional acupuncture, the other placebo acupuncture. The “real acupuncture group” (RAG) demonstrated less shortness of breath after 3 months than the “placebo acupuncture group” (PAG).
Immediately, acupuncture advocates hailed the study as vindication for their craft. The study’s authors stated, “This study clearly demonstrates that acupuncture is a useful adjunctive therapy in reducing DOE in patients with COPD.” Healthcare Medicine Institute (a pro-acupuncture group) hailed this as a “groundbreaking study”.
Skeptics weren’t convinced. David Gorski, M.D, wrote on the Science-based Medicine blog, “true believers like Suzuki (who is an acupuncturist, which is why I refer to him as a “true believer”) produce studies that on the surface appear sound but are riddled with problems when examined more closely.”
So what is going on here? This is a study published in a mainstream peer-reviewed journal. It is easy for acupuncture proponents to claim hypocrisy on the part of the scientific skeptics. To understand this, we have to review some basics.
Why Won’t You Skeptics Accept Our Study?
We are all wrong. It is just that some are less wrong than others.
Good skeptics may seem like they are trying to be know-it-alls. They can sometimes be annoying. Many fantastic ideas seem to land on the deaf ears of the naysayers. But there is a world of difference between good skeptics and cynics. It would seem that both true believers and cynics cling to positions that they claim to know to be true with little or no evidence. Skeptics on the other hand are less concerned with being right, rather, they are primarily concerned with being less wrong than others. That is, skeptics ask for proper evidence in sufficient magnitude to accept a claim. And then it is only to tentatively accept the claim, at least until a better explanation comes along.
There have been many studies published in peer-reviewed journals that appear to support treatments usually considered to be quackery by skeptics. These studies are held up like badges of honor among the proponents of the treatment in question. So, why won’t the skeptics accept their treatments as real? Are they being closed-minded? Are they being cynics?
Skepticism 101
To understand the answer requires understanding of Bayesian thinking, cognitive biases and Occam’s Razor.
When it comes to claims that fly in the face of our understanding of biology, physics and physiology, skeptics recognize that the the burden of proof is very, very large. This points to the concept of “prior probability” in Bayesian reasoning. Such claims would have a very low prior probability, because in order for the claim to be true, our current understanding of biology and physiology would have to be overturned (or at least modified). This is okay, but such a claim would be considered unlikely to be true given our current understanding.
To overcome the hurdle of a low prior probability, the evidence for the claim must be very high. It cannot be ambiguous. The studies that are touted must have very good design, power and be highly controlled against as many biases as possible. The evidence must be reproducible. Implausible ideas cannot be accepted without these things.
Our biases allow us to be systematically fooled. Confirmation bias is the classic. In fact, most of the biases that fool us in medical research are related to the confirmation bias. We all tend to perceive reality in ways that tend to confirm our previously held beliefs. We do this subconsciously. Most of us are biased to think of ourselves as smart and, therefore, not subject to such biases. This bias (self-deception) prevents us from realizing the confirmation bias.
Experimenters will tend to enlist patients who will likely lead to positive outcomes in their experiment. This is natural. We call this selection bias. If you want to test a treatment for anxiety, you may be tempted to put patients with calmer demeanors in the treatment group rather than in the placebo group, as they are likely to resolve their anxiety issues anyway. This would make the anxiety treatment appear to work, but the observation would be false. To guard against this, subjects must be randomized into the arms of the study.
Likewise, the subject-expectancy effect occurs when subjects of an experiment know what the observer is looking for and subtly influence the experiment in favor of the expected outcome. To compensate for this bias, the subjects of an experiment should be blinded to the variable being measured. Ideally, they should not know what question the experiment is designed to answer. In medical studies, this is not always ethical. The subjects must, however, be blinded as to whether they are actually getting the treatment or getting a placebo. By doing so, we minimize the subject-expectancy bias and are able to filter out the many forms of placebo effects.
Another bias that should be guarded against is the observer-expectancy effect. When experimenters are aware of which patients are receiving the treatment and which are getting the placebo, they tend to give subtle cues to the subjects as to how they expect the experiment to go. The Clever Hans Effect illustrates this. Clever Hans was a horse who apparently could solve simple math problems by reacting to the answer on note cards. In reality, it was his owner who was recognizing the correct answer and giving Clever Hans physical cues to which Hans reacted. To guard against this effect, the observer must be blinded about the treatment as well. This is often difficult, as when testing physical procedures such as knee surgery, spinal-manipulative therapy, and acupuncture. It is rarely impossible to blind the observer. It requires some imagination. It also requires that the experimenters acknowledge that they are subject to this effect. Scientists may be particularly resistant to this due to the self-deception bias (“I am smart enough to not bias the experiment!”).
For an experiment to claim the prestigious title of a prospective, randomized controlled trial (RCT), it must be designed to control for all of these biases to the extent that the imagination of the experimenters can muster. An RCT that does not use control groups, randomize its subjects, blind its subjects AND its observers, and submit itself for critical peer review cannot produce results that deserve to be taken without criticism.
Even if it accomplishes all that, its results must be reproducible and statistically significant enough to overcome the prior probability of the idea being tested.
When faced with data that seems to oppose our current science-based understanding of biology and physiology, we have to decide which ideas are in error. Is the conclusion of the study incorrect, or is our best understanding of physiology incorrect? To help us, we turn to Occam’s Razor.
It seems reasonable to say that the purpose of science is to decrease the mystery of the world. In other words, to answer questions to the best of our ability. History has surely shown that the answers to some questions lead to more questions. In other words, they seem to expand the mystery. However, if the answers are achieved through the proper use of science, then the expanded mystery simply reveals our prior ignorance. This would seem to occur in areas concerning frontier science, such as in the early days of medicine and our current search for dark matter. In many areas, we have a deep, tested and working understanding of how things work. It would seem reasonable to tentatively side with answers that do not require new assumptions to be true. In other words, it is reasonable to tentatively side with answers that decrease the mystery rather than increase it. This ,in essence, is Occam’s Razor.
For instance, many claim that manipulating the spine in a certain way will unblock the flow of a life-force energy, and therefore can cure many ailments, including back pain. Suppose that studies are done with proper randomization and blinding (with some imagination, even the investigators can be blinded). Suppose that the results consistently showed modest, short-lived decreases in pain with the treatment compared to placebo/sham groups and that these results were statistically significant. The authors conclude that the life-force exists and that back pain can be treated by unblocking it. However, such an explanation makes an assumption that a life-force exists -- thereby increasing the mystery.
Now, a skeptical reader will recognize that the back pain reduction may also be explained by the known physical dynamics of the musculoskeletal system. One explanation requires the existence of an unseen force that is not coherent with our knowledge of physiology and anatomy. The other requires no such new assumption. The second explanation may leave room for some smaller unknowns about muscle physiology, but it presents no contradictions. The skeptical explanation does not increase the mystery.
Occam’s Razor would have us favor the explanation that does not expand the mystery. To ignore Occam’s Razor is to engage in Toothfairy science. Before we study a phenomenon, we should first establish whether the phenomenon actually exists.
Applied Skepticism: The Suzuki COPD-Acupuncture Study
Getting back to our example study, what is it about Suzuki’s COPD-acupuncture study that causes skeptics to be so...well, difficult?
First, we should consider how Bayesian thinking would lead one to approach the issue. The skeptic considers the concept of acupuncture for treating a condition like COPD to have a very low prior probability. Everything that we know about physiology and neuroanatomy seems to contradict the idea of needles in the skin altering the dynamics of the pulmonary system. We certainly have no reason to accept the idea of qi flowing through meridians. This does not mean that it cannot be true because -- as believers are happy to point out -- we do not know everything.
However, we do have scientific knowledge about COPD and the nerves in the skin. We are faced with the dilemma of having to abandon well-tested, scientific knowledge in order to accommodate the notion that puncturing the skin in a certain way will change pulmonary physiology. Again, not impossible, but it is a big burden of proof to bear. The evidence will need to be outstanding. It automatically requires unequivocal results from large, double-blinded controlled trials that are consistently reproducible even when involving practitioners and patients that have not been preconditioned to believe in acupuncture.
Secondly, if the issue at hand has a small prior probability, and therefore carries a large burden of proof, we must ask why proponents accept such a small amount of “proof” in the first place. It would seem that proponents of such claims may already believe in the claim before any evidence is produced. Thus, the proponents may be subject to many cognitive biases such as confirmation bias, selection bias and others.
In the case of the above COPD study, the lead author is a licensed acupuncturist and works in the Department of Clinical Acupuncture and Moxibustion, Meiji University of Integrative Medicine. The study was funded by the Grants-in-Aid for scientific research from the Japan Society of Acupuncture and Moxibustion (JSAM). One can speculate that those involved in the study’s design and funding were likely “all-in” on their belief in acupuncture.
Much is made in the article that the study was randomized and blinded. However, as the authors actually point out, the study was only single-blinded -- meaning that the patients were supposedly not aware if the needles were actually puncturing the skin, but the acupuncturists were very much aware of what they were doing. Now imagine that you are the patient. You feel the needles touching the skin and you feel their pressure. It is difficult to tell if it is a real needle or a “sham” needle that actually retracts into the handle instead of penetrating the skin. So far, so good.
There is much more involved in the experience of acupuncture therapy than just the needling. It is an intimate interaction between the acupuncturist and the patient. In fact, previous evidence supports the idea that it is the experience of the interaction that seems to make the difference, rather than the needling itself. Imagine then that you are getting acupuncture treatment from a practitioner who knows he/she is deceiving you, but is trying to act otherwise. We can speculate that the experience of the interaction may be different, even on a very subtle level. Thus, the practitioner may subtly “unblind” the patient, even if the patient is not aware of this to the degree that they would admit. Patients thus may have been more prone to the non-specific benefits of an acupuncture experience in the “real” group.
The authors actually asked the patients their opinion as to whether they thought they were in the “real” group or the sham group. There were no significant differences reported. However, we are not talking about conscious differences. After all, the owner of Clever Hans was not aware that he was giving his horse the clues, and would have denied such allegations. It seems reasonable that all of those involved were likely prone to believing in acupuncture and likely wished the study to succeed. Thus, in such a study, there are inherently large channels through which many biases can enter.
Lastly (for this discussion), even if we accept the outcome (which is highly dubious), we must ask ourselves if the observations may be better explained by more mundane phenomena ala Occam’s Razor.
Which conclusion requires the fewest new assumptions? Which conclusion tends to decrease the mystery in the world?
Setting aside for the moment the notions of qi, meridians and other mystical baggage that burdens the scientific acceptance of acupuncture, we are asked to accept the notion that needling the skin can alter pulmonary function. The authors accept that it does and provide many "science-y" sounding possible explanations.
Skeptics may consider the known psychological phenomena of cognitive biases and the Clever Hans Effect to account for the apparent differences in shortness of breath between the “real” and “sham” groups. With such a mundane explanation, skeptics do not have to overcome the huge Bayesian hurdle of low prior probability. Outcomes such as distance walked, shortness of breath and pulmonary function testing may be influenced by effort and enthusiasm. For instance, it is recognized that motivation is a factor that can increase performance on the 6 minute walk test.
Acceptance of the study results requires extra assumptions and rationalizations. Indeed, the authors try hard to come up with scientific sounding explanations. We are reminded by the wise Dr. Harriet Hall to be wary of Toothfairy Science.
The skeptical position is to tentatively reject the study results on the grounds that it did not meet the inherently large burden of proof because of the issues mentioned above.
Conclusion
So, the next time skeptics seem unconvinced by a study that is receiving fanfare by the media and others, they are probably not being closed-minded cynics. They are also not trying to always be right.
They are just trying to be less wrong.
John Byrne, M.D.
References
Suzuki, M. "JAMA Network | JAMA Internal Medicine | A Randomized, Placebo ..." 2012.
<http://jamanetwork.com/article.aspx?articleid=1151703>
"Acupuncture Eases COPD - New Research - HealthCMi." 2013 .
<http://www.healthcmi.com/acupuncturist-news-online/803-copdcv4k3>
"Self-deception - The Skeptic's Dictionary - Skepdic.com."
<http://skepdic.com/selfdeception.html>
"More “bait and switch” acupuncture studies « Science-Based Medicine." 2013.
<http://www.sciencebasedmedicine.org/bait-and-switch-acupuncture-studies/>
Enright, PL. "The Six-Minute Walk Test - Respiratory Care." 2003.