STATISTICAL TIPS

Effect size and other tricky statistical terms

🎩 The Illusion of Magnitude: Demystifying "Effect Size"

In the world of clinical research, Effect Size is often treated like the holy grail. While p-values merely tell you if a result is a fluke, the effect size promises to tell you what you really want to know: How big is the punch this treatment packs? It is the metric that turns statistical jargon into clinical relevance.

But here is the catch: effect size is a master of disguise.

Think of it less like an objective tape measure and more like a camera angle. Depending on how you handle your data, a trick of the light can turn a minor ripple into a tsunami. If you only look at the patients who completed your study, your effect size will look massive and heroic—a true statistical champion. But if you look at everyone who started it, that same champion might suddenly shrink to the size of an ant.

The term implies you are measuring the "size" of the "effect." But in reality, if you are not careful, you might just be measuring the size of your selection bias.

THE CLINICAL TRIAL ILLUSION MAP

High trial drop-out rates can bias your results. Learn why a "Completers" analysis can lie, and how ITT and the CACE model reveal the true clinical efficacy.

When clinical trials face high, asymmetrical drop-out rates, standard analyses can yield misleading conclusions. This article explains the critical differences between Completers, Intention-to-Treat (ITT), and CACE (Complier-Average Causal Effect) models, helping clinicians uncover true biological efficacy without statistical bias.

In an ideal world, randomized clinical trials (RCTs) run perfectly: every enrolled patient takes their assigned medication, and no one misses a follow-up. In the real world, however, patients drop out.

When drop-out rates are high and asymmetrical—for instance, a 48% overall drop-out rate where 87% occurs in the treatment group and only 9% in the control group—standard statistical analyses can yield vastly different, and sometimes dangerous, conclusions.

If your trial on 230 patients shows a massive Cohen’s effect size of 1.10 among those who finished the study, can you trust it? Let's break down how three different analytical approaches interpret this scenario.

1. The "Completers" Analysis: A Dangerous Illusion

A Completers (or Per-Protocol) analysis evaluates only the patients who successfully made it to the end of the trial. In our scenario, this means looking at a tiny, highly selected group of "survivors" in the treatment arm versus nearly everyone in the control arm.

• The Result: It yields an exceptionally large effect size (1.10).

• The Clinical Trap: This is a classic selection bias (false positive). It ignores why 87% of the treated patients disappeared. Did they experience severe side effects? Did the treatment fail them? By evaluating only the "success stories," this method creates the illusion of a miracle drug that will completely fail to replicate in real-world clinical practice.

2. Intention-to-Treat (ITT): The Conservative Safeguard

The Intention-to-Treat (ITT) approach is the gold standard for clinical regulations. It analyzes all 230 patients based on their initial random assignment, regardless of whether they dropped out, dropped dead, or refused the medication.

• The Result: The massive 1.10 effect size plummets heavily (potentially down to 0.15–0.25, or even zero).

• The Clinical Value: ITT asks a pragmatic public health question: "What happens if I prescribe this drug to the general population, knowing half of them might stop taking it?" While it successfully prevents the approval of toxic or impractical treatments, ITT is heavily conservative and tends to underestimate the true biological potential of a drug for patients who can actually tolerate it.

3. CACE (Complier-Average Causal Effect): Finding the Scientific Truth

How do we bridge the gap between the overly optimistic Completers analysis and the overly punitive ITT? The answer lies in the CACE model (famously detailed by Dunn et al., 2005).

Using advanced structural models, CACE utilizes the original randomization as an "instrumental variable" to mathematically correct the distortion caused by high drop-outs. It isolates and estimates the treatment effect exclusively among "Compliers"—the subgroup of patients biologically predisposed to adhere to the protocol.

• The Result: It strips away the selection bias, offering a realistic, moderate-to-good effect size (likely around 0.30–0.45) rather than the inflated 1.10.

• The Clinical Value: CACE answers the fundamental scientific question: "Among patients who actually adhere to the therapy, what is the true, unbiased biological efficacy of the treatment?"

Summary for Clinicians

Analytical Strategy

Core Question Addressed

Clinical Risk

Completers

How did the patients who finished the drug do?

Extremely High. Risks promoting a toxic or ineffective drug based on a biased subgroup.

Intention-to-Treat (ITT)

What happens if I prescribe this drug to everyone?

Very Low (Safe). But it might mask a drug's true biological benefit by penalizing it for drop-outs.

CACE (Dunn et al. Model)

What is the true effect on patients who actually adhere?

Minimal. Provides the most accurate, unbiased estimate of true efficacy under heavy drop-out conditions.

Conclusion

When faced with high and uneven drop-out rates, a large effect size among completers is a red flag, not a victory. While ITT remains essential to prove real-world safety and policy feasibility, implementing a CACE analysis is the most robust way to uncover the true scientific efficacy of your intervention without falling into statistical traps.

📌 Take-Home Message for Clinicians

• Beware of the "Completers" Trap: A high effect size (like d = 1.10) among only the patients who finished a trial is often a statistical illusion. When drop-out is high and uneven, analyzing only the completers introduces severe selection bias.

• ITT Keeps You Safe, But CACE Finds the Truth: While Intention-to-Treat (ITT) is essential for regulatory safety because it reflects real-world clinical practice, it heavily penalizes the drug for every patient who drops out.

• CACE Is Your Scientific Lens: When patients disappear and you do not know why, the CACE model (Dunn et al., 2005) is the most robust tool to uncover the true, unbiased biological efficacy of a treatment specifically for the patients who adhere to it.

• The Golden Rule: Never publish or evaluate a trial with high drop-out rates using a single analysis. Always look at both the pragmatic real-world picture (ITT) and the unbiased causal effect (CACE) to see the whole truth.

🎯 Effect Size

• The Official Story: A standardized measure of the magnitude of a therapeutic effect, independent of sample size (e.g., Cohen’s d).

• The Reality Check: The metric that tells you if your treatment packs a punch or just a gentle tickle.

• The Trap: It’s a master of disguise. Calculate it only on the patients who finished the trial, and it looks like a giant. Calculate it on everyone who started, and it shrinks to an ant. Remember: sometimes a "huge effect size" is just a huge reflection of your selection bias!

🎲 P-value

• The Official Story: The probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct.

• The Reality Check: The scientific community’s favorite gatekeeper. It answers just one question: "Is this result a genuine discovery or did we just get incredibly lucky with the cosmic roulette?"

• The Trap: Clinicians love a p < 0.05 like a child loves Christmas. But a tiny p-value does not mean your treatment is a breakthrough. It just means the effect is likely real. If the effect size is microscopic, your drug might technically "work," but it will be clinically useless to your patients.

🩹 Data Imputation

• The Official Story: The process of replacing missing data with substituted values based on statistical assumptions.

• The Reality Check: Statistical damage control. When patients drop out of your study, they leave behind an ugly, empty void in your spreadsheet. Imputation is the art of using math to fill those blanks with educated guesses.

• The Trap: It’s like using a concealer to hide a blemish. Done right (e.g., Multiple Imputation), it respects the uncertainty of the data. Done poorly (like assuming everyone who dropped out perfectly recovered), it’s just creative fiction that makes your data look much prettier than it actually is.

🤝 Compliers

• The Official Story: Participants in a randomized trial who adhere strictly to the treatment protocol to which they were assigned.

• The Reality Check: The "unicorn" patients. They listen to instructions, take their pills at the exact right hour, never miss a follow-up visit, and don't disappear into thin air halfway through the trial.

• The Trap: If you base your entire analysis only on them, you are living in a fantasy world. Real-world clinics are full of non-compliers, which is why we need tools like the CACE model to figure out if the drug works because of its biology or just because these patients were exceptionally disciplined.

🔍 Small-Study Effects (The Small-Sample Amplifier)

• The Official Story: The tendency for smaller studies in a meta-analysis to show larger treatment effects than larger studies, often due to publication bias, lower methodological quality, or random sampling error.

• The Reality Check: The "loudest voice in the room" syndrome. When you test a drug on a small group of people, random luck can easily skew the results, making a mediocre treatment look like a medical miracle.

• The Trap: Small studies are highly volatile. If a small study shows a massive effect size, it’s often because it got lucky with random noise. When you pair a small sample size with a 48% drop-out rate, you aren't looking at scientific evidence anymore—you're looking at a statistical mirage.