Considerations for the Design and Analysis of Pragmatic Trials -- September 18, 2023 (September 18, 2023 via “ACCORDS Research”): 35-min presentation & 16-min Q&A (presentation begins at 8m15s)

• Summary: Dr. Kathryn Colborn, taking a statistician’s perspective, sets the stage for a discussion of statistical analysis in pragmatic trials. The types of interventions and outcomes examined in pragmatic trials are discussed, as well as the appropriate selection of a control (non-intervention) group. It is emphasized that while individual-level randomization is preferred (e.g., it simplifies the statistical analysis for the trial), cluster randomization is common in the pragmatic space. Advice for working with a statistician is provided, with an emphasis on the creation of a robust (statistical) conceptual framework. Example statistical language - that may appear in a statistical analysis plan or trial protocol - is provided. A brief comparison of the complexity of statistical methods in individual-level randomized trials (e.g., use of t-tests or Chi-square tests) vs. cluster-level trials (e.g., use of sophisticated regression modeling approaches) is given, with an introduction to the intraclass correlation coefficient (ICC), the design effect, and the effective sample size; these topics will be discussed in future modules. The types of cluster trials (parallel, crossover, and stepped-wedge) are summarized, and the presentation concludes with an overview of statistical models often used in the (cluster) trial space, e.g., generalized linear mixed models (GLMMs) and models where parameters are estimated using generalized estimating equations (GEEs). Lastly, confounding, and its relation to randomization (including covariate constrained randomization) are briefly discussed, with a mention of covariate adjustment in trials (to be discussed in a future module).

Winter K, Pugh SL. An investigator’s introduction to statistical considerations in clinical trials. Urol Oncol. 2019 May;37(5):305-312. (17-page paper)

• Summary: Presents fundamental statistical considerations for clinical trials, emphasizing that one of the most important considerations is the need to work with a statistician, who will play an integral role from the beginning (i.e., in the design of the trial) through to the publication of the results. This paper touches on several topics, including but not limited to: the types of trials (phase I to III, including superiority vs. non-inferiority), sample size determinations, type I and II errors, statistical power, 1-sided and 2-sided hypothesis tests, endpoints and outcomes, effect size, interim analyses, time-to-event analyses, stratification and blinding, p-values, censoring, intention-to-treat (ITT) vs. as-treated analyses, and missing data. (These topics will be covered in greater detail in future modules.)

Design and Analysis Strategies for Embedded Pragmatic Clinical Trials (ASN Kidney Week - November 2019, Qilu Yu, Office of Clinical and Regulatory Affairs, National Center for Complementary and Integrative Health, NIH): 23-slide presentation.

• Summary: Dr. Qilu Yu connects the design of pragmatic trials (e.g., selection of a randomization scheme, endpoints/outcomes, and a control group) to several analytical considerations. It is stressed that pragmatic trials do not require a completely different set of analytical methods, and that the choice of methods depends on the research question. Several randomization schemes are discussed (e.g., individual-level randomized trials, individually-randomized group treatment trials, as well as group/cluster-randomized trials, such as parallel cluster randomization, and stepped-wedge approaches), as well as options for control groups. Analytical considerations related to clustering (e.g., the ICC, the design effect, and sample size) are briefly overviewed, as well as specific types of statistical models and missing data.

Kammar-García A, et al. Statistical Considerations for the Design and Analysis of Pragmatic Trials in Aging Research. Geriatrics (Basel). 2024 Jun 4;9(3):75. (11-page paper)

• Summary: A review discussing various statistical topics relevant to those designing a pragmatic trial. The following topics are discussed: 1. The unit of analysis and randomization scheme, 2. The outcome variable (and its relation to the type of statistical model selected), and 3. Common statistical models in pragmatic trials (e.g., generalized linear mixed effects models and models where parameters are estimated via generalized estimating equations - GEEs). Links to resources related to statistical analysis and sample size calculations for pragmatic trials are provided. Secondary analyses (including but not limited to subgroup analyses) are also discussed.

Importance of advanced statistics in clinical trial design (October 5, 2020 via “NIHNINDS”): 12-min video.

• Summary: Dr. Jordan Elm emphasizes that statistical inferences are contingent upon an understanding of the study design, the data ascertainment/measurement methods, and the anticipated amount of missing data and non-compliance (to the intervention). It is noted that a pre-specified statistical analysis plan helps to ensure statistical analyses are robust and appropriate to the research objectives. The intention-to-treat (ITT) approach is discussed, as well as basics related to sample size calculations and their relation to non-compliance (to the intervention). Missing data is briefly discussed, along with methods, such as imputation, to address missing data (to be covered in a future module). Lastly, generalized linear mixed models are briefly discussed, as well as pre-specified interim analyses.

Cook AJ, et al. Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH Health Care Systems Collaboratory Biostatistics and Design Core. Clin Trials. 2016 Oct;13(5):504-12. (15-page paper)

• Summary: Summarizes the lessons learned during the initial pilot phase of several pragmatic trials conducted between 2012 and 2014 within the NIH Pragmatic Trials Collaboratory (then known as the NIH Health Care Systems Collaboratory). It is emphasized that while individually randomized trials are statistically more straightforward, cluster trials are preferred when cluster-level randomization facilitates the implementation of the trial, or where there is risk of contamination; this paper focuses largely on statistical considerations related to cluster trials. The choice and number of randomization units and considerations related to unequal cluster sizes are overviewed. *Note that while not all pragmatic trials involve cluster-level randomization, statistical considerations often become more complex in this setting; these complexities are often the focus of statistical papers in the pragmatic space.

Wang R, et al. Statistics in medicine--reporting of subgroup analyses in clinical trials. N Engl J Med. 2007 Nov 22;357(21):2189-94. (6-page paper)

• Summary: Given the resources required to conduct a clinical trial, researchers often use analyses of subgroups of participants to obtain as much information as possible from a single study. Such analyses, which assess the heterogeneity of treatment effects in subgroups, may provide useful information about the intervention and/or for guiding future research. However, subgroup analyses also introduce analytical challenges and can lead to misleading results. This report outlines the challenges associated with conducting and reporting subgroup analyses and provides recommendations for their use.

Morris TP, et al. Choosing sensitivity analyses for randomised trials: principles. BMC Med Res Methodol. 2014 Jan 24;14:11. (5-page paper)

• Summary: Sensitivity analyses investigate whether the results of the primary analysis are sensitive (or robust) to violations of the assumptions of the chosen analytical approach. The following questions can be used to determine whether a sensitivity analysis is appropriate: 1. Does the proposed sensitivity analysis address the same question as the primary analysis? 2. Is it possible for the proposed sensitivity analysis to arrive at a different conclusion to the primary analysis? 3. If the proposed sensitivity analysis leads to a different conclusion to the primary analysis, is there a genuine degree of uncertainty as to which will be believed? To qualify as a sensitivity analysis, the answer to all these questions should be “yes.”