An observational study has two evidence factors if there are two essentially independent statistical tests of the null hypothesis of no treatment effect that are likely to be affected in different ways by unmeasured confounding. In light of this, two evidence factors may provide mutual support, reducing concern about unmeasured confounding, or they may clash, heightening concern about unmeasured confounding. By virtue of being independent in the formal statistical sense, two evidence factors and their sensitivity analyses may be combined using meta-analytic techniques as if they came from unrelated studies of different people by different investigators, despite their origin as two analyses of the same data. For a brief, informal discussion of evidence factors, see Chapter 8 of my book Causal Inference (MIT Press, 2023). For detailed discussion, see my book Replication and Evidence Factors in Observational Studies (Chapman&Hall/CRC 2021).
Rosenbaum PR. Evidence factors in observational studies. Biometrika. 2010;97(2) Jun 1:333-45. In JSTOR: https://www.jstor.org/stable/25734089 https://doi.org/10.1093/biomet/asq019
Rosenbaum PR. Some approximate evidence factors in observational studies. Journal of the American Statistical Association. 2011 Mar 1;106(493):285-95. In JSTOR: www.jstor.org/stable/41415552 https://doi.org/10.1198/jasa.2011.tm10422
Zhang K, Small DS, Lorch S, Srinivas S, Rosenbaum PR. Using split samples and evidence factors in an observational study of neonatal outcomes. Journal of the American Statistical Association. 2011 Jun 1;106(494):511-24. In JSTOR: jstor.org/stable/41416388 https://doi.org/10.1198/jasa.2011.ap10604
Rosenbaum PR. The General Structure of Evidence Factors in Observational Studies. Statistical Science. 2017;32(4):514-30. In JSTOR: www.jstor.org/stable/26408856 Open access at ProjectEuclid: https://doi.org/10.1214/17-STS621
Karmakar B, Small DS, Rosenbaum PR. Using approximation algorithms to build evidence factors and related designs for observational studies. Journal of Computational and Graphical Statistics. 2019 Jul 3;28(3):698-709. https://doi.org/10.1080/10618600.2019.1584900
Karmakar B, Small DS, Rosenbaum PR. Using evidence factors to clarify exposure biomarkers. American Journal of Epidemiology. 2020 Mar 2;189(3):243-9. https://doi.org/10.1093/aje/kwz263
Karmakar B, Small DS, Rosenbaum PR. Reinforced designs: Multiple instruments plus control groups as evidence factors in an observational study of the effectiveness of Catholic schools. Journal of the American Statistical Association. 2021 Jan 2;116(533):82-92. https://doi.org/10.1080/01621459.2020.1745811
Rosenbaum PR. A second evidence factor for a second control group. Biometrics. 2023 Dec;79(4):3968-80. https://doi.org/10.1111/biom.13921
Rosenbaum PR. A New Construction of Evidence Factors in an Observational Study of Light Daily Alcohol Consumption and Longevity. Journal of the American Statistical Association. 2026, to appear. https://doi.org/10.1080/01621459.2026.2624858