This is a practical introduction to evidence factors in observational studies. It was delivered on-line to the MRC IEU at the University of Bristol in May 2022. The two earlier videos from 2020 are more technical in nature.
References about Evidence Factors
As this is a less technical talk, I have selected less technical references than the references for my two video talks in 2020.
Chapter 7, "Elaborate Theories," of my book, Observation and Experiment: An Introduction to Causal Inference (Harvard University Press 2017) has some discussion of evidence factors, as does Chapter 8, "Replication, Resolution and Evidence Factors," of my book Causal Inference (MIT Press 2023). For more about these books, see the books tab of this web page.
Rosenbaum Paul R. How to see more in observational studies: Some new quasi-experimental devices. Annual Review of Statistics and Its Application. 2015 Apr 10;2(1):21-48. https://doi.org/10.1146/annurev-statistics-010814-020201 Section 3 discusses evidence factors.
Rosenbaum PR. Cochran's causal crossword. Observational Studies. 2015;1(1):205-11. Available free: https://muse.jhu.edu/article/793420/pdf Also: https://doi.org/10.1353/obs.2015.0021 This article provides some informal links between evidence factors, certain articles by the statistician William Cochran, and the ideas of philosopher Susan Haack in her book Evidence and Inquiry.
Rosenbaum PR. Replicating effects and biases. The American Statistician. 2001 Aug 1;55(3):223-7. https://doi.org/10.1198/000313001317098220 In JSTOR: https://www.jstor.org/stable/2685805 Evidence factors seek two independent pieces of information likely affected by different forms of unmeasured confounding. This article explains why a bigger sample size is of limited benefit when trying to address systematic biases from unmeasured confounding. Big data is the solution only if small data was the problem, but unmeasured confounding has no reason to diminish as the sample size increases.
Karmakar B, Small DS, Rosenbaum PR. Using evidence factors to clarify exposure biomarkers. American Journal of Epidemiology. 2020 Mar 2;189(3):243-9. https://doi.org/10.1093/aje/kwz263
References about Triangulation in Causal Inference
Although evidence factors and triangulation are distinct topics in causal inference, they are related. See, for instance:
Lawlor DA, Tilling K, Davey Smith G. Triangulation in aetiological epidemiology. International Journal of Epidemiology. 2016 Dec 1;45(6):1866-86. https://doi.org/10.1093/ije/dyw314
Munafò MR, Davey Smith G. Robust research needs many lines of evidence. Nature. 2018 Jan 25;553(7689):399-401. https://doi.org/10.1038/d41586-018-01023-3
Related Ideas of Philosophers
Haack S. Evidence and inquiry: Towards Reconstruction in Epistemology. Oxford: Blackwell, 1993.
Haack S. Double-aspect foundherentism: a new theory of empirical justification. Philosophy and Phenomenological Research. 1993 Mar 1;53(1):113-28. https://doi.org/10.2307/2108056 In JSTOR: https://www.jstor.org/stable/2108056
Staley KW. Robust evidence and secure evidence claims. Philosophy of Science. 2004 Oct;71(4):467-88. https://doi.org/10.1086/423748 In JSTOR: https://doi.org/10.1086/423748