WE ARE NO LONGER ACCEPTING NEW COLLABORATORS. THE REMAINDER OF THE PAGE IS IN ITS ORIGINAL FORM:
A major goal of empirical social science is to produce results that are replicable. However, different empirical studies on the same question often produce different results. Different results show up for many valid reasons: the use of different data sources, different statistical assumptions, or different methods. Yet, there are choices that may cause differences in results that are less visible and may not be subject to critical review from editors, reviewers, and readers. Newly developing data-access policies, some of which require programs and code to be made available alongside data (e.g.) allow observers to check for errors themselves, but would give little sense of how sensitive results may be to a bevy of relatively non-controversial (i.e. non-"error") choices. Current trust in empirical results is based on the assumption that these minor decisions have little impact on results, which may or may not be true.
In this project, we are asking recruited researchers to perform a “blind” replication of one of two studies. Without telling researchers the methods used by the original study, we will instruct participants to use a particular data set and set of statistical assumptions in order to estimate a single specific causal estimate. Participants will clean the data, construct variables, and make the other minor decisions that go into a statistical analysis, aside from the data source, identifying assumption, and effect of interest, which will be held constant. By comparing the analyses that different researchers perform under these conditions, we will estimate the variability in estimates that occurs as a result of decisions that researchers make.
This approach is different from most replication studies in economics. We are not trying to test the validity of the original results. Instead, our aim is to measure the degree of variation in results that can be attributed to generally "invisible" features of analysis. You may have seen similar tests elsewhere, such as in the New York Times' The Upshot section. Our project is most similar to the "Crowdsourced Data Analysis" project described by Raphael Silberzahn and Eric Uhlmann here, although our goal is slightly different.
If you are interested in joining us, we are looking for researchers who have published at least one published or forthcoming paper in the empirical microeconomics literature and who are familiar with methods of causal identification. Participants will be offered authorship on the final publication. We are also currently working on securing funding. If we do, there may be financial compensation for your time.
We have designed the study to take as little of your time as possible. We expect that the amount of effort required on your part to participate is comparable to doing one or two problem sets in a graduate applied econometrics course. We have selected studies for which data is already available. We also are not asking for a full replication, but just for a table of summary statistics and an estimate of a single focal causal effect. If you normally use graduate students to perform data work, we encourage you to use them for this project as well. Particular replication results will be kept anonymous.
Please access our sign-up sheet here.
We also welcome you to share this page with others who may be interested.
This project is being led by (Names blinded during paper review process)
We have received approval from the (blinded) IRB to invite collaborators to this project.