We assess the rate of replication for empirical papers in the 2010 American Economic Review. Across seventy empirical papers, we find that 29 percent have one or more citation that partially replicates the original result. While only a minority of papers has a published replication, a majority (sixty percent) have either a replication, robustness test or an extension. Surveying authors within the literature we find substantial uncertainty over the number of extant replications.
We make a two-pronged proposal to (i) strengthen the incentivizes for replication work and (ii) better organize and draw attention to the replications that are conducted. First we propose that top journals publish short "replication reports." These reports could summarize novel work replicating an existing high-impact paper, or they could highlight a replication result embedded in a wider-scope published paper. Second, we suggest incentivizing replications with the currency of our profession: citations. Enforcing a norm of citing replication work alongside the original would provide incentives for replications to both authors and journals.
Can Subtle Provision of Social Information Affect What Job You Choose (and Keep)? Experimental Evidence from Teach For America (Appendix)
American Economic Journal: Applied Economics, January 2017, 9(1): 96-117.
It has been well documented that information about the actions of others can affect small-stakes decisions. We show that a subtle provision of such social information can also influence a very high-stakes decision: whether to take (and keep) a job as a public school teacher. In an experiment involving thousands of admits to Teach For America (TFA), those provided with data about the high matriculation rate in the previous year are more likely to accept the job. Moreover, this effect persists into the second semester of teaching, even though one-sixth of those in the control group who initially accepted the job have left TFA by then. As expected, the effects are stronger for those more marginal in their decision to join TFA. Our results suggest that social information can have a powerful effect on high-stakes behavior and should be considered as a potential tool for policy.
Forthcoming, Management Science
Measuring sexual orientation, behavior, and related opinions is difficult because responses are biased towards socially acceptable answers. We test whether measurements are biased even when responses are private and anonymous and use our results to identify sexuality-related norms and how they vary. We run an experiment on 2,516 U.S. participants. Participants were randomly assigned to either a “best practices method” that was computer-based and provides privacy and anonymity, or to a “veiled elicitation method” that further conceals individual responses. Answers in the veiled method preclude inference about any particular individual, but can be used to accurately estimate statistics about the population. Comparing the two methods shows sexuality-related questions receive biased responses even under current best practices, and, for many questions, the bias is substantial. The veiled method increased self-reports of non-heterosexual identity by 65% (p<0.05) and same-sex sexual experiences by 59% (p<0.01). The veiled method also increased the rates of anti-gay sentiment. Respondents were 67% more likely to express disapproval of an openly gay manager at work (p<0.01) and 71% more likely to say it is okay to discriminate against lesbian, gay, or bisexual individuals (p<0.01).The results show non-heterosexuality and anti-gay sentiment are substantially underestimated in existing surveys, and the privacy afforded by current best practices is not always sufficient to eliminate bias. Finally, our results identify two social norms: it is perceived as socially undesirable both to be open about being gay, and to be unaccepting of gay individuals.
Economic Inquiry, January 2017, 55(1): 409-424.
Charitable donations are frequently raised by an intermediary: a fundraiser (that is not the charity) solicits and accepts donations and subsequently sends the proceeds to the charity -- e.g. a workplace campaign for United Way, a 5km walk for Susan G. Komen, or a cookie-selling campaign by a Girl Scout troop. Such fundraisers can greatly increase donations received by a given charity, but how do they affect what types of charities we support? This paper shows intermediary fundraisers can make donors insensitve to charity quality: Unattractive charities can receive the same financial support as an attractive charity. In a series of experiments, when donations are framed as going directly to the charity, attractive charities receive larger (between 68% and 91% larger average donation across studies) and more (between 19% and 25% higher likelihood of receiving a gift across studies) contributions relative to unattractive charities; however, when donations for the same charities are collected by (meaningless) intermediary fundraising campaigns, donations become statistically indistinguishable across charities. The fundraising campaign does not affect donor recall of charity identity or evaluation of charity quality; it simply precludes donors from using these data in the donation decision. Follow-up experiments suggest it is a superfluity of information in the intermediary fundraiser context that clouds the judgment of the donor.
How and when do pre-analysis plans, hypothesis registries, and replications help us know what we know and what we don’t know? A few simple models suggest the following take-aways: (i) Pre-analysis plans are most valuable for hypotheses that will not be tested multiple times and when the pre-analysis plans effectively eliminate researcher bias (not just reduce it), (ii) Pre-analysis plans can be valuable for replication efforts, and (iii) very few replications, typically three to five, are required to achieve precise, correct beliefs about the hypothesis, even with modestly biased replication attempts.
Journal of Political Economy, June 2012, 120(3): 359-397.
Intermediation Reduces Punishment (And Reward)
paper experimentally analyzes the schooling decisions of poor
households in urban Brazil. We elicit parents’ choices between monthly
government transfers conditional on their adolescent child attending
school and guaranteed, unconditional transfers of varying sizes. In the
baseline treatment, an overwhelming majority of parents prefer
conditional transfers to larger unconditional transfers. However, few
parents prefer conditional payments if they are offered text-message
notifications whenever their child misses school. These findings suggest
important intergenerational conflicts in these schooling decisions, a
lack of parental control and observability of school attendance, and an
additional rationale for conditional cash transfer programs: the
monitoring they provide.
(Web Appendix), (Data, do file, and read me)
American Economic Journal: Microeconomics, 3(November 2011): 77-106.
This paper shows moral decision-making is not well predicted by
the overall fairness of an act but rather by the fairness of the
consequences that follow directly. In laboratory experiments, third
party punishment for keeping money from a poorer player decreases
when an intermediary actor is included in the transaction. This
is true (i) for completely passive intermediaries, (ii) even though
intermediation decreases the payout of the poorest player and hurts
equity, and (iii) because intermediation distances the transgressor
from the outcome. A separate study shows rewards of charitable
giving decrease when the saliency of an intermediary is increased.
While economic theories of persuasion emphasize self-interest, others posit an important role for other-regard. For example, a salesperson might describe product features but also try to build rapport. We study these two mechanisms within a simple but rich experimental framework in which sellers, in a free-form conversation, try to convince buyers to raise their valuations for objects. We find that sellers benefit from communication despite their material conflict of interest. Communication affects both buyers’ self-interest and their other-regard. Changes in other-regard are mean zero, but interestingly a minority of sellers target other-regard and substantially outperform their peers. More generally, however, who is buying is actually a better predictor of persuasion than who is selling. Buyer-seller homophily also strongly predicts persuasion: gender-match, for example, more than doubles the sellers’ expected gain.
with Alexander Gotthard-Real
Can an organization avoid some blame for an unpopular action by hiring an adviser to advise them to do it? We present experimental evidence suggesting this is the case – advice to be selfish substantially decreases punishment of being selfish. Further, this result is true despite advisers’ conflicts of interest, known to all: Through a relational contract, advisers are motivated to tell the principals what they want to hear. In follow-up treatments, we show advice does not decrease punishment solely by affecting beliefs of how necessary the selfish action was. Finally, in treatments where advisers are available, selfish principals act more selfishly.
“Beliefs, Information and the Education Plans of Middle School Children in the Dominican Republic”
funded by research grants from the International Growth Centre, $1.26M grant from US AID, and Fundacion INICIA
conduct a large-scale evaluation of interventions that present information on
the potential benefits of schooling to 7th and 8th grade students in the
Dominican Republic. The evaluation includes 600 schools, 25 percent of all
public middle schools in the country. The broadest intervention consists of
four 15-minute videos that discuss the benefits of additional schooling,
watched by classes altogether. We vary whether these videos present the
benefits qualitatively – e.g. schooling may increase wages — or quantitatively —for example, wage averages and distributions
at different levels of schooling — allowing us to isolate the impact of
providing quantitative information on the returns to schooling. We also conduct
one-on-one video sessions through a novel tablet application, both with parents
and children. Finally, we also hang posters in classes, designed to highlight
and reinforce key information provided in the videos. To understand the
mechanics behind our results, as well as the schooling decision in general, we
also conduct a panel series of surveys covering 30,000 students to measure
beliefs of the potential value of education and students’ educational plans. We present preliminary results of the impacts
of the interventions on both the survey outcomes and dropout.
"A Model of Information Nudges"
with Clayton Featherstone, Wharton, and Judd Kessler, Wharton
A growing empirical literature has demonstrated that providing decision-makers with information (e.g. about the actions of others or the returns to different actions) can affect behavior. However, the literature lacks a theory that can explain when such interventions will have a large effect or even the sign of the effect. We introduce such a theory, based on simple Bayesian updating in a setting of binary choice. It yields the following intuitive insight: the sign of the effect depends on whether the intervention causes the marginal agent to update her belief up or down. Further, the magnitude of the effect depends on both the density of agents at the margin and how much those agents’ beliefs move when treated. We also show that when it is prohibitively costly or impossible to directly measure the beliefs of marginal agents, we can proxy for these beliefs with the fraction of agents taking the action in the uninformed group. Utilizing this intuition, our model makes a strong prediction about how treatment effect sign and magnitude will vary with the proportion taking the action in the control group. Our model reasonably rationalizes results from the literature: we perform a meta-analysis of informational nudges and find that, even across very different experimental settings, the magnitude of the treatment effect varies in a way our theory predicts (Note: More data to come soon).
“Expectations Do Not Determine Punishment”
This paper reports a series of laboratory experiments investigating the hypothesis that expectations affect punishment. Despite support from Moral Psychology and recent reference-dependence experiments in Economics, I find third party punishment does not respond to exogenous changes in expectations of the targeted party's behavior. I use a random process for revealing the true action taken by the actor. This process varies the expectation the punisher holds just before the truth is revealed. Expectations are shown to vary significantly and substantially. However, in non-parametric and instrumental variables regression analyses, expectations are shown not to affect punishment at all. This is true either when expectations are exceeded or failed.
Works in Progress
“Declined Options as Reference Points”
"Money as a Medium of Immoral Exchange"
with Muriel Niederle, Stanford
in The Oxford Handbook of Economic Conflict Resolution, 2012, Oxford Press, Rachel Croson & Gary Bolton eds.