Abstract:
We model the problem of communicating scientific information with a heterogenous audience, and contrast this problem with standard statistical optimality criteria. We show that our model sometimes delivers substantially different recommendations than standard criteria, and that its recommendations are sometimes more consistent with common scientific practices. Finally, we examine the implications of our model for some common empirical settings, and illustrate with an example from economics.
Bio:
Isaiah Andrews is Professor of Economics at Harvard University, a Research Associate at the National Bureau of Economic Research (NBER), a fellow of the Econometric Society, and a coeditor at the American Economic Review. He specializes in econometrics, and his research focuses on developing methods for inference that are robust to common problems in empirical work, including insufficiently informative data (weak identification) and model misspecification. He received a MacArthur fellowship in 2020 and the John Bates Clarke Medal in 2021.
Summary:
Setting:
There is a decision problem that depends on the value of a parameter
Goal: estimate the value of this parameter
Traditional Process: data generating process -> data -> analyst -> estimate parameter/decide -> quantify error
New model: data generating process -> data -> analyst -> report that can be used by audience to infer parameters of interest within their models -> quantify error
Motivation: different objectives or different models of reality
We want to evaluate different rules for what information an analyst should communicate
Audience has many different beliefs (set A)
Report is rule c(X) X is data
Different notions of risk
Decision risk: prediction error of model using c(X) as data
Communication risk: audience member sees c(X), then
computes another function based on that to produce its own estimate of reality, AND/OR
Uses its own loss function
Then minimizes its own error based on that
Example:
If you know some set of model parameters is monotonically increasing (reality)
Experiment produces data that is not monotonically increasing
Report rule: reorder the experimental values so they’re monotonic, so they agree with constraints on the world (misordering was probably due to measurement error)
This is optimal for decision risk
But not optimal for communication risk because the distribution of error (degree/shape of misordering) is lost
General result: Decision-optimal report rules will not be communication optimal and vice versa
Extensions:
Weighted average risk/error
Maximum risk/error
Practical ways to reduce communication risk
Typical reports:
Summary statistics
Falsification tests
Auxiliary statistics
E.g. derivative of summary statistics wrt model parameters evaluated at the data point observed in experiment (this is sensitivity of the model at the point where the world is observed to be)
Example: Charitable giving
Questions: being asked to give charity is an opportunity to do what I want to do or is a social pressure to do what I don’t want to do
Treatments: solicitor visits and
No flyer before
Flyer given before informing people of visit
Opt-out flyer that is given before but allows people to avoid visit
Modeled utility as constant + disutility from social pressure (more pressure, less utility)
Observations:
Opened door,
Opting out
Amount of donation
The authors reported the sensitivity of the social pressure parameter to the treatments, stratified by outcomes
Statistics show that social pressure parameter is sensitive for people who give more money
Useful for allowing people to derive their own estimates of giving behavior and social pressure