Transparency and Communication in Scientific Research

Abstract:
We model the problem of communicating scientific information with a heterogenous audience, and contrast this problem with standard statistical optimality criteria. We show that our model sometimes delivers substantially different recommendations than standard criteria, and that its recommendations are sometimes more consistent with common scientific practices. Finally, we examine the implications of our model for some common empirical settings, and illustrate with an example from economics.

Bio:
Isaiah Andrews is Professor of Economics at Harvard University, a Research Associate at the National Bureau of Economic Research (NBER), a fellow of the Econometric Society, and a coeditor at the American Economic Review. He specializes in econometrics, and his research focuses on developing methods for inference that are robust to common problems in empirical work, including insufficiently informative data (weak identification) and model misspecification. He received a MacArthur fellowship in 2020 and the John Bates Clarke Medal in 2021.

Summary:

Setting:
- There is a decision problem that depends on the value of a parameter
- Goal: estimate the value of this parameter
- Traditional Process: data generating process -> data -> analyst -> estimate parameter/decide -> quantify error
- New model: data generating process -> data -> analyst -> report that can be used by audience to infer parameters of interest within their models -> quantify error
  - Motivation: different objectives or different models of reality
We want to evaluate different rules for what information an analyst should communicate
- Audience has many different beliefs (set A)
- Report is rule c(X) X is data
- Different notions of risk
  - Decision risk: prediction error of model using c(X) as data
  - Communication risk: audience member sees c(X), then
    - computes another function based on that to produce its own estimate of reality, AND/OR
    - Uses its own loss function
    - Then minimizes its own error based on that
- Example:
  - If you know some set of model parameters is monotonically increasing (reality)
  - Experiment produces data that is not monotonically increasing
    - Report rule: reorder the experimental values so they’re monotonic, so they agree with constraints on the world (misordering was probably due to measurement error)
    - This is optimal for decision risk
    - But not optimal for communication risk because the distribution of error (degree/shape of misordering) is lost
- General result: Decision-optimal report rules will not be communication optimal and vice versa
- Extensions:
  - Weighted average risk/error
  - Maximum risk/error
Practical ways to reduce communication risk
- Typical reports:
  - Summary statistics
  - Falsification tests
  - Auxiliary statistics
- E.g. derivative of summary statistics wrt model parameters evaluated at the data point observed in experiment (this is sensitivity of the model at the point where the world is observed to be)
- Example: Charitable giving
  - Questions: being asked to give charity is an opportunity to do what I want to do or is a social pressure to do what I don’t want to do
  - Treatments: solicitor visits and
    - No flyer before
    - Flyer given before informing people of visit
    - Opt-out flyer that is given before but allows people to avoid visit
  - Modeled utility as constant + disutility from social pressure (more pressure, less utility)
  - Observations:
    - Opened door,
    - Opting out
    - Amount of donation
  - The authors reported the sensitivity of the social pressure parameter to the treatments, stratified by outcomes
  - Statistics show that social pressure parameter is sensitive for people who give more money
  - Useful for allowing people to derive their own estimates of giving behavior and social pressure