See also Manuscripts from Adam.
Notes on advanced bias correction
Bias is traditionally conceived as a systematic error in an estimate that can be characterized as a simple, scalar signed value that can removed from the estimate by subtraction
there are several problems:
misestimation
self-interested bias
overconfidence
unacknowledged imprecision
nakedness (either extreme overconfidence, or lack of any expression or appreciation of uncertainty)
Bias can be distributional, or p-boxy
We shall use the phrase "naked estimate" to refer to an estimate of some quantity that has been offered as a scalar value without a quantitative characterization of the uncertainty. <<This continues the clothing metaphor, and could be less confrontational than Adam's phrase "arrogant estimate".>>
What about the notion of “underconfidence” (hesitancy, corruption) here?
There are several strategies that can be used to characterize the unstated uncertainty about a naked estimate, including
1) significant digit conventions,
2) interpretation of modifying hedge words, and
3) a validation study of previous estimates.
The third way involves assuming that the uncertainty that should ascribed to the current naked estimate can be estimated by a validation study in which an observed distribution of errors (differences) is computed from comparisons of historical forecasts of quantities against their eventually realized values.
Ways to account for and remove from estimate distributions the uncertainty due to a confounding measurement process
Several studies have established that estimates produced by experts and lay people alike are commonly biased as a result of self-interest on the part of the persons making the estimates. For example, bids made by contractors under cost-plus-fee contracts regularly underestimate the actual costs of a project. Likewise, economic estimates of compliance costs of industrial regulation commonly overestimate the eventual true costs.
The self-interest
When the estimates include expressions of uncertainty, they are also usually overconfident, that is, the uncertainties are smaller than they ought to be.
Although simple scaling or shifting corrections are widely used, much better information is commonly available and fully using this information creates corrected estimates that properly express uncertainty and make them more suitable for use in risk analysis and decision making.
To account for the other information, these advanced corrections express biases as distributions or probability boxes rather than simple scalar values.
Corrections can be made in two distinct ways.
In the first way, an empirical distribution or p-box of errors (collected by an ancillary study) is convolved with each observed value.
In the second way,
The first way acknowledges uncertainties and puts them into the estimates. The second way removes confounding uncertainty that contaminates the measurement process.
In both cases, the structure of errors can be characterized distributionally with arbitrary complexity. For instance, they may be zero-centered or directional, symmetric or asymmetric, balanced or skewed, and precisely or imprecisely specified.
Traditionally, quantitative bias or error is characterized by a scalar magnitude, that is, a simple, single number, and the correction of bias consists of untangling a model of how this error and the underlying true value collided in the observed measurement. The model is usually additive, so the correction is a subtraction like a tare weight, or multiplicative, so the correction is a division such as the intensity of greenhouse gases in terms of CO2-equivalents, although sometimes a more general error model is needed. In the context of risk analysis, however, this traditional conception of bias correction is insufficient because we’re often estimating distributions, rather than merely scalar values. The notion of bias is, consequently, potentially considerably more complicated. For instance, it might no longer be simply a leftward or rightward displacement of the value, but could also be an under- or overestimate of the variance of the distribution of values. For the present discussion, we consider
Consider what ????
This view also generalizes the statistical conception of bias which is usually understood to be a systematic distortion (as opposed to random) of a statistic as a result of sampling
<<Scott announces on 17 August: Adam says he should be able to locate several papers with comparisons between forecasts of regulatory costs and actualized regulatory costs. He thought he could find 12 maybe, or 6 meta analyses...there's an OTA review, a report by the consulting firm Putnam, Hayes & Bartlett, something by Winston Harrington at RFF entitled "On the accuracy of regulatory cost estimates. He offered the files now accessible at Manuscripts from Adam which happened to be on his memory stick, and the email with link below.
From: Branden Johnson <brandenjohnson59@gmail.com>
To: Adam Finkel <afinkel@law.upenn.edu>
In case you haven't seen it, the draft EPA report on whether ex post regulatory costs might be higher or lower than ex ante estimates, submitted to SAB for comment (http://yosemite.epa.gov/sab/sabproduct.nsf/fedrgstr_activites/3A2CA322F56386FA852577BD0068C654/$File/Retrospective+Cost+Study+3-30-12.pdf). Mostly hedging, but FYI (citation for later article? ideas for follow-on research?)>>
So it seems to me that you are proposing a “top down” set of methods to put Humpty Dumpty back together again (in the sense of reversing entropy by reassembling the shattered egg). This is great, because in my first long paper I basically say there are two ways to fix an arrogant (naked) point estimate—either break it down into pieces and simulate the uncertainty using Monte Carlo or other methods (“bottom-up”), or assume that the uncertainty in this thing is roughly of the same size as all the uncertainties in like things previously estimated. You are going further than this, by fixing both the “mean” and “variance” (location and scale) parameters together.
It seems like there should be a literature on how you model “surprise” (Schlaykhter?) and trends (see attached article by Gritsevskii—there must be much newer ones than this…)
Again, the more you can use COST examples—esp if you can construct an example of a risk estimate that needs no ABC or minimal ABC, and how doing a major ABC on the accompanying cost estimate can greatly improve our understanding of new benefit—the better.
<<
Given that analysts and decision makers start to account for the self-interested biases by partially discounting statements, the interests on both sides may create an escalating war of discounting and further exaggeration. A similar situation arises in traffic safety risk analysis. Green lights at traffic intersections are sometimes delayed by traffic engineers because some drivers do not respect yellow signals to clear the intersection. However, once aggressive drivers notice that their red lights are not associated with moving cross traffic, they tend to expand their transgressions and proceed through recently changed red lights. Indeed, the longer the time during which signals in both directions are red, the more likely it is that some driver will violate a red signal. So how can such escalation be prevented? Validation is the answer <<>>.
>>
Fascinating, Captain! So at least arguably, the conventional wisdom that risk assessors exaggerate(d) their estimates (the whole “conservatism” brouhaha) may have led economists to follow suit! Certainly it’s worth making the point that ABC needs to be applied on both sides of the ledger.
*******************
Notes on the Ludic exercise:
I didn’t keep good notes of this part of the discussion—sorry. I remember suggesting a couple of variants, including:
1. convert one person’s word to a number, then show it to a second person who translates it back into a word. For example, does my “about” return something I would describe as “maybe” when someone else sees it?
2. convert one person’s word to a pdf or range, and then see what word a second person would assign to it.
I’m not sure if we ever got around to grounding this paper in the subject of the grant—so of course if you can not only use some cost/money examples, but draw generalizations from the work about how agencies should describe reg’y costs and their uncertainties so laypeople understand them in a common way, that would be terrific.