Post date: Apr 12, 2021 1:32:43 AM
This thinkover attempts to identify principles by which to guide (and judge) the use of uncertainty analyses in risk assessments.
There have been several attempts to identify such principles for environmental risk assessments. Important publications include Morgan and Henrion (1990), Burmaster and Anderson (1994), Firestone et al. (1997), APHIS (2001), Hart et al. (2004), and various “framework” and guidance[1] documents from US EPA. Some of these reviews of guiding principles have become obsolete after recent methodological advances, and some contain ideas that appear misguided or unnecessary. A cogent and concise statement of the principles that should guide the specification and development of environmental risk assessments would be timely. The problems that are usually addressed by environmental risk assessments can be broadly categorized into four forms (cf. Anne Sergeant’s list of three):
1) What consequences will result from some new insult to the environment? For instance, what will happen if we, for example, grant this permit, build this facility, do nothing at this contamination site, etc.? Here, one starts with some information about stressors, and uses environmental risk assessment to predict what might happen in the future.
2) What does the future hold for some resource of particular concern? What in the environment or in our management of the environment might be a threat to it? In this case, one starts with some information about the resource and generally uses environmental risk assessment to predict what will happen if, say, we maintain the status quo, although one might also use it to work backwards and explain how things got the way they are.
3) What has to be done to protect some resource from some insult? This problem requires the planning of some remediation or mitigation strategy. The problem is different from the previous one which asked what would happen because it asks how to ensure a particular outcome.
4) What caused this effect in the environment? In this case, one starts with effects data and uses environmental risk assessment retrospectively to figure out what could have caused the effects that are observed. This is an investigative or perhaps a forensic use of the analysis.
In addressing these common problems, analysts employ many different kinds of mathematical methods, including uncertainty propagation, sensitivity analysis, backcalculation, calibration, model evaluation, decision analysis, compliance determination, visualization, etc. Below are several guiding principles that should govern the formulation, analysis and communication of (forward) risk assessments. They assume that a particular model structure has already been chosen for the problem, whether by regulatory precedent or fiat. If the risk assessment includes the selection of the model too, there would be additional principles to consider. These guiding principles cover some of the most important issues and standards of practice.
The following are proposed principles for discussion. Are these reasonable? Complete? Over-specific?
Ask a question that’s both relevant and specific.
Be prepared to rephrase a question that can’t be answered.
Analyze uncertainty.
You can always do an uncertainty analysis.
Having too little empirical information is surely not an argument to skip the uncertainty analysis.
Poor data may require using non-probabilistic methods.
Use bounding rather than intentionally biasing numerical values to account for incertitude.
Biasing cannot simultaneously reveal how bad and how good the outcome can be.
Avoid entanglement in purely mathematical problems.
Real risk assessments should not involve distributions with infinite ranges.
Don’t worry about results that hinge on a set being “closed” or “open”.
Admit what you don’t know.
Characterize the uncertainty for every input parameter.
Respect and use all available data, without believing their hype.
Do not assume the observed range is the possible range.
Do not assume a precise distribution without sufficient empirical or theoretical justification.
Don’t set “unimportant” variables to constants; but you can characterize their uncertainty coarsely.
Characterize the interaction or dependence among all the parameters.
Do not assume independence among variables without theoretical justification
Do not assume variables are merely correlated without reasonable justification.
Take account of any impossibility of certain combinations.
If possible, insist on empirical confirmation.
Characterize the uncertainty about the model itself, including its structural form.
Express model uncertainty as parametric uncertainty when possible.
Use prediction intervals rather than confidence intervals from regressions.
Account for the different kinds of uncertainty.
Represent incertitude with bounding or constraint propagation methods.
Incertitude includes plus-and-minus ranges and other forms of measurement uncertainty.
Incertitude includes data censoring and uncertainty arising from laboratory non-detects.
Incertitude usually includes doubt about model structure and other kinds of scientific ignorance.
Model variability using probability methods (i.e., mixture models).
Variability usually includes spatial variation and temporal fluctuation.
Variability usually includes genetic differences and heterogeneity among individuals.
Variability usually includes inconsistencies in fabrication and sometimes material nonuniformity.
Variability usually includes any of the consequences of natural stochasticity.
Treat incertitude and variability separately, and differently.
The uncertainty in inputs should not confuse the two kinds of uncertainty.
Treating incertitude as variability is much worse than treating variability as incertitude.
The convolutions of the risk model should preserve the two kinds of uncertainty in the outputs.
Avoid making assumptions.
The more assumptions you make, the less credible your conclusions are.
Just because a model is simple doesn’t mean it’s not making strong assumptions.
Linearity is a very strong assumption.
Independence is a very strong assumption.
Lognormality, normality, uniformity and triangularity are very strong assumptions.
Relax strong assumptions that are not supported by evidence or argument.
Non-parametric and distribution-free techniques avoid making assumptions.
Bounding and enveloping avoid making assumptions.
Fréchet inequalities and Fréchet bounds avoid making assumptions about dependence
What-if scenarios avoid making assumptions about models.
Discharge untenable assumptions.
Sensitivity analyses can be used to mollify an assessment.
Don’t average together incompatible models or use Bayesian model averaging.
Audit the analysis.
Check the correctness of the structure of the model.
Does the model obey the rules of dimensional soundness and do the units of parameters conform?
Is there a population or an ensemble explicitly specified for every distribution?
Are the ensembles conformant among combined distributions?
For instance, is spatial variation never confused or combined with temporal variation?
Are there no repeated uncertain parameters that would fallaciously inflate uncertainty?
Are there no multiple instantiations of probability distributions that underestimate uncertainty?
Are you dividing by or logging distributions that can take zero as a value?
Do the moments specifying each distribution satisfy the positive semi-definiteness condition?
Do the distributions obey other moment-range constraints (e.g., min £ max; variance £ range2/4)?
Is the matrix of correlation coefficients positive semi-definite?
Do correlations conform with functional relationships (e.g., if C=A+B, C isn’t independent of A)?
Does the structure of the model make sense and conform with scientific knowledge?
Does a food web model have a topology lacking intervals sensu Cohen?
Does the support of each uncertain number jibe with the theoretical range of its parameter?
Check the faithfulness of the model’s implementation.
Check the range of the outputs against a range analysis of the inputs.
Check the mean and variance of the outputs against a moments analysis of the inputs.
Was the random seed randomized? Does another value yield qualitatively similar results?
Are range checks satisfied?
Make the analysis as transparent as possible to reviewers.
Specify the intention[2] of the analysis.
Spell out all assumptions and indicate their likely qualitative influence on the results.
Explicity list all variables with their units and specify the uncertain numbers used to model them.
Recount how the correlations and dependencies among variables were modeled.
Explicitly state the model used to combine them, giving pseudocode or actual code where useful.
Say what propagation approach [3] was used.
Describe checks and sensitivity analyses employed.
Express answers (whether distributions, bounds or bounds on distributions) graphically.
Answer the “so what?” questions.
Speak to the specific concerns of the manager or the public.
Explain the import of the results obtained.
Quantitatively characterize the robustness of the conclusions.
Admit what you didn’t know.
Although such honesty can sometimes engender mistrust, it is the only sustainable approach.
Always ask the audience for help and avoid suggesting you know best.
[1] The earliest EPA guidance was abstract and written more like the guiding principles one finds in a constitution than detailed statements common in regulatory guidance today. For instance, the original discussion within EPA of the two-stage framework for cancer risk assessment, which preceded any other agency's consideration, is recounted in (Albert et al. 1977). The guidelines were published in their entirety in the Federal Register in 1976.
[2] Project uncertainty in a risk estimate, explore possible remediation strategies, choose among management strategies, calculate a cleanup goal (remediation target), evaluate compliance to some risk goal or criterion, determine how best to allocate future empirical efforts, calibrate or validate a model for use elsewhere, etc.
[3] Probability theory (Monte Carlo simulation or analytical derivation), interval analysis / worst-case analysis, possibility theory / fuzzy arithmetic, probability bounds analysis / Dempster-Shafer theory / evidence theory / random sets, imprecise probabilities, combination or hybrid methods, etc.