Exemplar problems

Challenge problems for uncertainty assessments from sparse information

Uncertainty analysts face a host of difficult problems in estimating the inputs to be used in real-world quantitative assessments. Many disparate approaches have been proposed and, in practice, analysts use multiple, sometimes incompatible methods, even within a single assessment. The seventeen exemplar problems below are designed to highlight the differences, strengths, and weaknesses among the various approaches on offer. These problems seek solutions to make practical use of sample data, constraint information, and expert elicitations in several elementary estimation and projection calculations. The problems are very simple, but they are exemplary in that, if we cannot agree about what the correct answers to them would be, then surely it is premature to address more complex problems.

Everyone is invited to contribute to this discussion, and solutions to the challenge problems as well as suggestions about the challenge problems themselves are most welcome. Terse contributions will typically be appended as comments at the bottom of this page, but more elaborate contributions, including alternative solutions to the challenge problems, may merit their own affiliated page. Contributors retain copyright to their submissions. Contributions will be moderated for relevance.

The problems are designed to be read by R. The first line of each problem states the assumptions in a comment (anything after a hashtag #). The tilde ~ is read "is distributed as". Additional relevant information in the form of random sample data, constraint information or expert judgements are expressed below the first line. In R, the c() function constructs a vector or list, so the x.1 variable is a list of six zeros.

The solution to each problem will quantify what can be said, given only these assumptions and data, about the variable x and any parameters it may depend on.

# 1 x ~ Bernoulli(p)

x.1 = c(0, 0, 0, 0, 0, 0)

# 2 x ~ exponential(λ), where E(x) = 1/λ

x.2 = c(2.16, 0.447, 8.13, 0.006, 0.279)

# 3 x ~ binomial(n,p)

x.3 = c(6, 7, 5, 7, 7, 5, 6, 5)

# 4 x ~ normal(μ, σ)

x.4 = c(24.5, 22.5, 21.1, 23.5, 22.3, 23.6, 22.4, 23.4)

# 5 x ~ uniform(min, max)

x.5 = c(0.0336, 0.0359, 0.0332, 0.0449, 0.0116, 0.0177, 0.0186)

# 6 x ~ continuous distribution

x.6 = c(140, 121, 154, 163, 137, 216, 118, 167, 165, 129, 151, 214)

# 7 x ~ distribution whose minimum, maximum and mean are known (or fluctuate) within intervals

x.7 = list(min=c(90,162), max=c(870,940), mean=c(140,222))

# 8 x ~ unimodal distribution with several constrained parameters

x.8 = list(min=0, max=1, mode=0.1, mean=c(0.2,0.4)) # mean is an interval

# 9 x ~ unimodal continuous distribution

x.9 = x.6

# 10 x ~ exponential(λ), where E(x) = 1/λ

x.10 = c(I(11.8, 14.3), I(1.9, 2.3), I(8.8, 10.6), I(0.7, 1), I(5, 6), I(2.7, 3.2)) # 6 points interval censored

# 11 x ~ truncated normal(mu, sigma), only values above 7.3 can be observed

x.11 = c(7.62, 8.38, 7.41, 8.33, 10.20, 7.67)

# 12 x ~ normal(mu, sigma)

x.12 = x.4 # but values came from nonrandom, nonrepresentative "hotspot" sampling which yields elevated values

# 13 x ~ a + b

a.13 = c(1.2, 2.4); b.13 <- c(4, 5.7) # addends are known to be in the respective interval ranges

# 14 x ~ y + z, with y~normal(μ,σ) positively dependent with z~lognormal(m,s) where E(z)=m, V(z)=s2

y.14 = x.4

z.14 = c(12.1, 6.45, 73, 24.6, 15.2, 44.3, 19.0)

# 15 x ~ (u + v) / w, u ~ uniform(α, β), v ~ normal(μ, σ), w ~ beta(γ, δ), independent u, v, and w

x.15 = c(0.23, 0.42, 0.32, 0.45, 0.38, 0.3, 0.36, 0.48, 0.55, 0.47, 0.48, 0.58, 0.18, 0.21, 0.27, 0.51, 0.34, 0.28, 0.27, 0.53, 0.34, 0.34, 0.28, 0.33, 0.43, 0.37, 0.18, 0.52, 0.35, 0.43)

# 16 x ~ 1 – (1 ‒ p)n , where n ~ Poisson(λ), and p is a probability of an event seen once in 153 trials

N.16 = c(8, 15, 16, 17, 10) # note that n and p are not assumed to be independent

# 17 x ~ y × z, where y ~ uniform(min, max), z ~ exponential(λ), E(z)=1/λ, with bivariate data

y.17 = c(3.5, 8.86, 10.6, 5.3, 6.1, 7.4, 7.87, 5.2, 4.4, 12.8, 8.7, 7.5, 9.7, 14, 11)

z.17 = c(0.79, 0.2, 0.116, 0.67, 0.53, 0.29, 0.25, 0.47, 1.24, 0.1, 0.202, 0.394, 0.272, 0.0396, 0.14)

Problems 1–5 involve only random sample data with an assumed distribution shape. These are elementary but fundamental problems. Problem 6 is nonparametric in that no distributional assumption at all is offered other than its continuity. Problems 7–8 involve only constraint information without any sample data or basic structural information, and may not allow traditional assumptions such as random sampling, stationarity, independence, and negligible measurement uncertainty. Problems 9–12 have both sample data and some constraint information, and include problems with data censoring and data truncation. Problems 13–17 are more elaborate and involve multiple variables, dependence issues and propagation through convolutions or compositions.

Background

Estimation. Among the first steps in any quantitative uncertainty assessment is the characterisation of relevant variables and constants that will be the inputs. Estimation is the basic problem of how to integrate available information about the quantities to be used as calculation inputs. There is a huge literature on this subject and many schools of thought about how to best make estimates. There are several basic approaches, including simple direct observation or measurement, analysis of random sample data, constraint propagation, modelling, and assumption. Some argue that estimation involves all of statistical science, or at least a good chunk of it. Several other terms are used for or within this process, including parameter identification, expert elicitation, data fusion, aggregation, and simply data analysis. And there are genuine differences of opinion and quantitative discrepancies across the schools of thought, even for the simple special case of analysing random sample data.

Aggregation. When multiple disparate sources of information are available, particularly in different or incommensurate formats, estimation includes more elaborate considerations under what might be called aggregation or sometimes data fusion. In such cases, judgments must be made about whether to include data sets, and with what relative weightings, and what aggregation operation will be used to synthesise the sources of information. Traditional operations like averaging or stochastic averaging (mixture models) are known to create biases that could lead to serious miscalculations so these considerations deserve some care.

Dependence. Estimating the various quantities to be used as inputs is usually only part of an uncertainty analysis. It is generally important to model stochastic dependencies among distributions used as inputs in quantitative assessments, as fallacious assumptions of independence can in some circumstances lead to underestimation of tail risks, i.e., the probabilities of extreme outcomes which are often the central focus in a risk assessment. The characterisation of input distributions includes the responsibility to assess and report intervariable correlations or dependencies, and to recommend methods to replace or relax unwarranted independence if necessary.

Expert knowledge. The exemplar problems using constraint or structural information may be considered problems involving trustworthy expert knowledge or opinions, the results from expert elicitation after all the possible complexities of the elicitation process have been worked out. Note that there is a separate project (led by Caroline Morais and Pedro Silva) on wrangling information gleaned from experts, and that project may be developing its own separate set of challenge problems that consider how to characterise the uncertainties that should be ascribed to opinions, whether and how multiple experts’ opinions should be aggregated, whether and how to weight different experts, and how to address the issue of non-commutativity of aggregation and propagation.

Many methods. There is a bewildering array of popular methods for estimating quantities and simultaneously characterising the attendant uncertainty, including

Confidence intervals or rigorous intervals, Method of matching moments,

Maximum entropy, Bayes and robust Bayes analysis,

Empirical or bootstrap distributions, Expert elicitations,

Maximum likelihood, Info-gap or possibility distributions,

Polynomial chaos expansions, Probability boxes,

Second-order distributions, Constraint propagation,

Confidence distributions, and Confidence structures.

There are, in fact, many more contenders. (See the on-going project to review this diversity of methods at https://sites.google.com/site/niharrachallengeproblems/.) Although there are plenty of strong opinions on the matter, it is fair to say that a consensus has not yet formed about how to construct inputs for use in probabilistic uncertainty analyses when data is limiting. The purpose of the exemplar problems is to illustrate the use of each method on various issues that analysts often face.

Contribute your own solution

Everyone is welcome to join this discussion. Email your contribution to Scott Ferson at ferson(at)liv(dot)ac(dot)uk with the subject line "challenge problems". It may be in the form of simple text, PDF file, slide show or other document. Your contribution will be added to this site as you direct, perhaps appended to the bottom of this page, or maybe made into a new affiliated solution page. Your email address will not be published unless you indicate you want it to be used to sign your contribution. Suggestions of other challenge problems about estimation uncertainty are welcome, as are comments about the problems detailed above.

Links to related challenge problems

Uncertainty Problems

An abbreviated introduction to the exemplar problems

https://sites.google.com/view/uncertaintyproblems

Beyond the Statistician's Bag of Marbles

Other challenge problems related to estimating probabilities

https://sites.google.com/site/beyondthebagofmarbles

Model-form Uncertainty

Other challenge problems related to model uncertainty

https://sites.google.com/view/modelformuncertainty/home

Wrangling Epistemic Uncertainty, Ten Little Problems

Other challenge problems for the thinkover on epistemic uncertainty

https://sites.google.com/site/epistemicunc/thinkover/challenge-problems

F&PIG Exemplar Problems

Characterising uncertainty with consonant possibility structures

https://sites.google.com/view/fnpig/exemplar-problems

NASA LaRC UQ Challenge 2014

https://uqtools.larc.nasa.gov/nda-uq-challenge-problem-2014/

https://uqtools.larc.nasa.gov/wp-content/uploads/sites/17/2017/10/uq_talk.pdf

NASA Langley UQ Challenge on Optimization Under Uncertainty

https://uqtools.larc.nasa.gov/nasa-uq-challenge-problem-2020/?doing_wp_cron=1661332264.6769719123840332031250

NAFEMS Stochastic Challenge Problems

Sequence of challenge problems by the Stochastics Working Group promoting engineering simulation
https://www.nafems.org/community/working-groups/stochastics/challenge_problem/

https://www.nafems.org/blog/posts/uncertain-knowledge-a-challenge-problem/

Black-box Reliability Challenge 2019

https://rprepo.readthedocs.io/en/latest/reliability_problems.html

https://rprepo.readthedocs.io/en/latest/introduction.html

Sandia Verification and Validation Challenge Workshop

https://asmedigitalcollection.asme.org/verification/article/1/1/010301/369065/Introduction-The-2014-Sandia-Verification-and

Sandia Epistemic Uncertainty Workshop (2002)

Original workshop with (a+b)a and the mass–spring–damper challenges

https://www.sciencedirect.com/science/article/abs/pii/S0951832004000493

https://www.sciencedirect.com/science/article/abs/pii/S0951832004000729

Characterizing Distributions [website by invitation]

Grand collaboration on estimation methods in risk analysis

https://sites.google.com/site/niharrachallengeproblems

https://sites.google.com/site/niharrachallengeproblems/challenge-problems

https://sites.google.com/site/niharrachallengeproblems/project-definition

General links

- Society for Risk Analysis

Acknowledgements and disclaimers

This website originated as part of the outreach efforts of the DigiTwin project, which is a research consortium funded by UK Research and Innovation (UKRI) through the Engineering and Physical Sciences Research Council (EPSRC reference EP/R006768/1, principal investigator D. Wagg). The views and opinions expressed herein and in comments below are those of the individual contributors and commenters, and should not be considered those of any of the other authors or collaborators, nor of the institutions with which they may be affiliated, nor of UK Research and Innovation, or other sponsors or affiliates. Copyrights for the contributed material and commentary remain with their respective authors.

challenge problems introduction.ppt

Introduction [Ferson]

Your contribution

...

Page updated

Google Sites

Report abuse