Accounting for uncertainty in inferences and calculations
Across engineering there is almost always uncertainty in quantitative calculations and risk assessments that arises from imprecise measurements, constrained sample sizes, and limited or imperfect understanding of the underlying physical phenomena. In some cases, these uncertainties are too large to ignore or neglect. But the proper characterisation of the uncertainties and the correct methods for projecting them through calculations have always been controversial, and several disparate approaches have been suggested. This page identifies some challenge problems involving the inference and projection of uncertainty through quantitative expressions. How should we be accounting for these uncertainties?
Many methods for estimating quantities and characterising their attendant uncertainty have been suggested, including confidence intervals, matching moments, maximum likelihood, maximum entropy, and Bayesian methods, as well as many other contenders (see the on-going project to review such methods). Although there are plenty of strong opinions on the matter, it is fair to say that a universal consensus has not yet formed about how to construct inputs for use in probabilistic uncertainty analyses when data is limiting. The purpose of the challenge problems is to illustrate the use of each method on various issues that analysts often face. This on-line collaboration intends to be a virtual and continuing round-table discussion on the advantages and disadvantages of various approaches in the face of commonly arising uncertainties. The purpose of this exercise is to highlight the practical differences among various approaches.
These challenge problems are simple and unrealistic, but they have been formulated to represent several fundamental issues in uncertainty quantification. Agreeing on practical answers to these problems is important in developing a workable and appropriately universal approach to handling uncertainty in science and engineering. If we cannot agree about the correct answers to these simple problems, then it is surely premature to be addressing more complex problems.
Everyone is invited to contribute to this discussion, and solutions to the challenge problems as well as suggestions about the challenge problems themselves are most welcome. Terse contributions will typically be appended as comments at the bottom of this page, but more elaborate contributions, including alternative solutions to the challenge problems, may merit their own affiliated page. Contributors retain copyright to their submissions. Contributions will be moderated for relevance.
Keywords: uncertainty, uncertainty propagation, uncertainty quantification, inferential uncertainty, risk analysis, challenge problems, digital twins
Challenge problems
Uncertainty analysts face a host of difficult problems in estimating the inputs to be used in real-world quantitative assessments. Many disparate approaches have been proposed and, in practice, analysts use multiple, sometimes incompatible methods, even within a single assessment. The challenge problems below are designed to highlight the differences, strengths, and weaknesses among the various approaches on offer. These problems seek solutions to make practical use of sample data and constraint information in several elementary estimation and projection calculations.
Everyone is invited to contribute to this discussion, and solutions to the challenge problems as well as suggestions about the challenge problems themselves are most welcome. Terse contributions will typically be appended as comments at the bottom of this page, but more elaborate contributions, including alternative solutions to the challenge problems, may merit their own affiliated page. Contributors retain copyright to their submissions. Contributions will be moderated for relevance.
What can be said about the probability of an event given only that it has never been observed in 6 independent trials? The event is related to a novel device for which observations are very expensive to collect.
What can be said about a uniformly distributed random variable x for which we have observed seven random values: 0.0336, 0.0359, 0.0332, 0.0449, 0.0116, 0.0177, and 0.0186? The experimentalist professes no other knowledge about the parameters for the uniform distribution.
What can be said about the quantity 1 – (1 – p)n , where n is distributed as Poisson(λ), and p is a probability of an event seen exactly once in 153 random trials? The value of λ parameterising the Poisson distribution is unknown, but five random values for n have been observed: 12, 0, 21, 14, 6. Note that the distributions for n and p are not assumed to be independent, and analysts instead assert they are positively dependent.
Suppose that a random variable x = (u + v) / w, where u is distributed as uniform(α, β), v is distributed as normal(μ, σ), and w is distributed as beta(γ, δ), and the random variables u, v, and w are mutually independent, but no other direct information is available about u, v, and w or the parameters on which they depend. However, thirty random sample values of x have been observed: 0.23, 0.42, 0.32, 0.45, 0.38, 0.3, 0.36, 0.48, 0.55, 0.47, 0.48, 0.58, 0.18, 0.21, 0.27, 0.51, 0.34, 0.28, 0.27, 0.53, 0.34, 0.34, 0.28, 0.33, 0.43, 0.37, 0.18, 0.52, 0.35, and 0.43. What can be said about x? What can be said about α, β, μ, σ, γ, and δ ?
Consider a time-indexed random variable xt+1 which analysts are sure is governed by one of these two recursion equations or a function intermediate between them:
xt+1 = at / (1/bt + 1/xt)
xt+1 = xt exp(ct(1 – xt/dt))
where at, bt, ct, dt are mutually independent and also serially independent. Suppose that at is uniformly distributed over the range zero to one, bt is normally distributed with mean 10 and unit variance, and ct is an epistemically uncertain quantity between zero and one which may be varying or may be a fixed value. The random variable dt has a distribution that is unknown except analysts are sure that it ranges between zero and one and has a mean of 0.25. The initial distribution x0 is uniform over the range zero to one, but is recursively corrupted. What can be said about x1, and what can be said about x4?
Get involved
Everyone is welcome to join this discussion. Email your contribution to Scott Ferson at ferson(at)liv(dot)ac(dot)uk with the subject line "challenge problems". Your contribution may be in the form of simple text, a PDF file, slide show or other document. If you like, your contribution will be added to this site as you instruct, perhaps appended to the bottom of this page, or made into a new affiliated solution page. Your email address will not be published unless you indicate you want it to be used to sign your contribution. We also welcome suggestions to refine these challenge problems.
The website Exemplar problems offers a more elaborated if not more comprehensive set of challenge problems for uncertainty analysts. The websites at the links below (or at the links therein) provide several related sets of challenge problems from other perspectives.
An expanded version of the challenge problems discussed above
https://sites.google.com/site/exemplarproblems
Links to several other uncertainty challenge problem sets from various sources
https://sites.google.com/view/exemplarproblems/exemplar-problems#h.wscvv5btej4s
Other challenge problems specifically related to estimating probabilities
https://sites.google.com/site/beyondthebagofmarbles
Other challenge problems related to model uncertainty
https://sites.google.com/view/modelformuncertainty/home
Collaboration for a review of methods to characterise inputs for quantitative risk assessments [request to be invited]
https://sites.google.com/site/niharrachallengeproblems/challenge-problems
R (statistical computing language)
This website originated as part of the outreach efforts of the DigiTwin project, which is a research consortium funded by UK Research and Innovation (UKRI) through the Engineering and Physical Sciences Research Council (EPSRC reference EP/R006768/1, principal investigator D. Wagg). The views and opinions expressed herein and in comments below are those of the individual contributors and commenters, and should not be considered those of any of the other authors or collaborators, nor of the institutions with which they may be affiliated, nor of UK Research and Innovation, or other sponsors or affiliates. Copyrights for the contributed material and commentary remain with their respective authors.