Agenda

Fernand Gobet

https://stream.liv.ac.uk/s/tkuv7z7y

Mark Burgman

https://stream.liv.ac.uk/s/dnk2trgg

Scott Ferson

https://stream.liv.ac.uk/s/km8vchsu

Jinglai Li

https://stream.liv.ac.uk/s/hcuk6cne

Risk Institute Seminar Room, Chadwick Building, University of Liverpool

Wednesday, 13 February 2019

Thursday, 14 February 2019

Directions on campus

The Institute for Risk and Uncertainty is in the Chadwick Building at the end of Peach Street in the heart of the University of Liverpool campus. Use the south entrance to the Chadwick Building; other entrances have no access to the Risk Institute. When you enter the building, you'll see the Muspratt Lecture Theatre. Turn left and enter the brown door. Follow signs to the Seminar Room down the corridor, making two right turns.

Google Maps: https://goo.gl/maps/84yrvRFmLHE2

What3words: https://w3w.co/curiosity.memory.daring

Latitude: 53.404110 / 53°24'14.8"N, Longitude: -2.964600 / 2°57'52.6"W

For travel suggestions, see https://riskinstitute.uk/findus.

Abstracts and biographical sketches

Elicitation, subjective but scientific

Anthony O'Hagan

Science can never be objective because subjective judgements are unavoidably made at all stages of every scientific endeavour. But science strives to be as objective and rigorous as possible. This talk will be about eliciting expert knowledge as objectively and rigorously as possible. I will concentrate on the SHELF protocol but will highlight points of agreement and difference with other leading protocols.

Anthony O'Hagan is an emeritus professor at the University of Sheffield. His research focusses on the theory and application of Bayesian statistics, in particular in the elicitation of expert knowledge, managing and quantifying uncertainty in the use of complex mechanistic modelling and Bayesian modelling. He has done extensive research and consulting activities in many application areas, particularly in medicine, environmental science, asset management, and health economics. On the topic of expert elicitation, he is responsible for a number of significant developments. With his extensive experience in research, teaching and practice, he is one of the foremost experts in the field.

Eliciting subjective priors for health care decision making

Laura Bojke, York University

Health care decision problems are generally complex. Not least because they typically involve a number of alternative courses of action, but also because a range of health outcomes and cost implications may need to be considered. Such decisions are often informed using cost-effectiveness modelling, however, evidence to inform such models is often uncertain. Subjective priors in the form of formal expert opinion can provide valuable information to inform such assessments, particularly where evidence is missing, less developed or less appropriate. This presentation attempts to summarise the available guidance on expert elicitation for health care decision making. The choices available and complexities arising are illustrated using applied examples in the area.

Laura Bojke is a reader in Health Economics, at the Centre for Health Economics, University of York. Laura has worked in economic evaluation for over 18 years and has considerable experience of both applied and methodological studies. She has worked as part of an assessment team for the National Institute for Health and Care Excellence since it began in 1999 and also co-leads the Health Economics and Outcomes Measurement team as part of the Collaboration for Leadership in Applied Research and Care (CLAHRC) for Yorkshire and Humber. The program of work includes the use of economic evaluation across sectors, economic evaluation for local decision makers and the use of routine data for economic evaluation. Laura has worked on a number of projects involving the use of expert elicited data within decision analytic models and is particularly interested in this area and its application to issues of extrapolation uncertainty. She currently leads an MRC funded project to develop a protocol for expert elicitation in health care.

Making the most of mental models: advancing the methodology for mental model elicitation and documentation with expert stakeholders

Kelsey LaMere, University of Helsinki

Eliciting expert-stakeholders’ mental models is an important participatory modelling (PM) tool for building systems knowledge, a frequent challenge in natural resource management. Therefore, mental models constitute a valuable source of information, making it imperative to document them in detail, while preserving the integrity of stakeholders’ beliefs. We propose a methodology, the Rich Elicitation Approach (REA), which combines direct and indirect elicitation techniques to meet these goals. We describe the approach in the context of climate change’s effects on Baltic salmon. The REA produced holistic depictions of mental models, including more variables and causal relationships than direct elicitation alone, thus providing a solid knowledgebase to begin PM studies. The REA was well received by stakeholders, provided an educational experience, and fulfilled the substantive, normative, and instrumental functions of PM. However, motivating stakeholders to confirm the accuracy of their models after completing the REA presented a challenge.

Kelsey LaMere is a PhD candidate from the University of Helsinki's Doctoral Program for Interdisciplinary Sciences (DENVI). Her research focuses on the effects of climate change on Baltic salmon and how we can use stakeholder knowledge to understand the problem. She has also have been researching risk communication within interdisciplinary scientific teams. Her background is in zoology and natural resource management and is interested in the involvement of stakeholders in the scientific process, science communication, risk assessment, and the science-policy interface.

Expert elicitation: don’t forget the expert!

Fernand Gobet, University of Liverpool

The field of expert elicitation has developed a number of elegant methods for eliciting and mathematically combining quantitative estimates from experts. However, the success of these methods depends on the extent to which experts can reliably provide quantitative estimates. In this talk, I will present data from psychology and expert-system research showing that accessing expert knowledge is often difficult. I will argue that this is in part due to the fact that expert knowledge is perceptual and implicit, and hence outside of experts’ awareness. Implications for expert elicitation will be drawn.

Fernand Gobet is professor of psychological sciences in the Institute of Population Health Sciences. His research focusses on cognitive science, cognitive psychology, cognitive neuroscience, artificial intelligence, education, and philosophy. He brings the psychological perspectives on human expertise, how this reflects basic traits such as personality and intelligence, as well as knowledge and skills acquired through training. His publications include many books and scholarly articles on expert elicitation.

Expert judgement, Delphi, ACE and CREATE

Mark Burgman, Imperial College London

Expert judgement is essential when decisions are imminent and the requisite data and models are unavailable or unobtainable. This presentation outlines the development of a Delphi procedure, termed IDEA. I describe its origins in biosecurity applications and its further development and testing in the IARPA ACE project. This work identified some important general lessons. Finally, I describe the development of the IARPA CREATE program, and the SWARM platform, aimed at improving the quality of reasoning and communication of expert groups.

Mark Burgman is director of the Centre for Environmental Policy at Imperial College London. His research focusses on expert judgement, ecological modelling, conservation biology, and risk assessment. He has done practical and theoretical research in application areas such as biosecurity, medicine regulation, marine fisheries, forestry, irrigation, electrical power utilities, mining, and national park planning. On the topic of expert elicitation, he has contributed extensively to the field combining psychologically robust interactions among experts with mathematical aggregation of individual estimates, structured elicitation protocols to improve the accuracy of expert judgement, and examination of trade-offs of group-based versus individual elicitation. Burgman has advised government agencies on the use of expert elicitation in policy decisions.

Naked expert elicitations of probabilities of rare events

Scott Ferson, University of Liverpool

There are two approaches to estimating probabilities when there are no data: expert elicitation (i.e., guessing), and disaggregation into constituent parts whose probabilities are easier to estimate (i.e., breaking into subproblems). When the latter approach is no longer workable, analysts must resort to the former and rely on expert opinion and estimation. But how should we characterize probabilities of events that are so rare that they have never been observed? By what principles can such characterizations be projected in probabilistic analyses? Sometimes elaborate elicitation strategies are employed to estimate rare-event probabilities, but the results are often expressed as probabilities with no indication of the uncertainty associated with the estimate. How might analysts model expert opinions about event probabilities of the form “1 in 10 million”, “about 1 in 1000”, or “it’s never seen in over 100 years of observation”, so they can be used in calculations that account for rather than ignore epistemic uncertainty?

Scott Ferson is director of the Institute for Risk and Uncertainty at the University of Liverpool. His recent research has focused on developing methods and software to solve quantitative assessment problems when data are poor or lacking and structural knowledge about the model is severely limited. Ferson's contribution to the field of expert elicitation includes quantitative decoding of natural-language words used to express uncertainty ('hedges'), estimation of rare-event probabilities without data, and extensions of probability theory to deal with analyses with little or no data.

A surrogate accelerated multicanonical Monte Carlo method for uncertainty quantification

Jinglai Li, University of Liverpool

In this work we consider a class of uncertainty quantification problems where the system performance or reliability is characterized by a scalar parameter y. The performance parameter y is random due to the presence of various sources of uncertainty in the system, and our goal is to estimate the probability density function (PDF) of y. We propose to use the multicanonical Monte Carlo (MMC) method, a special type of adaptive importance sampling algorithms, to compute the PDF of interest. Moreover, we develop an adaptive algorithm to construct local Gaussian process surrogates to further accelerate the MMC iterations. With numerical examples we demonstrate that the proposed method can achieve several orders of magnitudes of speedup over the standard Monte Carlo methods.

Jinglai Li is a reader in Mathematical Sciences at teh University of Liverpool. He received the B.Sc. degree in applied mathematics from Sun Yat-sen University and the PhD degree in mathematics from SUNY Buffalo. He did postdoctoral research at Northwestern University and MIT. He was an associate professor at Shanghai Jiao Tong University. His current research interests are in scientific computing, computational statistics, uncertainty quantification, data science and their applications in various scientific and engineering problems.

Page updated

Google Sites

Report abuse