11. expert elicitation

Statistical Expert elicitation involves quantifying subjective and implicit expert knowledge and is useful when there is insufficient data available to quantify model parameters and their uncertainty (Bojke et al., 2021). Elicitation could be used to inform the parameters for the long-term model assumptions, particularly for the intervention effects, when there is a lack of quantitative data. Multiple experts should provide input, and these should include behavioural scientists. It is important to understand dependencies between elicited parameters (e.g., different points on a survival curve) so that the dependencies can be incorporated within the model explicitly (Bojke et al., 2021).

In order to reduce bias, elicitation protocols should be followed (O'Hagan, 2019). Leading protocols include:

- the Delphi method, where individual judgements are made, a summary of all the individual judgements is shared, before one or more additional rounds of providing judgements followed by group summaries, until these are mathematically aggregated;

- the Cooke protocol, where experts individually make judgements about uncertain quantities as well as quantities known to the researcher and then the uncertain judgements are weighted by their performance on the known quantities and mathematically aggregated; and

- the Sheffield Elicitation Framework (SHELF) protocol, where individual judgements are made and then these are discussed with the group, including the reasons for differing opinions, until a consensus judgement is made.

For all of these protocols, questions should be piloted to ensure they are valid, intuitive and clear (Bojke et al., 2021). There are existing software and tutorials available to facilitate elicitation (O'Hagan, 2019).