Draft as of 30/05/19
Review on Expert Elicitation
Authors: Caroline Morais, Pedro Silva, Scott Ferson
Aim of the paper: stablish guidelines on where to use experts or data analysts
Who are the experts?
Experts are individuals with recognised knowledge or skill in a specific domain (Mosleh at al., 1988).
What is expert elicitation?
Experts opinion is sometimes the only available source when data is missing (Mosleh at al., 1988).
When expert elicitation is used?
Lack of data, lack of time to collect data or/and lack of money to collect data?
In some fields, the lack of data can be due to rare events, as major accidents in engineering (Moura et al., 2016).
In engineering the lack of data may also be due to components or systems that have never failed. (Bailey, 1997).
(add reference for other fields!)
Engineering – depends on project phase (new design: lack of data; during operation: lack of time)
In Oil & Gas industry, at the very first beginning of the design phase (called conceptual design), the most common practice is relying on expert judgement due to data gap. An example is the use of the hazard identification technique called HAZID (Hazard Identification).
On the detailed phase, with more data at hand to specify the systems and equipment, usually a quantitative analysis is conducted. At this moment, quantitative risk assessments are conducted which usually relies on dataset of components reliability and previous accidents.
However, during operation phase, the most common practice is relying on expert judgement, even if there is no data gap. This can be evidenced by the spread use of qualitative risk assessments, such as HAZOP (hazard and operability study). This seems controversial, as in changes of design during operation, there is no way for the decision makers to compare with the previous baseline delimited by the quantitative studies.
Where expert elicitation is used?
Engineering - Risk assessments (qualitative risk assessments or quantitative risk assessments with lack of data), reliability analysis
Human Reliability Analysis (to feed risk assessments):
Nevertheless, expert elicitation is sometimes considered the least credible source of data if compared to data obtained from real operating experience and data from simulators. This is identified for the field of human error data for engineering risk assessments. (Taylor and Kirwan, 1995).
Models built from expert elicitation
Ultimately, for some fields that need a model where to accommodate the data it is improbable to have an assessment that does not rely on expert judgement to some extent.
For instance, at the human reliability analysis field all the quantitative methods in use (that estimate human error probabilities) start with a qualitative analysis of the possible scenarios (Laumann et al., 2018) – which means that a model of how organisational factors propagates and triggers human errors have to elicited by experts.
Model uncertainty
The structure of a model can be a source of uncertainty (Droguett and Mosleh, 2008).
(cite American lady of the workshop)
Medical field
In the medical field, expert elicitation is commonly used when calculating health care costs and the long-term effects conditions have on the health care economy. Hadorn et al (2014) notes that when lacking empirical data, health care economists often relied upon “guesstimates” to establish parameter values. Due to the uncertainty within these “guesstimates”, economists would use sensitivity analyses to manage the arising uncertainty.
Health Economic Decision Making (HEDM) is a field that could benefit from expert elicitation, however Hadorn et al (2014) notes that of an initial analysis of 100 randomly sampled studies, 57% had used at least one parameter derived from expert elicitation. Furthermore, of these that used expert elicitation only 10% had described a formal process of expert elicitation. The rest of the estimated values were based on the authors “expertise” and followed no standard protocol or guidelines for expert elicitation.
A second analysis of 60 randomly selected studies (primarily cost-effectiveness analysis of clinical interventions) found that 62% had used external experts for parameter estimation. Of the studies that used expert elicitation, only 57% described an aspect of the expert elicitation with 40% stating that the Delphi procedure was used. Despite this, there was much variability in the way these protocols were implemented, varying in how information was presented to and acquired from experts, as well as how often these experts were contacted.
Aside from HEDM, the National Institute for Health and Care Excellence (NICE) relies on experts when developing guidelines. NICE guidelines consist of multiple programmes including diagnostic guidance, NICE guidelines, and medical technologies guidance. Using semi-structured interviews, Peele et al (2018) looked at how expert elicitation was implemented across six programmes within the NICE guidelines. Variation exists across programmes regarding expert elicitation. One example is the medical technologies guidelines that used independent external experts but were not formally made members of the decision-making committee. Conversely diagnostic guidance would use both external experts and have topic experts as part of the decision-making committee.
Additionally, NICE uses expert elicitation for multiple purposes including problem definition (including background information), the interpretation of evidence, sense checking data HEDMs, and drafting guidance. Overall the implementation of expert elicitation is rather unstructured, with different programmes using different systems. Some may use standardised questionnaires, that can be completed over the phone or through emails, some use a modified Delphi approach (in the form of an online survey), whilst others may use interactive workshops. Experts aren’t limited to professionals (for primarily quantitative data), but also lay experts (such as patients) to provide their experiences in guiding policies.
Peele et al (2018) describes availability being a common problem for expert elicitation. Suggestions for improvements included a protocol or tool that NICE can implement when soliciting experts to help save time, as well potential training to reduce bias and ensure information required fills the current evidence gap. Peele notes however that protocols and tools may need to vary between programmes due to the niche specialities each overview.
Tools such as MATCH and ExpertLens don’t fulfil the current needs of NICE guidelines, however as Peel mentions these tools can be adapted to match the criteria of NICE.
Law Enforcement
Expert Elicitation forms a core component of Law Enforcement across multiple stages. It is routinely in the formation of public policy and legislation, as well as strategy design for officers and teams.
One study by Alison et al (2012) looks at the influence criminal profiles can have on an individual’s opinion of a suspect, and subsequent judgements. These profiles are often designed by experts, often utilising empirical and data driven approaches. They exist to compliment an officer’s experienced based judgement, and as decision-makers, to act on said judgement. Profiling becomes useful as it helps minimise the potential biases inherent in the decision-maker. Previously, opinion-based approaches could result in false accusations and arrests. This is due to simplifying cognitive heuristics, applying a set of rules to a comlpex situation (Kahneman et al, 1982). An example of this can be seen in the role of US traffic stops, where race was a significant predictor, as officers would factor race into their “personal profiles” (Williams and Stahl, 2008).
According to Alison (2005) profiling techniques can be divided into “contemporary” and “traditional” approaches. Traditional profiling approaches uses assumed characteristics such as personality, demographics and lifestyle. This approach has been heavily criticised due to a lack enough evidence in its efficacy (Alison et al, 2002). Often these profiles have been found to contain irrelevant or ambiguous statements, impacting the judgements of investigators (Alison et al, 2003), and result in confirmation bias (a core problem in police investigations; Rassin et al, 2010; Kassin, 2005; Meissner and Kassin, 2002).
Alison et al (2012) used a novel approach using lay people and police officers to ascertain the impact expert advice (traditional profile) can have on an individual’s assumption of guilt. Participants were given a scenario and asked to develop a personal profile of the potential criminal. They would then gauge the potential guilt of two suspects (one orthodox and one unorthodox). Following this, participants were provided with a an “expert” profile that would match either the orthodox or unorthodox suspect and ask to re-evaluate their assumptions of guilt. If the expert profile matched the suspect, the participant would assume that the matching suspect was guiltier and revise down the assumed guilt of the other suspect. However, for unorthodox suspects, similarity between the participant’s own profile and the expert’s profile was a significant factor for increased assumption of guilt. This result is interesting, as it suggests expert advice is most influential when it is unorthodox or incongruent with the decision-maker’s own beliefs, encouraging decision-makers to re-evaluate their judgements. It has been suggested that profilers should consider the opinions of police officers, as well document the process in which the profile is developed and used.
Building on the previous study Christiansen et al (2015) examined the effect of source credibility in profiling. Using a similar approach, this study introduced two different profiles, one developed by a “novice” and one developed by an “expert”. As previously shown, the profiles presented (regardless of credibility) influence the decision-makers opinion on perceived guilt, however there was no significant difference in belief change when participants were presented with either the novice or expert profiles.
This poses a problem regarding profiling and its impact on investigations. Although it may be that participants were unfamiliar with the situation, failure to acknowledge source credibility during real world operations can hinder an investigation.
Business
How to conduct an expert elicitation?
Selection of experts
Step 1: Identifying the experts
Gustafson et al. (2003) have identified and selected the experts through a ‘snowball nomination process’, i.e. when an expert was nominated twice by his/her peers. When analysing the nominated experts, they have
Step 2: Shortlisting the experts
After identifying the possible experts for their elicitation, Gustafson et al. (2003) have shortlisted by considering a mixed panel with theoreticians and practitioners.
However, some works suggest that the quality of the expert elicitation may vary with the experts’ attributes, such as their skills and knowledge.
Step 3: Invitation of experts
(cite reference on how to invite and what is usually offered for their time. Something is stated by Gustafson, Kirwan and SHELF creators)
Expert attributes vs Performance:
Mosleh (2018) have shown that in engineering field expert’s performance have varied with the stage of their carreer (early and late career performed better than mid-career).
Kirwan (1997) have observed that experts with more than 10 years of experienced field have better validation results. On the other hand, if novice or students, the expected validity of the results is lowered. (Human reliability analysis field)
Kirwan (1997) suggests that elicitation demands also experience in the taxonomy used (Human reliability analysis field).
SHELF creators (cite reference) have suggested that they have to have experience in probability. This is provided by a short training before the elicitation (SHELF procedure)
Avoiding experts bias
Experts are usually seem as the least credible source of data, because they can be oriented by different sources of bias (Mosleh et al., 1988)
Avoiding experts Overconfidence
Experts are systematically overconfident about the accuracy of their judgments (Lin and Bier, 2008)
Multidisciplinary teams
Gustafson et al. (2003) have opted for a panel with theoreticians and practitioners. However, some researches state that for some fields it is preferable to have an even broader distinction within experts.
Elicitation meetings and procedures
On their guidance on uncertainty and use of Experts for Probabilistic Seismic Hazard Analysis, Budnitz et al. (1997) have concluded that the differences in results are due to procedural rather than technical differences. To conduct the meetings, two entities were formally defined: the Technical Integrator and the Technical Facilitator Integrator.
The same conclusion about the importance of meetings’ procedures and the role of the facilitator is highlighted by the creators of SHELF (refer to SHELF procedure, presented in the workshop).
(Cite other expert elicitation formalised procedures)
Translating qualitative answers into quantitative boundaries estimation.
Some work was conducted using uncertainty quantification to translate experts answers into probability.
Using experts to estimate conditional probabilities (e.g. in Bayesian networks)
In Bayesian networks, the Posterior Odds are a Product of Likelihood Ratios_times the Prior Odds.
“In many situations, likelihood ratios and prior odds can be estimated empirically. However, if necessary data do not exist or are insufficient, behavioral decision theorists assert that likelihood ratios can be estimated subjectively by trained experts (Edwards, Lindman, and Savage 1963; Slovic and Lichtenstein 1971; Hogarth 1975; von Winterfeldt and Edwards 1986). While it may appear that experts could directly estimate the probability of success, the cognitive burden in forming an evaluative response to the simultaneous influence of n factors (D1, D2,y, Dn) is much greater than the cognitive burden in forming an evaluative response to one factor at a time (Gustafson et al. 1993; Luke, Stauss, and Gustafson 1977). Thus, experts are better equipped in subjectively estimating n individual likelihood ratios than in subjectively estimating posterior odds given n pieces of data presented simultaneously.”(Gustafson et al., 2003).
One of the challenge of eliciting for Bayesian networks is to elicit the conditional probability distribution, because the bigger the amount of parent nodes the bigger the possibility of possible combinations (Caroline, explain better).
The cause of is due to limitations inherent to human information processing. Working memory is the system by which we retain and manipulate information in the short-term to solve problems or make decisions (Baddeley and Hitch, 1974). Limitations can be divided into two problems; storage (central capacity), and processing (bounded rationality).
Central capacity refers to the quantity of information capable of being stored within working memory at any one time. The capacity limit tends to range from three to five items (Cowan, 2010). Retaining information beyond five units requires a process known as chunking. An example given is a list of eight words:
Sky
Fall
Dog
Toy
Blind
Bird
Time
Loss
While this can be viewed as eight units of information, individuals can condense this list into four “chunks”:
Sky-Fall
Dog-Toy
Blind-Bird
Time-Loss
By condensing information, we can process this much more rapidly.
The second limitation is the theory of bounded rationality (Simon, 1982). Bounded rationality refers to the limited ability to focus attention to one item at a time. As visual short-term/working memory is limted to a range of 3-5 items, and an individual’s ability to navigate the problem space is slow. These issues occur regardless of expertise
These limitations have led to development of the theory of cognitive load (Sweller, 1988) and has been influential in educational and cognitive psychology (Stepp et al, 2019; Paas, Gog, and Sweller, 2010; Gog, Paas, and Sweller, 2010). This has brought focus to the importance of instructional design when presenting a problem, creating a structure for which individuals can build knowledge and generate a solution.
There are three forms for which cognitive load exists; intrinsic, extraneous and germane. Intrinsic cognitive load refers to the difficulty that is inherent to the problem at hand (e.g. multiplying two values is easier than solving a differential equation). Extraneous cognitive load is driven the complexity of the material presented, with more complex presentation inducing a split attention effect (Tarmizi and Sweller, 1988; e.g. identifying a horse from a verbal description requires greater attention to process, than identifying a horse from a picture). Germane cognitive load occurs during the processing of information and constructing schemas from an individual can automate their responses. All three forms of cognitive load share a single resource (attention), interfering with one another. Although intrinsic cognitive load may be immutable, by applying the principles of instructional design, it is possible to create instructional materials that reduce extraneous cognitive load and encourage germane cognitive load (Sweller et al, 1998) – this allows for improved information processing and problem solving. When designing these materials one should be aware of factors such the split attention effect (division of attention between different modalities e.g. visual and auditory), and the expertise reversal effect (the influence of an individual’s prior knowledge on other effects and cognitive load e.g an expert may be better at mentalising the problem, but are also prone to higher cognitive loads as they process redundant information already integrated into their schemas; Kalyuga, 2007).
This is based upon working-memory and Simon's theory of bounded rationality (Simon, 1982). Bounded rationality refers to limitations within working-memory systems, and of the limited ability to focus attention to one item at a time. Visual short-term/working memory is limted to only four items, and an individual’s ability to navigate the problem space is slow. These issues occur regardless of expertise.
Other researchers believe that it is better to give freedom to the experts, by allowing them to inform boundaries instead of crisp probabilities, such as the research conducted with creedal network procedure (Antonucci et al., 2013).
Research with c-boxes ?? (ask Scott)
Opinion aggregation
It is common sense that, if decided to use expert elicitation, it is preferable to have opinions from different experts than just one. (add reference!) Thus, there must be a way to aggregate their opinion.
There are different ways of aggregating the opinion of multiple experts and adopting methods to reduce expert elicitation variability (Mkrtchyan et al., 2016; Shirazi, 2009).
Mathematical aggregation
Usually mathematical aggregation is chosen when forms are submitted to the experts (instead of having them on a meeting).
If mathematical aggregation is going to be used (especially if using Bayesian framework), Shirazi (2009) has evidenced that more than three experts do not give meaningful increment on the quality of the measure.
“To study the impact of number of experts on the accuracy of aggregated estimate collected expert judgments were combined in a Bayesian framework using likelihood distributions developed in the first part of the research study. Total number of estimates with reduced errors was depicted against corresponding expert panel size. The objective achieved was the determination of the correlation between the number of experts and the accuracy of the combined estimate to recommend an expert panel size. The result of the study showed weak to moderate correlation between the expert panel size and the accuracy of aggregate. It was noted that eliciting two experts (instead of one) could lead to reduction in relative error of estimates.” (Shirazi, 2009)
Consensus aggregation:
(refer to SHELF procedure, presented in the workshop)
Why experts (and not data analysts)?
Is expert elicitation increasing?
Does it compare with data analysis?
Credibility of data elicited from experts
The lack of credibility on data obtained from experts is explained by different reasons:
Experts can be oriented by different sources of bias (Mosleh at al., 1988);
Experts can be systematically overconfident about the accuracy of their judgments (Lin and Bier, 2008);
In some fields, such as human error probability, it is claimed that experts should be highly experienced not only in the field, but also in the technique and taxonomy used. The lack of experience demands more time for the elicitation process and lower levels of correlation to real-world data (Kirwan, B., 1997).
Works published on validation of experts elicitation
Kirwan (1997) have compiled the validation efforts in using expert judgement for human reliability techniques. In his opinion, the evidence from the rigorous experimental validation studies (using high quality data and experienced experts and practitioners) should be given more weight than the evidence from the relatively informal studies of the techniques (using a few subjects and lower quality data).
Kirwan (1997) define expert validation as: ‘Internal validity’ when experts agree within themselves and ‘External validity’ when experts opinion agree with real data.
Taylor-Adams, S. and Kirwan, B., 1995. Human reliability data requirements. International Journal of Quality & Reliability Management, 12(1), pp.24-46.
Mosleh, A., Bier, V.M. and Apostolakis, G., 1988. A critique of current practice for the use of expert opinions in probabilistic risk assessment. Reliability Engineering & System Safety, 20(1), pp.63-85.
Lin, S.W. and Bier, V.M., 2008. A study of expert overconfidence. Reliability Engineering & System Safety, 93(5), pp.711-721.
Kirwan, B., 1997. Validation of human reliability assessment techniques: part 1—validation issues. Safety Science, 27(1), pp.25-41.
Mkrtchyan, L., Podofillini, L. and Dang, V.N., 2016. Methods for building conditional probability Tables of bayesian belief networks from limited judgment: an evaluation for human reliability application. Reliability Engineering & System Safety, 151, pp.93-112.
Shirazi, C.H., 2009. Data-informed calibration and aggregation of expert judgment in a Bayesian framework (Doctoral dissertation).
Laumann, K., Blackman, H. and Rasmussen, M., 2018. Challenges with data for human reliability analysis. In Safety and Reliability–Safe Societies in a Changing World (pp. 315-321). CRC Press. 14
Droguett, E.L. and Mosleh, A., 2008. Bayesian methodology for model uncertainty using model performance data. Risk Analysis: An International Journal, 28(5), pp.1457-1476.
Bailey, R.T., 1997. Estimation from zero‐failure data. Risk Analysis, 17(3), pp.375-380.
Gustafson, D.H., Sainfort, F., Eichler, M., Adams, L., Bisognano, M. and Steudel, H., 2003. Developing and testing a model to predict outcomes of organizational change. Health services research, 38(2), pp.751-776.
Mosleh’s presentation, key-note speaker of ESREL 2018: https://www.ntnu.edu/documents/1272224149/0/keynote-lecture-ali-mosleh.pdf/c324fe37-ab05-4f8c-8a77-fd23a1c7d3af
Shirazi, C.H., 2009. Data-informed calibration and aggregation of expert judgment in a Bayesian framework (Doctoral dissertation). Swain, A.D. and Guttmann, H.E., 1983. NUREG/CR-1278. Handbook of Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications, US Nuclear Regulatory Commission.
Budnitz, R.J., Apostolakis, G. and Boore, D.M., 1997.Recommendations for probabilistic seismic hazard analysis: guidance on uncertainty and use of experts (No. NUREG/CR-6372-Vol. 1; UCRL-ID-122160). Nuclear Regulatory Commission, Washington, DC (United States). Div. of Engineering Technology; Lawrence Livermore National Lab., CA (United States); Electric Power Research Inst., Palo Alto, CA (United States); USDOE, Washington, DC (United States).
Antonucci, A., Huber, D., Zaffalon, M., Luginbühl, P., Chapman, I. and Ladouceur, R., 2013, July. CREDO: a military decision-support system based on credal networks. In Information Fusion (FUSION), 2013 16th International Conference on (pp. 1942-1949). IEEE.
Tolo, S., Patelli, E. and Beer, M., 2018. An open toolbox for the reduction, inference computation and sensitivity analysis of Credal Networks. Advances in Engineering Software, 115, pp.126-148.
Pedro notes:
Notes on Expertise:
Many of the following papers use medical experts and chess grandmasters as examples
The following terms are common in modern theories of expertise, and based on data acquired from cognitive psychology:
Chunk: A unit of information within the short-term memory, composed of perceptual and semantic information – often encoded into long-term memory
Schema: A cognitive framework used to describe a broader concept, built upon and updated by chunks I.E. A schema for dogs would be based ones experience with dogs, such as the physical traits observed and their interactions, this allows humans to distinguish dogs from cats
Domain Knowledge (Alexander 1997): Breadth of knowledge within a field, I.E. An oncologist knowing a variety of cancer types
Topic Knowledge: Specialist knowledge of a topic within the field, knowing things in greater depth I.E. An oncologist knowing the mechanisms that induce and drive the growth of lung carcinoma
Stages of Expertise
A controversial question within the field of expertise research is whether development can be described as discrete stages. Raufaste et al (1998) describes six stages of expertise, ranging from “novice” to “super-expert”, each stage describes an incremental increase in domain knowledge and ability. These stages however are poorly defined, alternatively it has been suggested that development may appear stage-like, but the boundaries for stages may be an artefact of the methodology (Patel and Groen, 1991). The development of expertise is not purely based on the acquisition of knowledge, but also involves the organisation of knowledge (allowing it to be more readily available).
Alexander (2003) presents a Model of Domain Learning (MDL) based upon broader stages and designed the model upon data acquired from student learning within academic domains. The first stage is “Acclimation”, a stage in which the student often possesses fragmentary knowledge of a domain, and little knowledge of a topic. During this stage students are unable to completely distinguish between accurate and inaccurate information, or to discern the relevance of a topic. In order to navigate this new landscape, students often implement surface level strategies (I.E. Rereading, or paraphrasing in order to interpret text).
The Second stage is “Competence”, marked by qualitative and quantitative changes in the student's knowledge base. The increase in domain knowledge is also followed by a restructuring of said knowledge, following sets of rules or principles, whilst being more cohesive and complex. The development of more elaborate schemas allows for a mix of surface level strategies and deep-processing strategies (I.E. Delving into text, critically evaluating it and the formation of mental imagery).
The third and final stage is “Proficiency/Expertise” is marked by the synergy of multiple schemas. This allows the expert to perform two new functions; problems-finding and proceduralisation. Problem-finding allows the expert to create new bodies of knowledge or expand upon existing bodies of knowledge. This is achieved via proceduralisation, the consolidation of knowledge to generate a problem-solving strategy. The acquisition of new knowledge is primarily based upon deep-processing strategies.
Decision-Making in Expertise
The approach towards decision-making between experts and non-experts vary, with non-experts often reasoning backwards from a hypothesis, whilst experts build a hypothesis on existing data (Arocha et al, 1993). By prioritising critical cues and key features, an expert can develop a set of hypotheses from which they can select (Coughlin and Patel, 1987; Kushniruk et al, 1998). This selection process is often unconscious, an expert's intuition, and once selected a hypothesis can then be consciously reflected upon and regulated (Boreham, 1994).
Intermediate effects have also been investigated, however data conflicts as to how performance varies between novice and intermediate decision-making capabilities (Lesgold et al, 1988; Van de Wiel et al, 1998).
It should be noted that conditions in a laboratory are different from those in a natural setting. Decision-Makers are often subject to multiple pressures and time-constraints; such hinderances to decision-making include externality of events (some things are outside the expert’s control), large and complex systems (often requiring experts from different fields), the impact of uncertainty, and the known and unknown risks.
These pressures can invoke decision-making biases in experts (Gaba, 1992) such as cognitive tunnel vision (forcing data to fit pre-existing views, rather than updating your own), fixation on surface issues, and believing a decision will create a net positive despite evidence to the contrary. It should be noted that an expert's body of knowledge is highly personal and subject to various biases (Grant and Marsden, 1988).
An expert's ability is reflected in their ability to analyse problem and applying their knowledge to potential solutions. According to a study by Lesgold et al (1988), experts can develop mental models of the problems quicker than novices, whilst invoking schemas to resolve and critically inspect the problem at hand. The schema of experts also tends to be more flexible, reinterpreting the problem to better fit what they know.
Cognitive Models of Expertise
Dreyfus and Dreyfus (1972, 2005) developed a five-stage theory of expertise, arguing the differences between AI and human decision-making. Whilst machines rely on discrete symbols and rules, Dreyfus and Dreyfus argue that humans take a holistic approach to problem solving. Where expert systems (AI) may use discrete, context-free, disembodied rules, human experts can act intuitively based on their embodied knowledge. This theory holds that AI may not perform better than human experts. This has been disproven though with numerous stories of expert systems competing with human experts (Silver et al, 2017; Kermany et al, 2018).
The progression from novice to expert follows previous arguments regarding skill curves, in that as a novice progresses to an expert decision-making becomes more intuitive and less effortful. The ability to identify salient features of problem becomes streamlined and almost natural to the expert. As theory of pure intuition, knowledge is often non-declaritive, with deliberation being a sign of a lack of expertise.
In response to Dreyfus and Dreyfus, Montero and Evans (2011) developed their own theory of the rational expert. Rather than an expert automating their performance through intuition, the expert consciously follows rules and can represent their rationale for the choices made. Two principles are embedded in the expert, explicit knowledge of the rules, and the usage of heuristic rules. Montero and Evans utilise two classes of heuristics which to an expert can be stated explicitly and are consciously known. Basic heuristic rules, and advanced heuristic rules. How these heuristics differ from intuition is that the expert is still conscious of the decisions made and can explain their choices. What differentiates the expert from the novice, is the ability to readily access their knowledge and greater access to their thoughts.
In contrast to Dreyfus and Dreyfus, expertise is completely conscious. Rules and knowledge are declarative whilst introspection and deliberation are crucial to the decision-making process.
The Template theory of Expertise (Gobet and Chassy, 2009) is a generalisable model of expertise that can also be used to explain the development of other cognitive functions such as language acquisition and concept formation (Gobet et al, 2001; Gobet and Simon, 2000). This is based upon working-memory and Simon's theory of bounded rationality (Simon, 1982). Bounded rationality refers to limitations within working-memory systems, and of the limited ability to focus attention to one item at a time. Visual short-term/working memory is limted to only four items, and an individual’s ability to navigate the problem space is slow. These issues occur regardless of expertise.
To circumvent these limitations experts, utilise heuristics and efficient problem-solving methods. Furthermore, by possessing a template/schema regarding this body of knowledge the expert can simplify a problem. Having more complex schemas can allow experts to adapt previous solutions to novel problems.
Within this theory, information is stored as chunks, which can be encoded into long-term memory. As an individual gain more knowledge on a subject, chunks are updated, and can form complex connections or patterns. These patterns allow for the formation of a schema/template, built upon a core (chunk). As a complex structure, the template possesses “slots”, a substructure within the template that allows for the encoding of variable/contextual information. Templates allow for a further concept known as “productions”. Described by Newell and Simon (1972), a production is an action in response to a known problem. As an expert develops, they begin to recognise patterns and can produce automated responses. These productions are the result of emotional networks alerting the expert to existence of known problem, and automatically recall information from the long-term memory (template).
Naturalistic theories of expertise also exist but they less defined due to issues replicating natural problems experienced by experts. In reality, problems are often ill-defined, subject to pressures and are often dynamic. Lipshitz (1993) has summarised some of the thematic properties of these theories, such as diversity of form (that expert decision-making is unique to topic knowledge, with experts using processes unique to their field), and mental imagery (The expert builds a mental model of the situation, and does not operate mathematically when calculating costs and benefits of a solution).
Biology of Expertise
Expertise from a neurological standpoint often focuses on two types of expertise – motor skills and domain specificity (facial recognition being the primary example). Different regions of the brain are crucial different forms of problems solving. The following will look at two examples, sports performance, and facial recognition.
Sports performance:
Professional athletes rely heavily on mental imagery of their movements to enhance performance (Oishi and Maeshima, 2004). These mental images follow Fitt's Law, the trade-off between speed and accuracy just as actual movement does (Maruff et al, 1999). Regions of the brain often used in the motor planning and execution (Pre-motor area and supplementary motor area) are activated to different degrees between novices and experts. Expert athletes often produce smaller activation patterns in comparison to novices (Ross et al, 2003). This is the inverse to chess experts who utilise larger patterns of activation, often due to needing greater mental imagery and planning for the position of pieces on a board (Altherton et al, 2003).
Furthermore, neural networks are often reinforced with use, creating stronger connections between brain regions. Networks involved in visually-triggered goal orientated movements such as the prefrontal-parietal-occipital network are crucial to athletics (Buccino et al, 2001). Other regions such as the posterior cingulate gyrus are used for the encoding of visuomotor tasks, monitoring extrapersonal space, topokinetic and topographical memory, as well as the dynamic relocation of spatial attention (Mesulam et al, 2001).
Facial Recognition:
Facial recognition provides a fascinating area of research for expertise. Many of the areas involved in facial recognition, are also recruited for non-face objects by experts (Gauthier et al, 2000), I.E. An ornithologist will utilise the same region for face recognition, when observing and identifying a species of bird. A region of the inferior temporal cortex known the Fusiform Face Area (FFA) was for a long time considered an area dedicated to facial processing. Whilst early neuroimaging studies have found that the right FFA possess a greater number of cells representing facial features (Kanwisher et al, 1997), other studies using single-cell recordings have found that certain neurons respond more strongly to objects than faces (Ishai et al, 1999).
This has resulted in the idea (Championed by Gauthier) that the FFA and connected regions form part of a visual expertise network (E.G. Hippocampus, entorinhal cortex and the amygdala). Experts display increased activity in the FFA in response to relevant object classes (Gauthier et al, 2000), and when an individual is trained to recognise novel stimuli such as “greebles” (Gauthier et al, 1999). The FFA is crucial for distinguishing between stimuli that similar in appearance and has even been found to activate when identifying chess displays (Bilalic et al, 2011). The FFA may be a region dedicated to the visual components of domain-specificity.
References
Alexander, P. A. (1997). Mapping the multidimensional nature of do-main learning: The interplay of cognitive, motivational, and strategic forces. In M. L. Maehr & P. R. Pintrich (Eds.), Advances in motivation and achievement (Vol. 10, pp. 213–250). Greenwich, CT:JAI Press.
Alexander, P. A. (2003). The development of expertise: The journey from acclimation to proficiency. Educational researcher, 32(8), 10-14.
Altherton M, Zhuang J, Bart WM, Hu X, He S (2003) A functional MRI study of high-level cognition: the game of chess.Cogn Brain Res16:26–31.
Arocha, J.F., Patel, V.L., and Patel, Y.C. (1993). Hypothesis generation and the coordination of theory and evidence in novice diagnostic reasoning. Medical Decision Making, 13(3):198-211.
Bilalić, M., Langner, R., Ulrich, R., & Grodd, W. (2011). Many faces of expertise: fusiform face area in chess experts and novices. Journal of Neuroscience, 31(28), 10206-10214.
Boreham, N.C. (1994). The dangerous practice of thinking. Medical Education, 28(3):172-179.
Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Seitz RJ,Zilles K, Rizzolatti G, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study.EurJ Neurosci13:400–404.
Coughlin, L.D. and Patel, V.L. (1987). Processing of critical information by physicians and medical-students. Journal of Medical Education, 62(10):818-828.
Dreyfus, H. L. (1972)What Computers Can’t Do: A Critique of Artificial Reason(New York, NY,Harper and Row).
Dreyfus, H. L. and Dreyfus, S. E. (2005) Expertise in Real World Contexts. Organization Studies,26.5, pp. 779–792.
Gaba, D.M. (1992). Dynamic decision-making in anesthesiology: cognitive models and training approaches. In Evans, D.A. and Patel, V.L., editors, Advanced Models of Cognition for Medical Training and Practice, pages 123-148. Springer-Verlag, Berlin Heidelberg. Proceedings of the NATO Advanced Research Workshop on Advanced Models of Cognition for Medical Training and Practice held at Il Ciocco, Barga, Italy June 19-22 1991.
Gauthier, I., Skudlarski, P., Gore, J. C., & Anderson, A. W. (2000). Expertise for cars and birds recruits brain areas involved in face recognition. Nature neuroscience, 3(2), 191.
Gauthier, I., Tarr, M. J., Anderson, A. W., Skudlarski, P., & Gore, J. C. (1999). Activation of the middle fusiform 'face area' increases with expertise in recognizing novel objects. Nature neuroscience, 2(6), 568.
Gobet, F. and Chassy, P. (2009) Expertise and Intuition: A Tale of Three Theories. Minds and Machines, 19, pp. 151–180
Gobet, F., Lane, P. C. R., Croker, S., Cheng, P. C. H., Jones, G., Oliver, I. and Pine, J. M. (2001)Chunking Mechanisms in Human Learning. Trends in Cognitive Sciences, 5, pp. 236–243.
Gobet, F. and Simon, H. A. (2000) Five Seconds or Sixty? Presentation Time in Expert Memory. Cognitive Science, 24, pp. 651–682.
Grant, J. and Marsden, P. (1988). Primary knowledge, medical-education and consultant expertise. Medical Education, 22(3):173-179.
Ishai, A., Ungerleider, L. G., Martin, A., Schouten, J. L., & Haxby, J. V. (1999). Distributed representation of objects in the human ventral visual pathway. Proceedings of the National Academy of Sciences, 96(16), 9379-9384.
Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of neuroscience, 17(11), 4302-4311.
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C., Liang, H., Baxter, S. L., ... & Dong, J. (2018). Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell, 172(5), 1122-1131.
Kushniruk, A.W., Patel, V.L., and Marley, A. A.J. (1998). Small worlds and medical expertise: implications for medical cognition and knowledge engineering. International Journal of Medical Informatics, 49(3):255-271.
Lesgold, A., Rubinson, H., Feltovitch, P., Glaser, R., Klopfer, D., and Wang, Y. (1988). Expertise in a complex skill: Diagnosing x-ray pictures. In Chi, M. T.H., Glaser, R., and Farr, M.J., editors, The Nature of Expertise, pages 311-342. Lawrence Erlbaum Associates, Publishers, Hillsdale, New Jersey.
Lipshitz, R. (1993). Converging themes in the study of decision making in realistic settings.
In Klein, G.A., Orasanu, J., Calderwood, R., and Zsambok, C.E., editors, Decision Making in Action : Models and Methods, pages 103-137. Ablex, Norwood, NJ.
Maruff P, Wilson PH, DeFazio J, Hedt A, Ceretelli B, Currie J (1999) Asymmetries between dominant and non-dominant hands in real and imagined motor ask performance. Neuropsychologia37:379–384.
Mesulam MM, Nobre AC, Kim YH, Parrish TB, Gitelman DR (2001) Heterogeneity of cingulate contributions to spatial attention. Neuroimage13:1065–1072.
Montero, B. and Evans, C. D. A. (2011) Intuitions without Concepts Lose the Game: Mindednessin the Art of Chess. Phenomenology and the Cognitive Sciences, 10.2, pp. 175–194.
Newell, A. and Simon, H. A. (1972) Human Problem Solving(Englewood Cliffs, NJ, Prentice-Hall).
Oishi K and Maeshima (2004) Automatic nervous system activities during motor imagery in elite athletes. J Clin Neurophys21:170–179.
Patel, V.L. and Groen, G.J. (1991). Developmental accounts of the transition from medical-student to doctor - some problems and suggestions. Medical Education, 25(6):527-535.
Ross JS, Thach J, Ruggieri PM, Lieber M, Lapresto E (2003) The mind’s eye: functional MR imaging evaluation of golf motor imagery. AJNR Am JNeuroradiol24:1036–1044.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., ... & Lillicrap, T. (2017). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815.
Simon, H. A. (1982) Models of Bounded Rationality: Behavioral Economics and Business Organi-zation(Cambridge, MA, The MIT Press)
van de Wiel, M.W., Schmidt, H.G., and Boshuizen, H.P. (1998). A failure to reproduce the intermediate effect in clinical case recall. Academic Medicine, 73:894-900.