RA · Foundations
Expert judgement and elicitation: what to learn, in what order, and why
A learning guide to expert judgement and elicitation: biases, structured protocols, calibrated language, key texts and an order of study.
- As of
- 17 November 2026
- Lektüre
- 11 minutes
- Evidenz
- 10 sources, cited in order
- Stufe
- Introductory to intermediate
- Lernzeit
- 15 minutes
What you will be able to do
- Name three heuristics that affect expert judgement.
- Describe the stages of a structured elicitation.
- Distinguish a confidence statement from a probability statement.
- Write an elicitation record with question, experts, method and result.
- Stufe
- Introductory to intermediate
- Lernzeit
- 15 minutes
- Before you start
- The learning guide to the Faculty of Risk Science and Methods, or equivalent background
- Guidance
- 2 metrics
- used by IPCC authors to communicate certainty: confidence and quantified probability5
01What expert judgement and elicitation covers
What expert judgement and elicitation covers
Expert judgement and elicitation is the Academy discipline of expert judgement and elicitation, one of the disciplines of the Faculty of Risk Science and Methods. It covers how to ask experts for estimates in a way that is honest about uncertainty, and how to record, combine and test what they say.
The field exists because data are often missing. A hazard may be rare, a technology new or a system hard to observe. In those cases a decision still has to be made. M. Granger Morgan, writing in the Proceedings of the National Academy of Sciences, describes the elicitation of scientific and technical judgments from experts as a valuable addition to other forms of evidence in support of public policy decision making 6. The judgments take the form of subjective probability distributions 6.
The discipline sits beside uncertainty and decision making, quantitative risk analysis and statistics and scenario analysis and stress testing.
02Why structure matters
Why structure matters
Tversky and Kahneman described three heuristics that people use when they judge probabilities and predict values: representativeness, availability of instances or scenarios, and adjustment from an anchor 1. Tversky and Kahneman describe these heuristics as highly economical and usually effective, but leading to systematic and predictable errors 1.
Structured protocols respond to this. Hemming and colleagues state that expert judgements can be prone to contextual biases, and that structured elicitation protocols mitigate these biases and improve the accuracy and transparency of the resulting judgements 2. Morgan notes that it may be tempting to view expert elicitation as a low-cost, low-effort alternative to serious research and analysis, but that it is neither 6.
Public bodies have codified the practice. The US Nuclear Regulatory Commission has a Branch Technical Position on the use of expert elicitation, completed in August 1996 3, and practical guidelines for studies at levels 3 and 4 of the Senior Seismic Hazard Analysis Committee process 4. The learner should know that such texts exist and read the one that fits their field.
03From judgement to a stated distribution
From judgement to a stated distribution
A structured elicitation turns a question into a number with a range. The Sheffield Elicitation Framework (SHELF) describes itself as a package of documents, templates and software to carry out elicitation of probability distributions for uncertain quantities from a group of experts. It says elicitation is increasingly important where hard data are sparse, and that it is generally important to elicit from a group rather than a single expert 7. The steps are usually the same in any protocol: define the question precisely, choose and brief the experts, elicit individual judgements, discuss, and record the result with its reasons.
The way results are stated matters as much as how they are collected. The IPCC guidance note for lead authors of the Fifth Assessment Report relies on two metrics for communicating the degree of certainty in key findings. The first is confidence in the validity of a finding, expressed qualitatively and based on the type, amount, quality and consistency of evidence and the degree of agreement. The second is a quantified measure of uncertainty expressed probabilistically 5. A learner should be able to say which of the two a statement uses.
Disagreement between experts is information, not noise to be removed. A structured process lets the reasons for a difference come out, so that a reader can see whether it comes from different evidence, different definitions or different values. Averaging without looking at the reasons can hide the very thing a decision maker most needs to know.
Colson and Cooke describe using the Classical Model to validate experts' judgments 8. Calibration asks how well stated probabilities match what happens, and a learner should read the paper to see how the Classical Model tests it. Figure 2 shows the stages of a structured elicitation.
Fig. 02Stages of a structured elicitation
Figure 2 in words
Four layers stacked from top to bottom. Record the result with its reasons, highlighted. Elicit and discuss judgements. Choose and brief the experts. Define a precise question. Each layer rests on the one below it.
04A suggested learning order in four stages
A suggested learning order in four stages
The Academy describes study at four levels: introductory, intermediate, advanced and integrative. The four stages below follow the same arc for expert judgement and elicitation. They are a suggestion, and a learner can enter at any stage.
Stage one: why experts err. Start with the psychology. Read Tversky and Kahneman on heuristics and biases 1. Then read Morgan on the use and abuse of elicitation 6. List the biases in your own words and note one way a protocol can reduce each.
Stage two: structured protocols. Next, study the protocols. Read the practical guide to the IDEA protocol 2, which Hemming and colleagues write for conservation and natural resource management, and the SHELF pages 7. Note where protocols differ on whether experts discuss, and how many rounds they use.
Stage three: stating and combining. Then study how to state and combine results. Read the IPCC uncertainty guidance 5. Study the Classical Model 8. Practise turning a verbal judgement into a probability with a range.
Stage four: documentation and review. Finally, study documentation. Read how regulators set out expert elicitation in their own programmes 3 4, and how the IPBES guide supports experts taking part in assessments 9. Practise writing an elicitation record: the question, the experts, the method, the results and the reasons. Ask a colleague to read it and say whether they could repeat the exercise from the record alone. If they cannot, the record is not yet complete.
The order is a suggestion. A practitioner may begin at stage three and return to stage one later. What matters is that each new idea has the ideas beneath it in place.
Fig. 01Four stages of study in expert judgement and elicitation
Figure 1 in words
Four steps rise from left to right. Step one, why experts err: heuristics and biases and the use and abuse of elicitation. Step two, structured protocols: the IDEA protocol and SHELF. Step three, stating and combining: IPCC uncertainty guidance and the Classical Model. Step four, highlighted, documentation and review: regulatory elicitation programmes and the IPBES guide.
Fig. 03Two ways to state certainty
Figure 3 in words
Four stages run from left to right. Evidence and agreement: the type, amount, quality and consistency of evidence and the degree of agreement. Confidence: a qualitative statement of the validity of a finding. Analysis or judgment: statistical analysis of observations or models, or expert judgment. Probability: a quantified measure of uncertainty expressed probabilistically.
05Six core ideas every learner should hold
Six core ideas every learner should hold
Six ideas carry most of the weight in this discipline. Each is simple to state and takes practice to use.
Ask a precise question. The IDEA guide includes a step on developing clear questions 2.
Judgements are probabilities. Morgan describes the judgements as subjective probability distributions 6, and SHELF is a package for eliciting probability distributions for uncertain quantities from a group of experts 7.
Bias is expected. Tversky and Kahneman describe heuristics as a general feature of judgement under uncertainty 1, and Hemming and colleagues state that structured protocols mitigate contextual biases 2.
Validate where you can. Colson and Cooke describe using the Classical Model to validate experts' judgments 8.
Say what kind of statement it is. Confidence in a finding and probability of an outcome are different measures 5.
Document the process. Hemming and colleagues advise that all steps taken and results collected during an elicitation be documented to provide a transparent record of the process and results 2.
06Canonical references and data to know
Canonical references and data to know
A learner should know a short list of references and what kind of evidence each holds. Read primary documents first.
For method: the IDEA protocol guide 2, SHELF 7 and the Classical Model paper 8. For the psychology: Tversky and Kahneman 1 and Morgan 6. For official guidance: the US NRC technical position 3 and SSHAC guidelines 4, the IPCC uncertainty guidance 5 and the IPBES guide on assessments 9. These are methods and guidance documents, not datasets. Learners should record the edition, date and licence of each text they use.
07Skills that make the knowledge usable
Skills that make the knowledge usable
Knowledge becomes useful through a small set of skills. The discipline treats these as its practical core.
Framing a question. Make it precise, testable and answerable by the experts chosen. Selecting experts. Say why each is suitable and how conflicts are handled. Briefing. Explain the task and the common biases before eliciting. Eliciting a distribution. Ask for ranges and quantiles, not only a best guess. Aggregating. Choose and justify a rule for combining judgements. Reporting. State the result, its uncertainty and the process in plain terms.
08Open courses and pathways
Open courses and pathways
The SHELF site lists an e-learning course and a training course page 7. On 17 November 2026 this guide could not confirm the terms on which the e-learning course is offered, so it does not describe it as free or open. The training course page describes an in-person course in September 2026 on survival extrapolation in health technology assessment, and its date has passed.
The documents named in this guide are free to read at their issuers, or through their journal records, as confirmed on 17 November 2026. They are a pathway in themselves. A suggested order is the psychology, then one protocol guide, then the official guidance that applies to your field.
No course in this guide is described as leading to a qualification, because none of the pages opened says so.
09How expertise is shown
How expertise is shown
Expertise in this field is shown by work that others can check. The Academy records learning, not credentials.
Finishing a course gives a completion record. It is not a certificate, an accreditation or a qualification, and it does not state that a person is competent in a profession. Achievements on the Academy recognise assessed learning only.
The Academy catalogue lists two courses for this discipline: Expert judgement and elicitation: foundations and practice, at the intermediate level, and Expert judgement and elicitation: frontier methods, at the advanced level. On 17 November 2026 the catalogue page shows the first as planned and the second as planned. Nothing is open for enrolment until it is published in the NXS Learning catalogue, so this guide does not point to an Academy course as a place to enrol.
Outside the Academy, the strongest evidence of expertise is a body of work: an elicitation record with its question, experts, method and results, a calibration exercise on questions with known answers, or a short paper that compares two protocols. Learners are encouraged to keep such a record from their first course.
10Emerging areas to watch
Emerging areas to watch
Several areas are changing what an elicitation practitioner needs to know. Each is a place to read on once the core is in place.
Validation of experts. The Classical Model is used to validate experts' judgments 8. Structured protocols outside the lab. The IDEA protocol was written for conservation and natural resource management, and its guide aims to make the method usable by others 2. Regulatory practice. The US Nuclear Regulatory Commission has published guidance on elicitation for its own programmes 3 4, most recently NUREG-2255 in November 2024, which says it is intended for use by NRC staff when performing an expert elicitation 10. Calibrated language. The IPCC guidance note sets out two metrics for communicating the degree of certainty in key findings, confidence and quantified probability 5.
Related work across the Nexus network gives wider context. See risk science research at the Global Centre for Risk and Innovation, methods and professional models at the Global Risks Alliance, foresight and anticipation at the Global Risks Forum and the proposed Nexus Standards on verification and monitoring. These pages describe programmes and proposals. A link to them implies no endorsement of the sources this guide cites.
What this guide does not claim: it is suggested learning, not a curriculum that anyone must follow, and it is not professional, legal, medical or technical advice. It names programmes, courses and bodies as places to learn and implies no partnership with any of them. It reports figures as their issuers published them and draws no new estimates from them. Risk Academy does not certify competence, and a completion record is not a professional qualification. The Academy records no partners, cohorts or learner outcomes for this faculty, and this guide implies none. The Nexus Standards are proposed and in draft; nothing here treats them as adopted.
Fragen
Fragen und Antworten
Where should a complete beginner start in expert judgement and elicitation?
Start with the Tversky and Kahneman article on heuristics and biases, then stage two of this guide on structured protocols.
Why not just ask one expert?
A single expert reflects one view and one set of biases. SHELF states that eliciting from a group is generally important to synthesise the range of knowledge in the expert community.
Is expert elicitation a substitute for data?
No. It adds to other evidence where data are sparse, and done poorly it can mislead.
What is calibration?
A check on how well the probabilities that experts state match what actually happens over many questions, used to judge and improve their estimates.
Do courses in this discipline lead to a qualification?
No. A completion record shows that you finished a course. It is not a certificate or a professional qualification.
Nutzungsbedingungen
Glossar
- Elicitation
- The structured process of obtaining judgements from experts, often as probability distributions.
- Heuristic
- A mental shortcut used in judgement that can be useful but can cause systematic error.
- Ankereffekt
- Adjusting an estimate from an initial value, usually insufficiently.
- Kalibrierung
- The match between stated probabilities and how often events occur.
- Aggregation
- Combining the judgements of several experts into one result.
- Confidence
- In the IPCC approach, a qualitative measure of the validity of a finding.
Verwandtes
Further reading across the network
- Expertenurteil und Elicitation
- Core course of the discipline
- Frontier course of the discipline
- Fakultät für Risikowissenschaft und Methoden
- Learning guide: risk science and methods
- Learning guide: catastrophe, exposure and vulnerability modelling
- Learning guide: risk indicators and composite indices
- Learning guide: model risk and validation
- The Global Centre for Risk and Innovation (GCRI): risk science
- The Global Risks Alliance (GRA): models and professional methods
- The Global Risks Forum (GRF): future generations and anticipation
- Nexus Standards: NS-11 verification and monitoring (proposed)
Check your understanding
Why elicit from a group of experts?
A group synthesises the range of knowledge and opinion in the expert community, which a single expert cannot.
What does a structured protocol aim to do?
Mitigate contextual biases and improve the accuracy and transparency of the resulting judgements.
What are the two IPCC metrics?
Confidence in the validity of a finding, expressed qualitatively, and a quantified measure of uncertainty expressed probabilistically.
Continue with a course
- Expert judgement and elicitation: foundations and practice expert-judgement-elicitation.core
- Expert judgement and elicitation: frontier methods expert-judgement-elicitation.frontier
Quellen
Quellen
10 sources, numbered in the order they are first cited
- Science (Tversky and Kahneman) (1974-09). Judgment under Uncertainty: Heuristics and Biases. doi.org. Accessed 2026-11-17.
- Methods in Ecology and Evolution (Hemming, Burgman, Hanea and McBride) (2017-09). A practical guide to structured expert elicitation using the IDEA protocol. doi.org. Accessed 2026-11-17.
- U.S. Nuclear Regulatory Commission (1996-08). Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program (NUREG-1563). www.nrc.gov. Accessed 2026-11-17.
- U.S. Nuclear Regulatory Commission (Revision 1). Practical Implementation Guidelines for SSHAC Level 3 and 4 Hazard Studies (NUREG-2117, Revision 1). www.nrc.gov. Accessed 2026-11-17.
- Intergovernmental Panel on Climate Change (2010-07). Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties. www.ipcc.ch. Accessed 2026-11-17.
- Proceedings of the National Academy of Sciences (Morgan) (2014-05). Use (and abuse) of expert elicitation in support of decision making for public policy. doi.org. Accessed 2026-11-17.
- University of Sheffield (2026). Sheffield Elicitation Framework (SHELF). shelf.sites.sheffield.ac.uk. Accessed 2026-11-17.
- Review of Environmental Economics and Policy (Colson and Cooke) (2018-01). Expert Elicitation: Using the Classical Model to Validate Experts’ Judgments. doi.org. Accessed 2026-11-17.
- Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (2026). Guide on the production of assessments. www.ipbes.net. Accessed 2026-11-17.
- US Nuclear Regulatory Commission (2024-11). Guidance for Conducting Expert Elicitation in Risk-Informed Decisionmaking Activities (NUREG-2255). www.nrc.gov. Accessed 2026-11-17.
Eintrag
Editorial record
Facts and course states are given as of 17 November 2026. This guide is reviewed at least every 270 days and whenever a cited report publishes a new edition or a course of the faculty opens.
Published by the Academy desk of Risk Academy. To report an error or request a correction, write through the Risk Academy contact page; corrections are dated and listed with the page. The Academy's statement on AI use explains how AI assistance is used and reviewed. Text, visuals and data are licensed CC BY 4.0, except where a cited source states other terms.
Cite this article
Academy desk, Risk Academy (2026, November 17). Expert judgement and elicitation: a learning guide. Risk Academy. https://risksacademy.com/risk-science/learning-expert-judgement-and-elicitation-guide/
@online{academy2026learningexpertjudgemen,
author = {{Academy desk, Risk Academy}},
title = {{Expert judgement and elicitation: a learning guide}},
publisher = {Risk Academy},
year = {2026},
month = {11},
url = {https://risksacademy.com/risk-science/learning-expert-judgement-and-elicitation-guide/},
language = {en}
} TY - ELEC AU - Academy desk, Risk Academy TI - Expert judgement and elicitation: a learning guide PB - Risk Academy PY - 2026 DA - 2026/11/17 UR - https://risksacademy.com/risk-science/learning-expert-judgement-and-elicitation-guide/ LA - en ER -
Republish
Republish or translate this article free of charge, in print or online, under CC BY 4.0: keep the byline and Risk Academy, link to the original and note any change. Figures, maps and twins made by the institutions carry the same licence; third-party data keeps its own credit line.
Attribution to paste
<p>Expert judgement and elicitation: what to learn, in what order, and why — Academy desk, Risk Academy, Risk Academy. First published 17 November 2026 at <a href="https://risksacademy.com/risk-science/learning-expert-judgement-and-elicitation-guide/">https://risksacademy.com/risk-science/learning-expert-judgement-and-elicitation-guide/</a>. Republished under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).</p>