Illustrative: a joint probability density whose dependence shifts over time

Risk Academy, led by the Nexus Consortium · Course catalogue · Risk Science and Methods

Decision, Behaviour and Communication

Expert judgement and elicitation

Expert judgement and elicitation, taught twice over: as the working methods of the field and as its frontier since 2024, with labs on the Academy's tools, open data and the Nexus Standards.

Core course Frontier course

The discipline

01Core course · L2 Intermediate

Expert judgement and elicitation: foundations and practice

This area covers how uncertain quantities are estimated from experts when data are thin: structured elicitation protocols, the calibration and weighting of experts, and the aggregation of their judgements into a distribution.

Level
L2 Intermediate
Study time
about 24 hours (planning estimate)
Status
Planned
Prerequisites
An L1 course in the area, or an entry review

Why now

Structured elicitation now feeds high-stakes assessments, from IPCC calibrated language to EFSA risk assessment and volcano eruption forecasting, and the IDEA protocol and Cooke's classical model give tested ways to score and pool experts. Methods in Ecology and Evolution, A practical guide to structured expert elicitation using the IDEA protocol (Hemming et al.), 2018

Learning outcomes

  1. Define structured expert judgement and the other key terms of the discipline and use them correctly in a written case
  2. Explain structured expert judgement protocols and the Classical Model and performance-based weighting with a worked example and its source
  3. Apply the discipline's method to a documented case and explain each result with its source
  4. Analyse a named case against IDEA and Delphi protocols, stating evidence, date and limits

Modules

  1. Structured expert judgement protocols
    • Structured expert judgement protocols
    • The Classical Model and performance-based weighting
    • IDEA and Delphi protocols
  2. SHELF and behavioural aggregation
    • SHELF and behavioural aggregation
    • Seed questions and expert calibration
    • Heuristics and biases in judgement
  3. Eliciting probability distributions
    • Eliciting probability distributions
    • Documenting and reviewing an elicitation
    • Expert judgement in catastrophe and climate assessment
  4. Lab: the discipline's method
    • An expert states quantiles for seed questions whose answers are known; how often the truth falls inside the stated range gives a calibration score, how narrow the ranges are gives an information score, and their product is the weight the expert's judgement carries in the combined distribution for the question nobody can yet answer.
    • Work the method through a documented case and explain each result
    • Read the method's sources and stated values

Labs with Academy tools

Assessment

  • 40%Lab recordWork the discipline's method through a documented case and explain each result with its source and date
  • 40%Case memoApply the discipline's methods to a named place or organisation, stating evidence and limits
  • 20%Key terms checkShort answers on the key terms, marked against the definitions

Delivery

  • Self-paced online, with the labs in the browser
  • Cohort study over six to eight weeks, with a weekly lab review
  • Co-delivered with a partner institution, inside its own courses

Partner on this course

  • Co-teach a module and bring a current case from your work
  • Host a cohort for your learners, with your own schedule and review
  • Contribute a lab: a dataset your institution publishes openly, or a model
  • Review the course content and its references before it opens
  • Translate the course and adapt its cases to your region

How institutions and experts take part

In the Academy LMS

Course on the learning platform
Intermediate, 4 topics
Cohort group
A community group for each cohort of learners
Achievement
Recorded on completion of the assessed work

References

  1. Cooke, Experts in Uncertainty: Opinion and Subjective Probability in Science, 1991
  2. Colson and Cooke, Cross validation for the classical model of structured expert judgment, Reliability Engineering and System Safety, 2017
  3. Aspinall, A route to more tractable expert advice, Nature, 2010
02Live map

Hazards and exposure in Italy

Flood water, active fires and surface heat beside population and night-time lights, over Italy.

The interactive map needs JavaScript; the layer list and sources are below.

Layers 3

Hazards and exposure

Latest available

NASA GIBS / EOSDIS · ASTER GDEM (METI, NASA) · Blue Marble (NASA) · © OpenStreetMap contributors (ODbL) · Natural Earth

Loading the latest NASA imagery.

Drag to move. Ctrl or ⌘ and scroll, pinch or the buttons to zoom; hover for the values under the pointer; reset returns to the region.

Hazards and exposure, framed on Italy.Satellite and model products as NASA publishes them, each with its own date. They are indicators, not assessments of any place.

Data: NASA GIBS · ASTER GDEM · Blue Marble · © OpenStreetMap contributors · Natural Earth · NASA EONET · U.S. Geological SurveySources and licences

  • We acknowledge the use of imagery provided by services from NASA's Global Imagery Browse Services (GIBS), part of NASA's Earth Science Data and Information System (ESDIS). Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • We acknowledge the use of imagery provided by services from NASA's Global Imagery Browse Services (GIBS), part of NASA's Earth Science Data and Information System (ESDIS). Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • VIIRS near-real-time products (flood, land surface temperature, snow cover, corrected reflectance), NASA LANCE; imagery via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • NASA Fire Information for Resource Management System (FIRMS), VIIRS 375 m active fire product, NASA LANCE; vector tiles via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • Center for International Earth Science Information Network (CIESIN), Columbia University. 2018. Gridded Population of the World, Version 4 (GPWv4): Population Density, Revision 11. NASA Socioeconomic Data and Applications Center (SEDAC). Licensed under CC BY 4.0. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). doi:10.7927/H49C6VHW. CC BY 4.0
  • VIIRS Black Marble night-time lights, NASA Goddard Space Flight Center; distributed by the NASA Level-1 and Atmosphere Archive and Distribution System (LAADS DAAC); imagery via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • MODIS and VIIRS atmosphere products, NASA LANCE and LAADS DAAC; imagery via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • ASTER GDEM is a product of METI and NASA (ASTER Global Digital Elevation Model V003, NASA LP DAAC); imagery via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). Cite as: NASA/METI/AIST/Japan Spacesystems and U.S./Japan ASTER Science Team (2019). ASTER Global Digital Elevation Model V003. doi:10.5067/ASTER/ASTGTM.003. NASA open data
  • Made with Natural Earth. Free vector and raster map data @ naturalearthdata.com. Public domain
  • © OpenStreetMap contributors (ODbL), via NASA GIBS (coastlines, boundaries, roads and place names). Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). ODbL
  • Blue Marble, NASA Earth Observatory; imagery via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • Context on the globe and flat map: coastlines, boundaries, roads and place names from NASA GIBS reference layers (OpenStreetMap data, ODbL, © OpenStreetMap contributors); Blue Marble imagery as an optional base; 3D terrain and the elevation read-out from the ASTER GDEM elevation colour index (5 m to 200 m classes, exaggerated for legibility). Boundaries are drawn as published and imply no position on any territory.
  • Globe: MapLibre GL JS 6.9.1, BSD-3-Clause, © MapLibre contributors.
  • Live events: NASA Earth Observatory Natural Event Tracker (EONET); events are indicative and keep their original source links.
  • Live: NASA EONET Events are indicative, not authority warnings.
  • Live: U.S. Geological Survey Events are indicative, not authority warnings.
  • NASA GIBS: We acknowledge the use of imagery provided by services from NASA's Global Imagery Browse Services (GIBS), part of NASA's Earth Science Data and Information System (ESDIS).
  • River discharge forecast, 30 days with range (GloFAS via Open-Meteo): Contains modified Copernicus Emergency Management Service information 2026; not a flood warning.
  • Weather data by Open-Meteo.com (https://open-meteo.com/), licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).
  • Optional layers (10 sources in the layer list: live, forecast, observed and context) print their provider credit on the map and here when switched on; forecasts are model output and derived indices are estimates.

Layers and dates

  • Flood water, 3-day composite: VIIRS Global Flood Product, NASA LANCE · 14 Oct 2026
  • Active fires and thermal anomalies: VIIRS NOAA-20 375 m (FIRMS), NASA LANCE FIRMS · 13 Oct 2026
  • Population density, 2020: Population: CIESIN, Columbia University, Gridded Population of the World v4.11 (2018), NASA SEDAC, CC BY 4.0 · static
  • Land surface temperature, day, 8-day composite: VIIRS NOAA-20 Level 3, 8-day, NASA LANCE · 14 Sep 2026
  • Night-time lights, daily: VIIRS Black Marble (VNP46A2), NASA LAADS DAAC · 12 Oct 2026
  • Aerosol optical depth, daily: VIIRS NOAA-20 Deep Blue AOD, NASA LANCE · 13 Oct 2026

03How learning runs

How learning runs, as a process you can run

Select a scenario and press Play: course, pathway, practice, faculty, cells and record.

04What is changing

What is changing, with standards and evidence

Expert judgement and elicitation: frontier methods

The methods, data, tools and studies that changed expert judgement and elicitation between 2024 and 2026, taught by reproducing them on open data.

Level
L3 Advanced
Study time
about 28 hours (planning estimate)
Status
Planned
Prerequisites
Expert judgement and elicitation: foundations and practice; Working knowledge of a scripting language (Python or R) for the labs

Why now

Structured elicitation now feeds high-stakes assessments, from IPCC calibrated language to EFSA risk assessment and volcano eruption forecasting, and the IDEA protocol and Cooke's classical model give tested ways to score and pool experts. Methods in Ecology and Evolution, A practical guide to structured expert elicitation using the IDEA protocol (Hemming et al.), 2018

Learning outcomes

  1. Explain what changed in expert judgement and elicitation since 2024 and cite the studies, data or rules behind each change
  2. Reproduce biases and structure on open data and state the uncertainty of the result
  3. Evaluate a 2024 to 2026 study: its question, data, method, limits and what it changes for practice
  4. Design an analysis that combines a frontier method with the faculty's instruments, and justify each choice

Modules

  1. Biases and structure
    • Overconfidence, anchoring and groupthink
    • Calibrated uncertainty language (IPCC)
    • Question design
  2. Protocols
    • The IDEA protocol
    • Cooke's classical model and seed questions
    • EFSA guidance on expert knowledge elicitation
  3. Pooling and use
    • Linear and performance-weighted pools
    • Forecasting tournaments
    • Documenting judgements for audit
  4. Reproduction lab
    • Extend the discipline's method with one frontier method and compare its results
    • State the uncertainty and the limits of the result
    • Peer review against a shared checklist

Labs with Academy tools

Assessment

  • 50%Method reproductionReproduce one frontier method on open data and state the uncertainty of the result
  • 30%Study reviewA critical review of a 2024 to 2026 study: question, data, method, limits, what it changes
  • 20%Peer reviewReview two other learners' reproductions against a shared checklist

Delivery

  • Self-paced online, with the labs in the browser
  • Cohort study over six to eight weeks, with a weekly lab review
  • Co-delivered with a partner institution, inside its own courses

Partner on this course

  • Co-teach a module and bring a current case from your work
  • Host a cohort for your learners, with your own schedule and review
  • Contribute a lab: a dataset your institution publishes openly, or a model
  • Review the course content and its references before it opens
  • Translate the course and adapt its cases to your region

How institutions and experts take part

In the Academy LMS

Course on the learning platform
Expert, 4 topics
Cohort group
A community group for each cohort of learners
Achievement
Recorded on completion of the assessed work

References

  1. Methods in Ecology and Evolution, A practical guide to structured expert elicitation using the IDEA protocol (Hemming et al.), 2018
  2. EFSA Journal, Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment, 2014
  3. IPCC, Guidance note on the consistent treatment of uncertainties, 2010
  4. Cooke, Experts in Uncertainty: Opinion and Subjective Probability in Science, 1991
  5. Colson and Cooke, Cross validation for the classical model of structured expert judgment, Reliability Engineering and System Safety, 2017
  6. Aspinall, A route to more tractable expert advice, Nature, 2010
05Take part

Take part in the work

Start from who you are.

06Questions

Questions, answers and key terms

Is the course free?
Yes. Every course and learning material of Risk Academy, led by the Nexus Consortium, is free to study, self-paced or with a cohort. Ordinary participation has no fee; paid work, internships and fellowships are separate arrangements.
What does completing a course give me?
A completion record: the course, its version, the date and what was assessed, private by default and shared by you. It stacks into discipline, department and programme achievements. A record states what was studied and assessed and does not certify competence; it is not a licence, a professional title or an employment status.
Where do the data and the places come from?
Every faculty teaches on a twin of a real place built from open data; each layer carries its provider, licence and date, and every result on the Academy states its source. Maps and figures are indicators for teaching, never an assessment of a place.
How do I take part beyond the course?
Form a competence cell of three to seven people, study a shared pathway, take a supervised assignment in the National Working Groups, or bring a cohort from your institution. For organisations.

Key terms

Term

Structured expert judgement

A documented protocol for eliciting and combining expert estimates so that the result can be reviewed.

Term

Classical Model

Cooke's method that weights experts by their calibration and informativeness on seed questions with known answers.

Term

Seed question

A question whose answer is known to the analyst but not the experts, used to score expert performance.

Term

Delphi method

An iterative, anonymous survey of experts with controlled feedback between rounds.

ReportsContribute

Contribute evidence from this page.

Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

Risk Academy
Related

Around Expert judgement and elicitation: courses

Risk Academy

Study expert judgement and elicitation with us, or teach it

Every course is free and open. Institutions and experts can co-teach, host a cohort, contribute a lab or review a course before it opens.

Browse the course catalogue Partner with the Academy
Risk Academy

Courses · practice · credentials

Not for profit

Our articles, maps and twins are free to read and free to republish. Readers and partners keep it that way.

Support our work

Who funds us

Risk Academy is a not-for-profit institution. To ask who funds a piece of work, use the funding questions form.

Independence

Editors and the expert reviewers named on each article decide what we publish. Funders and partners do not review or approve articles before publication.

Corrections policy

Licence

Articles: CC BY 4.0. Figures, maps, twins and data: CC BY 4.0. Other uses: ask through the permissions form.

Privacy choices