Expert judgement and elicitation: foundations and practice
- Level
- L2 Intermediate
- Study time
- about 24 hours (planning estimate)
- অবস্থা
- Planned
- Prerequisites
- An L1 course in the area, or an entry review
Structured elicitation now feeds high-stakes assessments, from IPCC calibrated language to EFSA risk assessment and volcano eruption forecasting, and the IDEA protocol and Cooke's classical model give tested ways to score and pool experts.
Learning outcomes
- Define structured expert judgement and the other key terms of the discipline and use them correctly in a written case
- Explain structured expert judgement protocols and the Classical Model and performance-based weighting with a worked example and its source
- Apply the discipline's method to a documented case and explain each result with its source
- Analyse a named case against IDEA and Delphi protocols, stating evidence, date and limits
Modules
-
Structured expert judgement protocols
- Structured expert judgement protocols
- The Classical Model and performance-based weighting
- IDEA and Delphi protocols
-
SHELF and behavioural aggregation
- SHELF and behavioural aggregation
- Seed questions and expert calibration
- Heuristics and biases in judgement
-
Eliciting probability distributions
- Eliciting probability distributions
- Documenting and reviewing an elicitation
- Expert judgement in catastrophe and climate assessment
-
Lab: the discipline's method
- An expert states quantiles for seed questions whose answers are known; how often the truth falls inside the stated range gives a calibration score, how narrow the ranges are gives an information score, and their product is the weight the expert's judgement carries in the combined distribution for the question nobody can yet answer.
- Work the method through a documented case and explain each result
- Read the method's sources and stated values
Labs with Academy tools
পদ্ধতি
- From an expert's quantiles to a weight: seed questions, calibration, information and the aggregate that resultsThe discipline's method on the discipline page, with its cited sources, worked through a documented case
উপকরণ
- Distribution fitterA faculty instrument on the faculty page, run on the faculty's digital twin data
- Return periodA faculty instrument on the faculty page, run on the faculty's digital twin data
Twin builder
- Digital twin builderBuild the faculty's place in 3D, switch layers and read the credits and dates of each
Map layer
- Natural events (NASA EONET)An open map layer the Academy's licence register admits, read in the map tools
- Earthquakes M4.5+, past week (USGS)An open map layer the Academy's licence register admits, read in the map tools
- Population density, 2020 (GPW v4.11)An open map layer the Academy's licence register admits, read in the map tools
Foundry room
- Decisions room, NXS-DSSDecisions that rest on elicited distributions with their basis recorded
- Council room, NXS-CMDStructured deliberation of expert panels with a reviewable record
Assessment
- 40%
Lab record
Work the discipline's method through a documented case and explain each result with its source and date
- 40%
Case memo
Apply the discipline's methods to a named place or organisation, stating evidence and limits
- 20%
Key terms check
Short answers on the key terms, marked against the definitions
Delivery
- Self-paced online, with the labs in the browser
- Cohort study over six to eight weeks, with a weekly lab review
- Co-delivered with a partner institution, inside its own courses
Partner on this course
- Co-teach a module and bring a current case from your work
- Host a cohort for your learners, with your own schedule and review
- Contribute a lab: a dataset your institution publishes openly, or a model
- Review the course content and its references before it opens
- Translate the course and adapt its cases to your region
How institutions and experts take part
In the Academy LMS
- Course on the learning platform
- Intermediate, 4 sections
- Cohort group
- A community group for each cohort of learners
- Achievement
- Recorded on completion of the assessed work