Risk Academy, led by the Nexus Consortium · Risk Science and Methods · Department of Decision, Behaviour and Communication
Turning expert knowledge into a reviewable distribution
నిపుణుల తీర్పు మరియు అభిప్రాయ సేకరణ
Where data are thin, risk still has to be quantified, and expert judgement is the method of record.
Map expert judgement and elicitation on a real placeSee the courses
Illustrative: a joint probability density whose dependence shifts over time
01పరిధి
What the discipline covers
The subtopics its courses and labs teach.
- అధ్యాపక విభాగం
- ప్రమాద శాస్త్రం మరియు పద్ధతుల ఫ్యాకల్టీ
- కోర్సులు
- In preparation
- స్థాయిలు
- In development
- ప్రయోగశాలలు
- In development
- Structured expert judgement protocols
- The Classical Model and performance-based weighting
- IDEA and Delphi protocols
- SHELF and behavioural aggregation
- Seed questions and expert calibration
- Heuristics and biases in judgement
- Eliciting probability distributions
- Documenting and reviewing an elicitation
- Expert judgement in catastrophe and climate assessment
02మ్యాప్లు
Build a twin of any place
Search any place or open the world; the layers and generators follow expert judgement and elicitation on the planet map, the globe and in 3D.
- Terrain 24 to 2,805 m
- 3 reference earthquakes, M6.2 to M6.6
- Earthquakes M4.5 and above, past week: read live when the 3D twin opens
- USGS ShakeMap intensity contours: read live when the 3D twin opens
Drag to move, right drag or Shift with a drag to turn and tilt, Ctrl with the wheel to zoom. Keyboard: arrows move, Shift with arrows turn and tilt, + and – zoom, 0 resets, V steps through the views, Escape clears.
Flat plan of the same data: relief, the three reference epicentres with magnitude and depth, and the numbers below; the sampler's figure shows one precomputed field with its parameters. The 3D twin opens on devices that can draw it, once its data and code have been checked.
The boundaries, names and designations shown do not imply any judgement on the legal status of any territory, or any endorsement or acceptance of such boundaries. Countries: Natural Earth (public domain).
03The subject
The subject, methods and measures and debates and bodies
The discipline
The field and its scope
For analysts, assessment authors and decision makers who must estimate uncertain quantities from experts when data are thin, and do it in a way that others can review.
Expert judgement and elicitation study how the knowledge of experts is turned into probability distributions for quantities that data alone cannot determine: the failure rate of a new component, the sea level contribution of an ice sheet, the case fatality ratio of a new pathogen, the chance of a volcanic eruption in the next year. The field treats expert judgement as data to be collected under a protocol, with the same concerns for bias, validation and documentation that apply to any measurement. It grew from nuclear safety and volcanology, and now serves climate assessment, catastrophe modelling, public health, ecology and finance.
Practice is organised around protocols. The Classical Model of Cooke weights experts by their performance on seed questions with known answers; the IDEA protocol (investigate, discuss, estimate, aggregate) and the Delphi method use structured rounds of anonymous estimates and feedback; SHELF (the Sheffield Elicitation Framework) brings experts to a behavioural consensus distribution with a facilitator. Each protocol specifies how experts are selected, how questions are framed, how distributions are elicited (quantiles, intervals, probabilities) and how judgements are combined. The field's recurring argument is whether experts should be weighted by calibration or treated equally, and its discipline is the documentation that lets an elicitation be reviewed and repeated.
Measures
Quantities and units
| Quantity | Unit or scale | What to know |
|---|---|---|
| Calibration score of an expert | Probability (p-value of a chi-squared statistic) from the Classical Model, 0 to 1 | How often realisations of seed questions fall in the expert's stated quantile bins. Scores depend on the number and relevance of seed questions; with ten seeds the score separates well-calibrated from over-confident experts but has little resolution among the former. |
| Information score | Relative entropy of the expert's distribution against a background measure, dimensionless | How concentrated an expert's distributions are. High information with low calibration is over-confidence; the Classical Model multiplies the two so that calibration dominates. |
| Elicited quantiles of the target quantity | The quantity's own unit at stated quantiles (commonly 5th, 50th and 95th) | The raw product of an elicitation. Experts typically give intervals that are too narrow; empirical hit rates for 90 per cent intervals are often nearer 50 to 70 per cent. |
| Between-expert spread and the aggregate distribution | The quantity's unit; width of the combined distribution | Spread between experts is information about epistemic uncertainty. Aggregates that are narrower than the spread of experts should be explained. |
| Robustness of the aggregate to removing experts or seeds | Change in the combined quantiles when one expert or seed question is dropped | A check that the result does not hinge on one person or one item. It is standard in Classical Model reporting and rare elsewhere. |
| Decision maker's own calibration and information | Same as for experts, computed for the combined distribution | The aggregate can be scored on the seed questions like an expert; a combination with better calibration than any individual is the usual result of performance weighting. |
పద్ధతులు
Methods and their limits
Each method is listed with what it assumes and where it fails.
| పద్ధతి | Assumes | Fails when |
|---|---|---|
| Classical Model (Cooke) with performance-based weights | That seed questions from the same domain with known answers predict performance on the target questions, that experts answer independently, and that a weighted linear pool is the right combination. | When seed questions are hard to find or are not representative of the targets, when experts collude or share a source, or when the number of experts is too small for weights to be stable. |
| IDEA protocol | That a first private estimate, a round of discussion that exposes reasoning, a revised private estimate and a mathematical aggregation reduce bias while keeping independence. | When discussion is dominated by senior voices, when the facilitator cannot keep the group on the quantity, or when the time available does not allow the second round. |
| Delphi method | That anonymous rounds with statistical feedback converge on a better group estimate than a meeting would. | When convergence is mistaken for accuracy, when the panel is selected for a view, or when the questions are vague so that experts answer different things. |
| SHELF behavioural aggregation | That experts can, with a facilitator and after seeing each other's judgements, agree a single distribution that a rational impartial observer would hold. | When one expert anchors the group, when the facilitator leads, or when the agreed distribution is narrower than the disagreements that produced it. |
| Eliciting distributions (quantile, probability, bisection and roulette methods) | That experts can report quantiles or probabilities with training and feedback, and that a parametric family can be fitted to the elicited points. | When experts are untrained, when the quantity is poorly defined (no clear resolution criterion), when anchoring on the first number asked distorts the rest, or when the fitted family imposes a tail the expert did not state. |
| Documentation and review of an elicitation | That the questions, briefing material, expert selection, raw judgements, aggregation rule and sensitivity checks are recorded so that another analyst can audit and repeat the exercise. | When only the aggregate is published, when experts are anonymous to the reader and to the reviewer, or when the elicitation is done once and never updated as evidence arrives. |
Debates
What the field debates
- Should experts be weighted by performance or treated equally?
- Cooke and Colson's cross-validation on the TU Delft database of more than eighty studies finds that performance-weighted combinations are better calibrated and more informative than equal weights out of sample, and that calibration on seeds carries over. Critics argue that seeds are often weak proxies for the target, that weights can hinge on a few questions, that equal weighting is more acceptable to experts and users, and that the out-of-sample gain is modest for most studies.
- Mathematical or behavioural aggregation?
- Mathematical pooling keeps the independence of judgements and allows scoring, and its advocates point to the Classical Model record. Behavioural approaches (SHELF, IDEA discussion rounds) hold that discussion surfaces evidence and definitions that private estimates miss, citing cases where the first-round spread reflected misreadings of the question; hybrid protocols that discuss and then pool are the common compromise.
- How far can expert judgement substitute for data in high-stakes assessment?
- Users such as the ice sheet assessment of Bamber and co-authors (2019) argue that structured elicitation is the only way to quantify tails that models cannot, and that it is transparent about its basis. Morgan (2014) warns that elicitation is often done badly, that experts share models and biases, and that a well-documented elicitation should be a complement to, and a prompt for, model development rather than a substitute.
Bodies and data
Bodies that set practice, and the data they publish
Bodies and standards are named so learners can find and apply their published work; the Academy teaches their use and links to the publisher, and no affiliation is implied.
| Body | Role in the field |
|---|---|
| Society for Risk Analysis (SRA) | The learned society where expert judgement methods are published and whose journal carries the Classical Model literature. |
| Delft University of Technology, Applied Probability group | Home of the Classical Model and the expert judgement database of applied studies used for cross-validation. |
| Intergovernmental Panel on Climate Change (IPCC) | User of structured expert judgement in sea level and other assessments and source of the guidance on expressing expert confidence. |
| United States Nuclear Regulatory Commission | Issued the Senior Seismic Hazard Analysis Committee guidance (NUREG/CR-6372 and NUREG-2117) that set formal expert elicitation practice for hazard analysis. |
| Royal Statistical Society | Professional body for the statistical treatment of subjective probability, calibration and the elicitation of priors. |
| Dataset | Steward |
|---|---|
| Good Judgment Open forecasting questions and track records | Good Judgment |
| Metaculus forecasting questions and resolutions | Metaculus |
| IPCC AR6 sea level projection tool | NASA Sea Level Change Team for the IPCC |
04పద్ధతి
From an expert's quantiles to a weight: seed questions, calibration, information and the aggregate that results
An expert states quantiles for seed questions whose answers are known; how often the truth falls inside the stated range gives a calibration score, how narrow the ranges are gives an information score, and their product is the weight the expert's judgement carries in the combined distribution for the question nobody can yet answer.
Sources: Cooke, Experts in Uncertainty: Opinion and Subjective Probability in Science (1991); Colson and Cooke, Cross validation for the classical model of structured expert judgment, Reliability Engineering and System Safety (2017); Aspinall, A route to more tractable expert advice, Nature (2010); Morgan, Use (and abuse) of expert elicitation in support of decision making for public policy, PNAS (2014); Hemming, Burgman, Hanea, McBride and Wintle, A practical guide to structured expert elicitation using the IDEA protocol, Methods in Ecology and Evolution (2018).
05At a glance
At a glance, hub and courses
- Build a twin of any placeTwin builderWorked example: the central Apennines: L'Aquila, Amatrice and Norcia · 4 generators · 7 layers · 4 instruments
3D గ్రాఫ్
ప్రమాద శాస్త్రం మరియు పద్ధతుల ఫ్యాకల్టీ
265 nodes1,691 relations9 views
06కన్సోల్
How Expert judgement and elicitation is taught, as a process you can run
Press Play to run this process from this page's own data.
నిపుణుల తీర్పు మరియు అభిప్రాయ సేకరణ
How Expert judgement and elicitation is taught
5 subjects in scope, 3 key terms, courses in development, 1 lab, in the Decision, Behaviour and Communication department of Risk Science and Methods: from the scope to the method and its terms, the courses that teach it, the labs on the twin, the assessment and the record.
-
01 Scope
- 01
Structured expert…
Start · the discipline
Structured expert judgement protocols
- 02
The Classical Model and…
Start · the discipline
The Classical Model and performance-based weighting
- 03
IDEA and Delphi…
Start · the discipline
IDEA and Delphi protocols
- 04
SHELF and behavioural…
Start · the discipline
SHELF and behavioural aggregation
- 05
Seed questions and…
Start · the discipline
Seed questions and expert calibration
- 01
-
02 Method and terms
- 06
From an expert's…
Model · the learner
An expert states quantiles for seed questions whose answers are known; how often the truth falls inside the stated range gives a…
- 07
Structured expert…
Record · the discipline
A documented protocol for eliciting and combining expert estimates so that the result can be reviewed.
- 09
Classical Model
Record · the discipline
Cooke's method that weights experts by their calibration and informativeness on seed questions with known answers.
- 11
Seed question
Record · the discipline
A question whose answer is known to the analyst but not the experts, used to score expert performance.
- 06
-
04 Labs
- 08
ప్రతిరూపం
Exercise · the learner, on the faculty twin
The method runs on the faculty twin and the admitted map layers.
- 08
-
05 Assess
- 10
Module checks
Gate · the course
Each module ends with a short check against its outcomes.
- 12
Assessed work
Decision · the learner
The assessed work that closes a course: a lab record, a case memo or an assignment, reviewed against the rubric.
- 10
-
06 Record
- 13
పూర్తి రికార్డు
Outcome · the learner
The record stacks into the discipline achievement.
- 14
Discipline achievement
Record · the Academy
The discipline achievement follows the discipline rule, read from the records.
- 13
Expert judgement and elicitation · Decision, Behaviour and Communication An outline of how the work is designed to run; it does not report the status of any decision or record.
07అభ్యసన
Achievements, where this is practised, practised in and across the ecosystem
Achievements
Achievements in this discipline
Course achievements
One for each course with this as its primary discipline, earned when you pass the course's assessed activities.
Department: Decision, Behaviour and Communication
Earn two discipline achievements in the Department of Decision, Behaviour and Communication. In development.
An achievement is a record of the learning completed and assessed here. How recognition works.
Where this is practised
Where the subject is practised
A field platform page on this subject is in development.
Practised in
The two Foundry systems where it is practised
- FoundryNXS-DSS systemExpert estimates enter decisions as stated distributions
- NXS-CMD systemPanels of experts deliberate and decide in council
Foundry system pages describe what each system and its rooms are designed to do; the courses teach the methods these systems apply.
Across the ecosystem
Across the ecosystem
The science, platforms and standards this discipline draws on and contributes to across the ecosystem.
Foundations in the STEM Commons
- The Global Centre for Risk and Innovation (GCRI Canada)Decision analysis under uncertaintyDecision analysis under uncertainty provides structured methods for choosing among options when outcomes are uncertain and objectives conflict.
- Probability and statisticsProbability and statistics provide the mathematical framework for describing uncertainty and learning from data.
- Uncertainty quantification and model validationUncertainty quantification and model validation characterise how uncertainty in inputs, parameters and model structure affects predictions.
Pages of the ecosystem, cited so learners can follow the field into research, platforms, standards and national practice.
08పాల్గొనండి
Take part in Risk Science and Methods
మీరు ఎవరో అక్కడ నుంచి ప్రారంభించండి.
Routes
మార్గం
జాతీయ సంస్థలు
Ministries, statistics offices and agencies: a national desk and chapter, and a seat in the working groups. National chapters
మార్గం
Sectors and professions
Operators, regulators, banks, insurers and professional bodies whose work touches risk science and methods. For organisations
మార్గం
అభ్యసన
Courses and practice, with a record of every course completed. Free to study. Open learning
మార్గం
ప్రమాణాలు
Draft, review and adopt the methods and data standards with the working groups. Working groups
మార్గం
Communities and research
Universities, civil society and individual experts: projects, data and review. Write to the Academy
Open calls
| Open call | Seeks | Closes |
|---|---|---|
| Founding call for experts to the working groups CALL-FOUNDING-EXPERTS | expert, editor, chair | 2026-12-15 |
| Open nomination for the three independent expert seats of the Global Standards Council CALL-GSC-IND | మండలి సభ్యులు | 2026-12-01 |
| Nominations of interim regional members (applies only if the transitional provision is decided) CALL-GSC-REG | మండలి సభ్యులు | 2026-12-15 |
| Review of the wave-1 risk core outlines (C1, C2, C3, C4, C8, C10, C11) CALL-REVIEW-WAVE1 | contributor | 2027-01-31 |
| ప్రమాద శాస్త్రం యొక్క ప్రధాన ప్రమాణాల కోసం నిపుణులు: అనిశ్చితి మరియు ఆధార శ్రేణీకరణ, స్కేల్లు, నిర్మాణాత్మక నిపుణుల తీర్పు, సహనీయత, తీవ్రతలు మరియు సమ్మేళన సంఘటనలు, విపత్కర మరియు అస్తిత్వ ప్రమాదం, నైతికత మరియు సమానత్వం, యూనిట్లు మరియు కొలత (NRS-C17 నుండి NRS-C24) CALL-CORE-SCIENCE | expert, editor, chair | 2027-01-31 |
| Independent reviewers for the families of the hazard atlas, three from at least two regions for each family (NHS2-R19), priority families first CALL-ATLAS-REVIEW | నిపుణులు | 2027-03-31 |
| తాత్కాలిక అగ్రగామి భద్రతా సమీక్ష ప్యానెల్ సభ్యులు మరియు రెడ్-టీమ్ సమీక్షకులు (జీవ భద్రత మరియు జీవ రక్షణ, AI భద్రత మరియు రక్షణ, కీలక మౌలిక సదుపాయాల భద్రత); దరఖాస్తులు గోప్యంగా పరిగణించబడతాయి CALL-FRONTIER-SAFETY-REVIEW | నిపుణులు | 2026-11-30 |
| అవసరాల రిజిస్టర్, NHS-2 యంత్ర ప్రాతినిధ్యం, అనుగుణత పరీక్షా సమూహాలు మరియు అనుగుణత అంచనా పథకం కోసం ఒంటాలజీ, మెట్రాలజీ, సాఫ్ట్వేర్ ధృవీకరణ మరియు అనుగుణత అంచనాలో నిపుణులు CALL-CONFORMANCE-ONTOLOGY | expert, editor | 2027-01-31 |
Calls past their closing date are hidden automatically. Every call, proposal and route: open areas and calls.
Cells and CERTs
Gather the cell
Bring together three to seven people with complementary skills: data, field practice, policy and communication.
Register it
Write to the team with the cell's name, members, area and region. You receive a reference and a contact. Write to the Academy
Choose a charter
Pick one focus: research and evidence, data and maps, standards review, or community preparedness.
Learn together
Follow shared study paths at Risk Academy, with practice and a record of every course completed. Hazards and resilience
Lead and report
Hold regular sessions. Publish what the cell learns through Reports. Bring findings to the working groups. Working groups
Specialise as a Community Emergency Response Team (CERT)
A CERT prepares its neighbourhood for the hazards it faces. It builds a local hazard profile from this site's map. It trains in household and street preparedness. It learns first aid and light search through recognised local providers, and it runs drills. A CERT works alongside the local emergency services and follows their direction. It never replaces them and is not an emergency service.
సూచిక
What a national team or a local community can do on The Global Risks Index (NXS-GRX) today.
NXS-GRX
Your country's profile
Open risk profiles for 195 countries, with sources and licences on every value. Open the Index
NXS-GRX
Correct or reply
Correct a value or reply for your country. The change is logged with its effect. Open the Index
NXS-GRX
Offer data
Offer a dataset. It enters through the licence gate, with your credit. Open the Index
NXS-GRX
Use the open data
Layers, profiles and tiles through the open API. Open the Index
09References
Core references and where to begin
Start with Aspinall, then Morgan, then O'Hagan.
- Cooke, R. M., Experts in Uncertainty: Opinion and Subjective Probability in Science, Oxford University Press (1991). global.oup.com
- O'Hagan, A. et al., Uncertain Judgements: Eliciting Experts' Probabilities, Wiley (2006). DOI
- Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F. and Wintle, B. C., A practical guide to structured expert elicitation using the IDEA protocol, Methods in Ecology and Evolution (2018). DOI
- Colson, A. R. and Cooke, R. M., Cross validation for the classical model of structured expert judgment, Reliability Engineering and System Safety (2017). DOI
- Morgan, M. G., Use (and abuse) of expert elicitation in support of decision making for public policy, Proceedings of the National Academy of Sciences (2014). DOI
- Aspinall, W., A route to more tractable expert advice, Nature (2010). DOI
- Clemen, R. T. and Winkler, R. L., Combining probability distributions from experts in risk analysis, Risk Analysis (1999). DOI
- Bamber, J. L., Oppenheimer, M., Kopp, R. E., Aspinall, W. P. and Cooke, R. M., Ice sheet contributions to future sea-level rise from structured expert judgment, Proceedings of the National Academy of Sciences (2019). DOI
- Gosling, J. P., SHELF: the Sheffield Elicitation Framework, in Elicitation: The Science and Art of Structuring Judgement, Springer (2018). DOI
- Dalkey, N. and Helmer, O., An experimental application of the Delphi method to the use of experts, Management Science (1963). DOI
- Lichtenstein, S., Fischhoff, B. and Phillips, L. D., Calibration of probabilities: the state of the art to 1980, in Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press (1982). DOI
10ప్రశ్నలు
ప్రశ్నలు, జవాబులు మరియు ముఖ్య పదాలు
ఇక్కడ అధ్యయనం ఉచితమా?
అవును. Nexus Consortium నడిపించే Risk Academy యొక్క ప్రతి కోర్సు మరియు అభ్యసన సామగ్రి ఉచితంగా చదవవచ్చు, సొంత వేగంతో లేదా బృందంతో. సాధారణ భాగస్వామ్యానికి రుసుము లేదు; చెల్లింపు పని, ఇంటర్న్షిప్లు మరియు ఫెలోషిప్లు వేరే ఏర్పాట్లు.
ఒక కోర్సు పూర్తి చేస్తే నాకు ఏమి లభిస్తుంది?
A completion record: the course, its version, the date and what was assessed, private by default and shared by you. It stacks into discipline, department and programme achievements. A record states what was studied and assessed and does not certify competence; it is not a licence, a professional title or an employment status.
డేటా మరియు ప్రదేశాలు ఎక్కడి నుంచి వస్తాయి?
Every faculty teaches on a twin of a real place built from open data; each layer carries its provider, licence and date, and every result on the Academy states its source. Maps and figures are indicators for teaching, never an assessment of a place.
How do I take part beyond the courses in expert judgement and elicitation?
Form a competence cell of three to seven people, study a shared pathway, take a supervised assignment in the National Working Groups, or bring a cohort from your institution. For organisations.
ముఖ్య పదాలు
- నిర్మాణాత్మక నిపుణుల తీర్పు
- A documented protocol for eliciting and combining expert estimates so that the result can be reviewed.
- Classical Model
- Cooke's method that weights experts by their calibration and informativeness on seed questions with known answers.
- Seed question
- A question whose answer is known to the analyst but not the experts, used to score expert performance.
- Delphi method
- An iterative, anonymous survey of experts with controlled feedback between rounds.
నివేదికలుసహకరించండి
ఈ పేజీ నుండి ఆధారాలను అందించండి.
Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
Risk Academy
- EmailGateway GW-ACADEMY Send it from this page Evidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen deposit Deposit it and submit to the community Publish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsRisk Academy See every way to contribute The contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.
నిపుణుల తీర్పు మరియు అభిప్రాయ సేకరణ
Prepare with a related course
Open courses in connected disciplines teach the methods this discipline will build on.
కోర్సు కేటలాగ్ను బ్రౌజ్ చేయండిBack to Department of Decision, Behaviour and Communication