Skip to content

Risk Academy, led by the Nexus Consortium · Risk Science and Methods · Department of Decision, Behaviour and Communication

Turning expert knowledge into a reviewable distribution

ماہرانہ رائے اور اس کا حصول

Where data are thin, risk still has to be quantified, and expert judgement is the method of record.

Map expert judgement and elicitation on a real placeSee the courses

Illustrative: a joint probability density whose dependence shifts over time

01دائرہ کار

What the discipline covers

The subtopics its courses and labs teach.

کورسز
In preparation
سطحیں
In development
تجربہ گاہیں
In development
  • Structured expert judgement protocols
  • The Classical Model and performance-based weighting
  • IDEA and Delphi protocols
  • SHELF and behavioural aggregation
  • Seed questions and expert calibration
  • Heuristics and biases in judgement
  • Eliciting probability distributions
  • Documenting and reviewing an elicitation
  • Expert judgement in catastrophe and climate assessment

02نقشے

Build a twin of any place

Search any place or open the world; the layers and generators follow expert judgement and elicitation on the planet map, the globe and in 3D.

The central Apennines: L'Aquila, Amatrice and Norcia: a flat plan of the faculty twin. Terrain 24 to 2,805 m; 3 reference earthquakes, M6.2 to M6.6; Earthquakes M4.5 and above, past week: read live when the 3D twin opens; USGS ShakeMap intensity contours: read live when the 3D twin opens. Contours every 500 metres from the elevation grid; reference earthquakes as marks.
The central Apennines: L'Aquila, Amatrice and Norcia, Italy. Flat plan of the same data: relief, the three reference epicentres with magnitude and depth, and the numbers below; the sampler's figure shows one precomputed field with its parameters. The 3D twin opens on devices that can draw it, once its data and code have been checked.
  • Terrain 24 to 2,805 m
  • 3 reference earthquakes, M6.2 to M6.6
  • Earthquakes M4.5 and above, past week: read live when the 3D twin opens
  • USGS ShakeMap intensity contours: read live when the 3D twin opens
Credits and dates per layer are listed under the stage. Relief in the twin is exaggerated ×1.5 for legibility.

Drag to move, right drag or Shift with a drag to turn and tilt, Ctrl with the wheel to zoom. Keyboard: arrows move, Shift with arrows turn and tilt, + and – zoom, 0 resets, V steps through the views, Escape clears.

Flat plan of the same data: relief, the three reference epicentres with magnitude and depth, and the numbers below; the sampler's figure shows one precomputed field with its parameters. The 3D twin opens on devices that can draw it, once its data and code have been checked.

The boundaries, names and designations shown do not imply any judgement on the legal status of any territory, or any endorsement or acceptance of such boundaries. Countries: Natural Earth (public domain).

03The subject

The subject, methods and measures and debates and bodies

The discipline

The field and its scope

For analysts, assessment authors and decision makers who must estimate uncertain quantities from experts when data are thin, and do it in a way that others can review.

Expert judgement and elicitation study how the knowledge of experts is turned into probability distributions for quantities that data alone cannot determine: the failure rate of a new component, the sea level contribution of an ice sheet, the case fatality ratio of a new pathogen, the chance of a volcanic eruption in the next year. The field treats expert judgement as data to be collected under a protocol, with the same concerns for bias, validation and documentation that apply to any measurement. It grew from nuclear safety and volcanology, and now serves climate assessment, catastrophe modelling, public health, ecology and finance.

Practice is organised around protocols. The Classical Model of Cooke weights experts by their performance on seed questions with known answers; the IDEA protocol (investigate, discuss, estimate, aggregate) and the Delphi method use structured rounds of anonymous estimates and feedback; SHELF (the Sheffield Elicitation Framework) brings experts to a behavioural consensus distribution with a facilitator. Each protocol specifies how experts are selected, how questions are framed, how distributions are elicited (quantiles, intervals, probabilities) and how judgements are combined. The field's recurring argument is whether experts should be weighted by calibration or treated equally, and its discipline is the documentation that lets an elicitation be reviewed and repeated.

Measures

Quantities and units

QuantityUnit or scaleWhat to know
Calibration score of an expertProbability (p-value of a chi-squared statistic) from the Classical Model, 0 to 1How often realisations of seed questions fall in the expert's stated quantile bins. Scores depend on the number and relevance of seed questions; with ten seeds the score separates well-calibrated from over-confident experts but has little resolution among the former.
Information scoreRelative entropy of the expert's distribution against a background measure, dimensionlessHow concentrated an expert's distributions are. High information with low calibration is over-confidence; the Classical Model multiplies the two so that calibration dominates.
Elicited quantiles of the target quantityThe quantity's own unit at stated quantiles (commonly 5th, 50th and 95th)The raw product of an elicitation. Experts typically give intervals that are too narrow; empirical hit rates for 90 per cent intervals are often nearer 50 to 70 per cent.
Between-expert spread and the aggregate distributionThe quantity's unit; width of the combined distributionSpread between experts is information about epistemic uncertainty. Aggregates that are narrower than the spread of experts should be explained.
Robustness of the aggregate to removing experts or seedsChange in the combined quantiles when one expert or seed question is droppedA check that the result does not hinge on one person or one item. It is standard in Classical Model reporting and rare elsewhere.
Decision maker's own calibration and informationSame as for experts, computed for the combined distributionThe aggregate can be scored on the seed questions like an expert; a combination with better calibration than any individual is the usual result of performance weighting.

طریقے

Methods and their limits

Each method is listed with what it assumes and where it fails.

طریقہAssumesFails when
Classical Model (Cooke) with performance-based weightsThat seed questions from the same domain with known answers predict performance on the target questions, that experts answer independently, and that a weighted linear pool is the right combination.When seed questions are hard to find or are not representative of the targets, when experts collude or share a source, or when the number of experts is too small for weights to be stable.
IDEA protocolThat a first private estimate, a round of discussion that exposes reasoning, a revised private estimate and a mathematical aggregation reduce bias while keeping independence.When discussion is dominated by senior voices, when the facilitator cannot keep the group on the quantity, or when the time available does not allow the second round.
Delphi methodThat anonymous rounds with statistical feedback converge on a better group estimate than a meeting would.When convergence is mistaken for accuracy, when the panel is selected for a view, or when the questions are vague so that experts answer different things.
SHELF behavioural aggregationThat experts can, with a facilitator and after seeing each other's judgements, agree a single distribution that a rational impartial observer would hold.When one expert anchors the group, when the facilitator leads, or when the agreed distribution is narrower than the disagreements that produced it.
Eliciting distributions (quantile, probability, bisection and roulette methods)That experts can report quantiles or probabilities with training and feedback, and that a parametric family can be fitted to the elicited points.When experts are untrained, when the quantity is poorly defined (no clear resolution criterion), when anchoring on the first number asked distorts the rest, or when the fitted family imposes a tail the expert did not state.
Documentation and review of an elicitationThat the questions, briefing material, expert selection, raw judgements, aggregation rule and sensitivity checks are recorded so that another analyst can audit and repeat the exercise.When only the aggregate is published, when experts are anonymous to the reader and to the reviewer, or when the elicitation is done once and never updated as evidence arrives.

Debates

What the field debates

Should experts be weighted by performance or treated equally?
Cooke and Colson's cross-validation on the TU Delft database of more than eighty studies finds that performance-weighted combinations are better calibrated and more informative than equal weights out of sample, and that calibration on seeds carries over. Critics argue that seeds are often weak proxies for the target, that weights can hinge on a few questions, that equal weighting is more acceptable to experts and users, and that the out-of-sample gain is modest for most studies.
Mathematical or behavioural aggregation?
Mathematical pooling keeps the independence of judgements and allows scoring, and its advocates point to the Classical Model record. Behavioural approaches (SHELF, IDEA discussion rounds) hold that discussion surfaces evidence and definitions that private estimates miss, citing cases where the first-round spread reflected misreadings of the question; hybrid protocols that discuss and then pool are the common compromise.
How far can expert judgement substitute for data in high-stakes assessment?
Users such as the ice sheet assessment of Bamber and co-authors (2019) argue that structured elicitation is the only way to quantify tails that models cannot, and that it is transparent about its basis. Morgan (2014) warns that elicitation is often done badly, that experts share models and biases, and that a well-documented elicitation should be a complement to, and a prompt for, model development rather than a substitute.

Bodies and data

Bodies that set practice, and the data they publish

Bodies and standards are named so learners can find and apply their published work; the Academy teaches their use and links to the publisher, and no affiliation is implied.

BodyRole in the field
Society for Risk Analysis (SRA)The learned society where expert judgement methods are published and whose journal carries the Classical Model literature.
Delft University of Technology, Applied Probability groupHome of the Classical Model and the expert judgement database of applied studies used for cross-validation.
Intergovernmental Panel on Climate Change (IPCC)User of structured expert judgement in sea level and other assessments and source of the guidance on expressing expert confidence.
United States Nuclear Regulatory CommissionIssued the Senior Seismic Hazard Analysis Committee guidance (NUREG/CR-6372 and NUREG-2117) that set formal expert elicitation practice for hazard analysis.
Royal Statistical SocietyProfessional body for the statistical treatment of subjective probability, calibration and the elicitation of priors.

04طریقہ

From an expert's quantiles to a weight: seed questions, calibration, information and the aggregate that results

An expert states quantiles for seed questions whose answers are known; how often the truth falls inside the stated range gives a calibration score, how narrow the ranges are gives an information score, and their product is the weight the expert's judgement carries in the combined distribution for the question nobody can yet answer.

Sources: Cooke, Experts in Uncertainty: Opinion and Subjective Probability in Science (1991); Colson and Cooke, Cross validation for the classical model of structured expert judgment, Reliability Engineering and System Safety (2017); Aspinall, A route to more tractable expert advice, Nature (2010); Morgan, Use (and abuse) of expert elicitation in support of decision making for public policy, PNAS (2014); Hemming, Burgman, Hanea, McBride and Wintle, A practical guide to structured expert elicitation using the IDEA protocol, Methods in Ecology and Evolution (2018).

05At a glance

At a glance, hub and courses

3D گراف

فیکلٹی برائے سائنسِ خطرہ اور طریقۂ کار

265 nodes1,691 relations9 views

Explore the 3D graph: Faculty of Risk Science and Methods

06کنسول

How Expert judgement and elicitation is taught, as a process you can run

Press Play to run this process from this page's own data.

ماہرانہ رائے اور اس کا حصول

How Expert judgement and elicitation is taught

5 subjects in scope, 3 key terms, courses in development, 1 lab, in the Decision, Behaviour and Communication department of Risk Science and Methods: from the scope to the method and its terms, the courses that teach it, the labs on the twin, the assessment and the record.

  1. 01 Scope

    1. 01

      Structured expert…

      Start · the discipline

      Structured expert judgement protocols

    2. 02

      The Classical Model and…

      Start · the discipline

      The Classical Model and performance-based weighting

    3. 03

      IDEA and Delphi…

      Start · the discipline

      IDEA and Delphi protocols

    4. 04

      SHELF and behavioural…

      Start · the discipline

      SHELF and behavioural aggregation

    5. 05

      Seed questions and…

      Start · the discipline

      Seed questions and expert calibration

  2. 02 Method and terms

    1. 06

      From an expert's…

      Model · the learner

      An expert states quantiles for seed questions whose answers are known; how often the truth falls inside the stated range gives a…

    2. 07

      Structured expert…

      Record · the discipline

      A documented protocol for eliciting and combining expert estimates so that the result can be reviewed.

    3. 09

      Classical Model

      Record · the discipline

      Cooke's method that weights experts by their calibration and informativeness on seed questions with known answers.

    4. 11

      Seed question

      Record · the discipline

      A question whose answer is known to the analyst but not the experts, used to score expert performance.

  3. 04 Labs

    1. 08

      جڑواں ماڈل

      Exercise · the learner, on the faculty twin

      The method runs on the faculty twin and the admitted map layers.

  4. 05 Assess

    1. 10

      Module checks

      Gate · the course

      Each module ends with a short check against its outcomes.

    2. 12

      Assessed work

      Decision · the learner

      The assessed work that closes a course: a lab record, a case memo or an assignment, reviewed against the rubric.

  5. 06 Record

    1. 13

      تکمیل کا ریکارڈ

      Outcome · the learner

      The record stacks into the discipline achievement.

    2. 14

      Discipline achievement

      Record · the Academy

      The discipline achievement follows the discipline rule, read from the records.

Expert judgement and elicitation · Decision, Behaviour and Communication An outline of how the work is designed to run; it does not report the status of any decision or record.

07تعلیم

Achievements, where this is practised, practised in and across the ecosystem

Achievements

Achievements in this discipline

  1. Course achievements

    One for each course with this as its primary discipline, earned when you pass the course's assessed activities.

  2. Department: Decision, Behaviour and Communication

    Earn two discipline achievements in the Department of Decision, Behaviour and Communication. In development.

An achievement is a record of the learning completed and assessed here. How recognition works.

Where this is practised

Where the subject is practised

A field platform page on this subject is in development.

Practised in

The two Foundry systems where it is practised

  • FoundryNXS-DSS systemExpert estimates enter decisions as stated distributions
  • NXS-CMD systemPanels of experts deliberate and decide in council

Foundry system pages describe what each system and its rooms are designed to do; the courses teach the methods these systems apply.

Across the ecosystem

Across the ecosystem

The science, platforms and standards this discipline draws on and contributes to across the ecosystem.

Foundations in the STEM Commons

  • The Global Centre for Risk and Innovation (GCRI Canada)Decision analysis under uncertaintyDecision analysis under uncertainty provides structured methods for choosing among options when outcomes are uncertain and objectives conflict.
  • Probability and statisticsProbability and statistics provide the mathematical framework for describing uncertainty and learning from data.
  • Uncertainty quantification and model validationUncertainty quantification and model validation characterise how uncertainty in inputs, parameters and model structure affects predictions.

Pages of the ecosystem, cited so learners can follow the field into research, platforms, standards and national practice.

08حصہ لیں

Take part in Risk Science and Methods

اپنی شناخت سے شروع کریں۔

Routes

  • راستہ

    قومی ادارے

    Ministries, statistics offices and agencies: a national desk and chapter, and a seat in the working groups. National chapters

  • راستہ

    Sectors and professions

    Operators, regulators, banks, insurers and professional bodies whose work touches risk science and methods. For organisations

  • راستہ

    تعلیم

    Courses and practice, with a record of every course completed. Free to study. Open learning

  • راستہ

    معیارات

    Draft, review and adopt the methods and data standards with the working groups. Working groups

  • راستہ

    Communities and research

    Universities, civil society and individual experts: projects, data and review. Write to the Academy

Open calls

Open callSeeksCloses
Founding call for experts to the working groups CALL-FOUNDING-EXPERTSexpert, editor, chair2026-12-15
Open nomination for the three independent expert seats of the Global Standards Council CALL-GSC-INDکونسل کا رکن2026-12-01
Nominations of interim regional members (applies only if the transitional provision is decided) CALL-GSC-REGکونسل کا رکن2026-12-15
Review of the wave-1 risk core outlines (C1, C2, C3, C4, C8, C10, C11) CALL-REVIEW-WAVE1contributor2027-01-31
Experts for the core standards of risk science: uncertainty and evidence grading, scales, structured expert judgement, tolerability, extremes and compound events, catastrophic and existential risk, ethics and equity, units and measurement (NRS-C17 to NRS-C24) CALL-CORE-SCIENCEexpert, editor, chair2027-01-31
Independent reviewers for the families of the hazard atlas, three from at least two regions for each family (NHS2-R19), priority families first CALL-ATLAS-REVIEWماہر2027-03-31
Members of the interim Frontier Safety Review Panel and red-team reviewers (biosafety and biosecurity, AI safety and security, critical-infrastructure security); applications treated in confidence CALL-FRONTIER-SAFETY-REVIEWماہر2026-11-30
Experts in ontology, metrology, software verification and conformity assessment for the requirement register, the NHS-2 machine representation, the conformance test suites and the conformity assessment scheme CALL-CONFORMANCE-ONTOLOGYexpert, editor2027-01-31

Calls past their closing date are hidden automatically. Every call, proposal and route: open areas and calls.

Cells and CERTs

  1. Gather the cell

    Bring together three to seven people with complementary skills: data, field practice, policy and communication.

  2. Register it

    Write to the team with the cell's name, members, area and region. You receive a reference and a contact. Write to the Academy

  3. Choose a charter

    Pick one focus: research and evidence, data and maps, standards review, or community preparedness.

  4. Learn together

    Follow shared study paths at Risk Academy, with practice and a record of every course completed. Hazards and resilience

  5. Lead and report

    Hold regular sessions. Publish what the cell learns through Reports. Bring findings to the working groups. Working groups

  • Specialise as a Community Emergency Response Team (CERT)

    A CERT prepares its neighbourhood for the hazards it faces. It builds a local hazard profile from this site's map. It trains in household and street preparedness. It learns first aid and light search through recognised local providers, and it runs drills. A CERT works alongside the local emergency services and follows their direction. It never replaces them and is not an emergency service.

انڈیکس

What a national team or a local community can do on The Global Risks Index (NXS-GRX) today.

  • NXS-GRX

    Your country's profile

    Open risk profiles for 195 countries, with sources and licences on every value. Open the Index

  • NXS-GRX

    Correct or reply

    Correct a value or reply for your country. The change is logged with its effect. Open the Index

  • NXS-GRX

    Offer data

    Offer a dataset. It enters through the licence gate, with your credit. Open the Index

  • NXS-GRX

    Use the open data

    Layers, profiles and tiles through the open API. Open the Index

09References

Core references and where to begin

Start with Aspinall, then Morgan, then O'Hagan.

  1. Cooke, R. M., Experts in Uncertainty: Opinion and Subjective Probability in Science, Oxford University Press (1991). global.oup.com
  2. O'Hagan, A. et al., Uncertain Judgements: Eliciting Experts' Probabilities, Wiley (2006). DOI
  3. Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F. and Wintle, B. C., A practical guide to structured expert elicitation using the IDEA protocol, Methods in Ecology and Evolution (2018). DOI
  4. Colson, A. R. and Cooke, R. M., Cross validation for the classical model of structured expert judgment, Reliability Engineering and System Safety (2017). DOI
  5. Morgan, M. G., Use (and abuse) of expert elicitation in support of decision making for public policy, Proceedings of the National Academy of Sciences (2014). DOI
  6. Aspinall, W., A route to more tractable expert advice, Nature (2010). DOI
  7. Clemen, R. T. and Winkler, R. L., Combining probability distributions from experts in risk analysis, Risk Analysis (1999). DOI
  8. Bamber, J. L., Oppenheimer, M., Kopp, R. E., Aspinall, W. P. and Cooke, R. M., Ice sheet contributions to future sea-level rise from structured expert judgment, Proceedings of the National Academy of Sciences (2019). DOI
  9. Gosling, J. P., SHELF: the Sheffield Elicitation Framework, in Elicitation: The Science and Art of Structuring Judgement, Springer (2018). DOI
  10. Dalkey, N. and Helmer, O., An experimental application of the Delphi method to the use of experts, Management Science (1963). DOI
  11. Lichtenstein, S., Fischhoff, B. and Phillips, L. D., Calibration of probabilities: the state of the art to 1980, in Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press (1982). DOI

10سوالات

سوالات، جوابات اور کلیدی اصطلاحات

کیا یہاں تعلیم مفت ہے؟

جی ہاں۔ Risk Academy، جس کی قیادت Nexus Consortium کرتا ہے، کا ہر کورس اور تعلیمی مواد مفت ہے، اپنی رفتار سے یا کسی گروپ کے ساتھ۔ عام شرکت کی کوئی فیس نہیں؛ معاوضے والا کام، انٹرن شپس اور فیلوشپس الگ انتظامات ہیں۔

کورس مکمل کرنے سے مجھے کیا ملتا ہے؟

A completion record: the course, its version, the date and what was assessed, private by default and shared by you. It stacks into discipline, department and programme achievements. A record states what was studied and assessed and does not certify competence; it is not a licence, a professional title or an employment status.

ڈیٹا اور مقامات کہاں سے آتے ہیں؟

Every faculty teaches on a twin of a real place built from open data; each layer carries its provider, licence and date, and every result on the Academy states its source. Maps and figures are indicators for teaching, never an assessment of a place.

How do I take part beyond the courses in expert judgement and elicitation?

Form a competence cell of three to seven people, study a shared pathway, take a supervised assignment in the National Working Groups, or bring a cohort from your institution. For organisations.

کلیدی اصطلاحات

منظم ماہرانہ رائے
A documented protocol for eliciting and combining expert estimates so that the result can be reviewed.
Classical Model
Cooke's method that weights experts by their calibration and informativeness on seed questions with known answers.
Seed question
A question whose answer is known to the analyst but not the experts, used to score expert performance.
Delphi method
An iterative, anonymous survey of experts with controlled feedback between rounds.

رپورٹیںتعاون کریں

اس صفحے سے شواہد فراہم کریں۔

Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

Risk Academy

Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.

ماہرانہ رائے اور اس کا حصول

Prepare with a related course

Open courses in connected disciplines teach the methods this discipline will build on.

کورس کیٹلاگ دیکھیںBack to Department of Decision, Behaviour and Communication