Skip to content

Risk Academy, led by the Nexus Consortium · Risk Science and Methods · Department of Quantitative Risk Analysis

Forecasts as distributions, judged by proper scoring rules

امکانی پیش گوئی اور اس کی توثیق

A risk number is a forecast, and a forecast is only as good as its verification.

Map probabilistic forecasting and verification on a real placeSee the courses

Illustrative: an ensemble of trajectories, its quantile fan and the outcome as it is observed

01دائرہ کار

What the discipline covers

The subtopics its courses and labs teach.

کورسز
In preparation
سطحیں
In development
تجربہ گاہیں
In development
  • Probabilistic and ensemble forecasts
  • Proper scoring rules (Brier, CRPS, logarithmic)
  • Calibration, sharpness and reliability diagrams
  • Discrimination, ROC curves and hit and false alarm rates
  • Verification of categorical warnings and triggers
  • Forecast value and the cost-loss model
  • Post-processing and statistical calibration of ensembles
  • Machine-learning weather and hazard forecasts and their verification
  • Forecast combination and model averaging

02نقشے

Build a twin of any place

Search any place or open the world; the layers and generators follow probabilistic forecasting and verification on the planet map, the globe and in 3D.

The central Apennines: L'Aquila, Amatrice and Norcia: a flat plan of the faculty twin. Terrain 24 to 2,805 m; 3 reference earthquakes, M6.2 to M6.6; Earthquakes M4.5 and above, past week: read live when the 3D twin opens; USGS ShakeMap intensity contours: read live when the 3D twin opens. Contours every 500 metres from the elevation grid; reference earthquakes as marks.
The central Apennines: L'Aquila, Amatrice and Norcia, Italy. Flat plan of the same data: relief, the three reference epicentres with magnitude and depth, and the numbers below; the sampler's figure shows one precomputed field with its parameters. The 3D twin opens on devices that can draw it, once its data and code have been checked.
  • Terrain 24 to 2,805 m
  • 3 reference earthquakes, M6.2 to M6.6
  • Earthquakes M4.5 and above, past week: read live when the 3D twin opens
  • USGS ShakeMap intensity contours: read live when the 3D twin opens
Credits and dates per layer are listed under the stage. Relief in the twin is exaggerated ×1.5 for legibility.

Drag to move, right drag or Shift with a drag to turn and tilt, Ctrl with the wheel to zoom. Keyboard: arrows move, Shift with arrows turn and tilt, + and – zoom, 0 resets, V steps through the views, Escape clears.

Flat plan of the same data: relief, the three reference epicentres with magnitude and depth, and the numbers below; the sampler's figure shows one precomputed field with its parameters. The 3D twin opens on devices that can draw it, once its data and code have been checked.

The boundaries, names and designations shown do not imply any judgement on the legal status of any territory, or any endorsement or acceptance of such boundaries. Countries: Natural Earth (public domain).

03The subject

The subject, methods and measures and debates and bodies

The discipline

The field and its scope

For forecasters, early warning analysts, modellers and the people who act on forecasts, who need to issue risk as a calibrated distribution and to judge forecasts fairly against what happened.

Probabilistic forecasting and verification study how a future quantity or event is forecast as a probability distribution rather than a single value, and how such forecasts are scored against outcomes so that better forecasts get better scores. The field was built in meteorology, where ensembles of model runs replaced deterministic forecasts in the 1990s, and its methods now serve hydrology, epidemiology, energy, finance, humanitarian early warning and the anticipatory action triggers that release funds before a shock. The principle set out by Gneiting, Balabdaoui and Raftery (2007) is to maximise sharpness subject to calibration: forecasts should be as decisive as possible while the stated probabilities match observed frequencies.

The discipline is organised around proper scoring rules (Brier, logarithmic, continuous ranked probability score), the decomposition of scores into reliability, resolution and uncertainty, diagrams for calibration and discrimination, contingency-table statistics for warnings, statistical post-processing of ensembles, and the economic value of a forecast under a cost-loss decision. Machine learning weather models have made verification the arbiter of a live contest between physical and learned forecasting systems, and the field's open questions are about verifying rare extremes, spatial forecasts and forecasts that change what they predict.

Measures

Quantities and units

QuantityUnit or scaleWhat to know
Brier score and Brier skill scoreDimensionless, 0 (perfect) to 1 for the score; skill relative to a reference forecastThe mean squared error of a probability forecast for a binary event, decomposable into reliability, resolution and uncertainty. For rare events the score is dominated by the climatological term, so skill scores against climatology are reported and can be unstable with few events.
Continuous ranked probability score (CRPS)Same unit as the forecast variableThe integral of the squared difference between forecast and observed cumulative distributions; it reduces to the absolute error for a point forecast. Sampling uncertainty is reported by bootstrapping over forecast cases.
Reliability (calibration) and sharpnessObserved frequency against forecast probability, dimensionless; width of forecast intervals in the variable's unitReliability diagrams show whether a 30 per cent forecast verifies about 30 per cent of the time; sharpness is how far the forecasts are from climatology. Both can be read only with enough cases per probability bin.
Hit rate, false alarm ratio and the relative operating characteristicProportions, dimensionless; area under the ROC curve from 0.5 to 1Contingency-table statistics for warnings at a threshold. The ROC measures discrimination and is insensitive to calibration; a well-discriminating system can still be badly calibrated.
Relative economic valueFraction of the value of a perfect forecast, by cost-loss ratioThe value of a forecast to a user who pays a cost to protect against a loss. It depends on the user's cost-loss ratio, so one forecast can be worth much to one user and nothing to another.
Lead time and spread-skill relationHours or days; ensemble spread against root mean square errorHow far ahead skill persists and whether ensemble spread tracks error. Under-dispersion at short lead times is common and is corrected by post-processing.

طریقے

Methods and their limits

Each method is listed with what it assumes and where it fails.

طریقہAssumesFails when
Ensemble forecasting with perturbed initial conditions and model physicsThat the spread of an ensemble represents forecast uncertainty, that members are exchangeable, and that the ensemble size resolves the probabilities of interest.When the ensemble is under-dispersive because model error is not sampled, when extremes need probabilities below the resolution of the member count, or when all members share a systematic bias.
Proper scoring rulesThat the score is strictly proper so the forecaster's best strategy is to report their true belief, and that enough independent cases are available for the score to discriminate between systems.When forecasts are compared across different case sets, when a single score aggregates over regimes where skill differs, or when an improper measure (such as a threshold-based accuracy) is used and rewards hedging.
Statistical post-processing (ensemble model output statistics, Bayesian model averaging, quantile mapping)That the relation between raw ensemble output and observations is stable over the training period and the forecast period, and that the training set covers the conditions of interest.After a model upgrade, under non-stationary climate, or for extremes outside the training sample where the calibration must be extrapolated.
Warning verification with contingency tablesThat events and warnings can be matched in space and time, that the event definition is fixed, and that non-events are countable.When the user's response to a warning prevents the event (so the warning looks like a false alarm), when the observation network misses events, or when the non-event count is arbitrary and inflates the apparent accuracy.
Cost-loss analysis of forecast valueThat the user faces a known protection cost and a known loss, that decisions are binary, and that the forecast probability is the only information.When losses are uncertain or non-monetary, when actions are graded rather than binary, or when the user already has other information so that the forecast's marginal value is smaller than the model suggests.
Forecast combination and multi-model ensemblesThat different models have partly independent errors and that weights can be estimated from past performance.When the models share data assimilation or parameterisations, when past performance does not predict future performance, or when weighting is fitted to a short period and overfits.

Debates

What the field debates

Have machine learning weather models surpassed physical models, and how should the claim be verified?
Papers such as Lam and co-authors (2023) show lower root mean square error and CRPS than operational physical models on most variables and lead times on reanalysis-based benchmarks. Operational centres reply that reanalysis targets flatter smooth forecasts, that extremes and physical consistency are verified less well, and that verification against observations and in the operational cycle is the fair test; both sides now use shared benchmarks such as WeatherBench 2.
Should forecasts be judged by scores or by the decisions they support?
The scoring-rule tradition holds that proper scores are the only incentive-compatible measure and that value follows from calibration and resolution. The decision-analytic view, from Murphy's framework to the anticipatory action community, holds that a forecast with a modest score can be worth more to a user with a particular cost-loss ratio than a better-scoring one, so verification should report value curves and trigger performance as well.
How should rare extremes be verified when there are few cases?
One position favours scores restricted to the tail (threshold-weighted CRPS, extreme dependency scores) and pooling across space and time to gain cases, accepting dependence between them. The other holds that any score over a handful of events is dominated by sampling noise, that pooling hides regional failures, and that process-based evaluation and case studies are the honest complement.

Bodies and data

Bodies that set practice, and the data they publish

Bodies and standards are named so learners can find and apply their published work; the Academy teaches their use and links to the publisher, and no affiliation is implied.

BodyRole in the field
World Meteorological Organization (WMO) and its Joint Working Group on Forecast Verification ResearchSets the standard verification practices for operational centres and publishes guidelines on verification of forecasts and warnings.
European Centre for Medium-Range Weather Forecasts (ECMWF)Runs the reference global ensemble, publishes verification scores and reanalyses (ERA5), and makes open forecast data available.
National Oceanic and Atmospheric Administration (NOAA) and the National Weather ServiceOperates the Global Ensemble Forecast System and the United States verification programmes for weather and flood warnings.
Royal Statistical Society and the American Statistical AssociationProfessional bodies for the statistical theory of scoring rules, calibration and forecast evaluation.
Anticipation Hub and the Red Cross Red Crescent Climate CentreCoordinate practice on forecast-based triggers for anticipatory action and the verification of trigger performance.
DatasetSteward
ECMWF open forecast dataEuropean Centre for Medium-Range Weather Forecasts
ERA5 reanalysisCopernicus Climate Change Service, implemented by ECMWF
WeatherBench 2 benchmarkGoogle Research
TIGGE global ensemble archiveECMWF on behalf of the WMO THORPEX programme

04طریقہ

From a probability forecast to its value for a user: hits, false alarms and the cost-loss ratio that decides

A probability forecast becomes a warning at a threshold; verified against outcomes it has a hit rate and a false alarm rate; for a user who pays a cost to protect against a loss, those two rates and the climatological frequency give the forecast's economic value, which is positive only for users whose cost-loss ratio lies within a band.

Sources: Richardson, Skill and relative economic value of the ECMWF ensemble prediction system, Quarterly Journal of the Royal Meteorological Society (2000); Gneiting, Balabdaoui and Raftery, Probabilistic forecasts, calibration and sharpness, Journal of the Royal Statistical Society B (2007); Murphy, What is a good forecast? An essay on the nature of goodness in weather forecasting, Weather and Forecasting (1993); Jolliffe and Stephenson (eds), Forecast Verification: A Practitioner's Guide in Atmospheric Science (2012); Brier, Verification of forecasts expressed in terms of probability, Monthly Weather Review (1950).

05At a glance

At a glance, hub, what is changing now and courses

3D گراف

فیکلٹی برائے سائنسِ خطرہ اور طریقۂ کار

265 nodes1,691 relations9 views

Explore the 3D graph: Faculty of Risk Science and Methods

06کنسول

How Probabilistic forecasting and verification is taught, as a process you can run

Press Play to run this process from this page's own data.

امکانی پیش گوئی اور اس کی توثیق

How Probabilistic forecasting and verification is taught

5 subjects in scope, 3 key terms, courses in development, 1 lab, in the Quantitative Risk Analysis department of Risk Science and Methods: from the scope to the method and its terms, the courses that teach it, the labs on the twin, the assessment and the record.

  1. 01 Scope

    1. 01

      Probabilistic and…

      Start · the discipline

      Probabilistic and ensemble forecasts

    2. 02

      Proper scoring rules…

      Start · the discipline

      Proper scoring rules (Brier, CRPS, logarithmic)

    3. 03

      Calibration, sharpness…

      Start · the discipline

      Calibration, sharpness and reliability diagrams

    4. 04

      Discrimination, ROC…

      Start · the discipline

      Discrimination, ROC curves and hit and false alarm rates

    5. 05

      Verification of…

      Start · the discipline

      Verification of categorical warnings and triggers

  2. 02 Method and terms

    1. 06

      From a probability…

      Model · the learner

      A probability forecast becomes a warning at a threshold; verified against outcomes it has a hit rate and a false alarm rate; for a user…

    2. 07

      Proper scoring rule

      Record · the discipline

      A score that is optimised in expectation only when the forecaster reports their true probability, such as the Brier score or CRPS.

    3. 09

      Calibration

      Record · the discipline

      Agreement between forecast probabilities and observed frequencies: events forecast at 30 per cent happen about that often.

    4. 11

      Sharpness

      Record · the discipline

      How concentrated a forecast distribution is, judged only once calibration holds.

  3. 04 Labs

    1. 08

      جڑواں ماڈل

      Exercise · the learner, on the faculty twin

      The method runs on the faculty twin and the admitted map layers.

  4. 05 Assess

    1. 10

      Module checks

      Gate · the course

      Each module ends with a short check against its outcomes.

    2. 12

      Assessed work

      Decision · the learner

      The assessed work that closes a course: a lab record, a case memo or an assignment, reviewed against the rubric.

  5. 06 Record

    1. 13

      تکمیل کا ریکارڈ

      Outcome · the learner

      The record stacks into the discipline achievement.

    2. 14

      Discipline achievement

      Record · the Academy

      The discipline achievement follows the discipline rule, read from the records.

Probabilistic forecasting and verification · Quantitative Risk Analysis An outline of how the work is designed to run; it does not report the status of any decision or record.

07تعلیم

Achievements, where this is practised, practised in and across the ecosystem

Achievements

Achievements in this discipline

  1. Course achievements

    One for each course with this as its primary discipline, earned when you pass the course's assessed activities.

  2. Department: Quantitative Risk Analysis

    Earn two discipline achievements in the Department of Quantitative Risk Analysis. In development.

An achievement is a record of the learning completed and assessed here. How recognition works.

Where this is practised

Where the subject is practised

A field platform page on this subject is in development.

Practised in

The two Foundry systems where it is practised

  • FoundryNXS-EWS systemForecasts and warnings are judged by how they verify
  • NXS-LAB systemCompeting forecast methods are compared under transparent tests

Foundry system pages describe what each system and its rooms are designed to do; the courses teach the methods these systems apply.

Across the ecosystem

Across the ecosystem

The science, platforms and standards this discipline draws on and contributes to across the ecosystem.

Foundations in the STEM Commons

Pages of the ecosystem, cited so learners can follow the field into research, platforms, standards and national practice.

08حصہ لیں

Take part in Risk Science and Methods

اپنی شناخت سے شروع کریں۔

Routes

  • راستہ

    قومی ادارے

    Ministries, statistics offices and agencies: a national desk and chapter, and a seat in the working groups. National chapters

  • راستہ

    Sectors and professions

    Operators, regulators, banks, insurers and professional bodies whose work touches risk science and methods. For organisations

  • راستہ

    تعلیم

    Courses and practice, with a record of every course completed. Free to study. Open learning

  • راستہ

    معیارات

    Draft, review and adopt the methods and data standards with the working groups. Working groups

  • راستہ

    Communities and research

    Universities, civil society and individual experts: projects, data and review. Write to the Academy

Open calls

Open callSeeksCloses
Founding call for experts to the working groups CALL-FOUNDING-EXPERTSexpert, editor, chair2026-12-15
Open nomination for the three independent expert seats of the Global Standards Council CALL-GSC-INDکونسل کا رکن2026-12-01
Nominations of interim regional members (applies only if the transitional provision is decided) CALL-GSC-REGکونسل کا رکن2026-12-15
Review of the wave-1 risk core outlines (C1, C2, C3, C4, C8, C10, C11) CALL-REVIEW-WAVE1contributor2027-01-31
Experts for the core standards of risk science: uncertainty and evidence grading, scales, structured expert judgement, tolerability, extremes and compound events, catastrophic and existential risk, ethics and equity, units and measurement (NRS-C17 to NRS-C24) CALL-CORE-SCIENCEexpert, editor, chair2027-01-31
Independent reviewers for the families of the hazard atlas, three from at least two regions for each family (NHS2-R19), priority families first CALL-ATLAS-REVIEWماہر2027-03-31
Members of the interim Frontier Safety Review Panel and red-team reviewers (biosafety and biosecurity, AI safety and security, critical-infrastructure security); applications treated in confidence CALL-FRONTIER-SAFETY-REVIEWماہر2026-11-30
Experts in ontology, metrology, software verification and conformity assessment for the requirement register, the NHS-2 machine representation, the conformance test suites and the conformity assessment scheme CALL-CONFORMANCE-ONTOLOGYexpert, editor2027-01-31

Calls past their closing date are hidden automatically. Every call, proposal and route: open areas and calls.

Cells and CERTs

  1. Gather the cell

    Bring together three to seven people with complementary skills: data, field practice, policy and communication.

  2. Register it

    Write to the team with the cell's name, members, area and region. You receive a reference and a contact. Write to the Academy

  3. Choose a charter

    Pick one focus: research and evidence, data and maps, standards review, or community preparedness.

  4. Learn together

    Follow shared study paths at Risk Academy, with practice and a record of every course completed. Hazards and resilience

  5. Lead and report

    Hold regular sessions. Publish what the cell learns through Reports. Bring findings to the working groups. Working groups

  • Specialise as a Community Emergency Response Team (CERT)

    A CERT prepares its neighbourhood for the hazards it faces. It builds a local hazard profile from this site's map. It trains in household and street preparedness. It learns first aid and light search through recognised local providers, and it runs drills. A CERT works alongside the local emergency services and follows their direction. It never replaces them and is not an emergency service.

انڈیکس

What a national team or a local community can do on The Global Risks Index (NXS-GRX) today.

  • NXS-GRX

    Your country's profile

    Open risk profiles for 195 countries, with sources and licences on every value. Open the Index

  • NXS-GRX

    Correct or reply

    Correct a value or reply for your country. The change is logged with its effect. Open the Index

  • NXS-GRX

    Offer data

    Offer a dataset. It enters through the licence gate, with your credit. Open the Index

  • NXS-GRX

    Use the open data

    Layers, profiles and tiles through the open API. Open the Index

09References

Core references and where to begin

Start with Murphy, then Gneiting, then Brier.

  1. Gneiting, T., Balabdaoui, F. and Raftery, A. E., Probabilistic forecasts, calibration and sharpness, Journal of the Royal Statistical Society Series B (2007). DOI
  2. Gneiting, T. and Raftery, A. E., Strictly proper scoring rules, prediction, and estimation, Journal of the American Statistical Association (2007). DOI
  3. Brier, G. W., Verification of forecasts expressed in terms of probability, Monthly Weather Review (1950). DOI
  4. Murphy, A. H., What is a good forecast? An essay on the nature of goodness in weather forecasting, Weather and Forecasting (1993). DOI
  5. Jolliffe, I. T. and Stephenson, D. B. (eds), Forecast Verification: A Practitioner's Guide in Atmospheric Science, second edition, Wiley (2012). DOI
  6. Wilks, D. S., Statistical Methods in the Atmospheric Sciences, fourth edition, Elsevier (2019). DOI
  7. Hersbach, H., Decomposition of the continuous ranked probability score for ensemble prediction systems, Weather and Forecasting (2000). DOI
  8. Richardson, D. S., Skill and relative economic value of the ECMWF ensemble prediction system, Quarterly Journal of the Royal Meteorological Society (2000). DOI
  9. Raftery, A. E., Gneiting, T., Balabdaoui, F. and Polakowski, M., Using Bayesian model averaging to calibrate forecast ensembles, Monthly Weather Review (2005). DOI
  10. Lam, R. et al., Learning skillful medium-range global weather forecasting, Science (2023). DOI

10سوالات

سوالات، جوابات اور کلیدی اصطلاحات

کیا یہاں تعلیم مفت ہے؟

جی ہاں۔ Risk Academy، جس کی قیادت Nexus Consortium کرتا ہے، کا ہر کورس اور تعلیمی مواد مفت ہے، اپنی رفتار سے یا کسی گروپ کے ساتھ۔ عام شرکت کی کوئی فیس نہیں؛ معاوضے والا کام، انٹرن شپس اور فیلوشپس الگ انتظامات ہیں۔

کورس مکمل کرنے سے مجھے کیا ملتا ہے؟

A completion record: the course, its version, the date and what was assessed, private by default and shared by you. It stacks into discipline, department and programme achievements. A record states what was studied and assessed and does not certify competence; it is not a licence, a professional title or an employment status.

ڈیٹا اور مقامات کہاں سے آتے ہیں؟

Every faculty teaches on a twin of a real place built from open data; each layer carries its provider, licence and date, and every result on the Academy states its source. Maps and figures are indicators for teaching, never an assessment of a place.

How do I take part beyond the courses in probabilistic forecasting and verification?

Form a competence cell of three to seven people, study a shared pathway, take a supervised assignment in the National Working Groups, or bring a cohort from your institution. For organisations.

کلیدی اصطلاحات

Proper scoring rule
A score that is optimised in expectation only when the forecaster reports their true probability, such as the Brier score or CRPS.
Calibration
Agreement between forecast probabilities and observed frequencies: events forecast at 30 per cent happen about that often.
Sharpness
How concentrated a forecast distribution is, judged only once calibration holds.
Continuous ranked probability score
A proper score for a forecast distribution of a continuous quantity, in the units of the quantity.

رپورٹیںتعاون کریں

اس صفحے سے شواہد فراہم کریں۔

Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

Risk Academy

Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.

امکانی پیش گوئی اور اس کی توثیق

Prepare with a related course

Open courses in connected disciplines teach the methods this discipline will build on.

کورس کیٹلاگ دیکھیںBack to Department of Quantitative Risk Analysis