Risk Academy, led by the Nexus Consortium · Risk Science and Methods · Department of Quantitative Risk Analysis
Forecasts as distributions, judged by proper scoring rules
நிகழ்தகவு முன்னறிவிப்பும் சரிபார்ப்பும்
A risk number is a forecast, and a forecast is only as good as its verification.
Map probabilistic forecasting and verification on a real placeSee the courses
Illustrative: an ensemble of trajectories, its quantile fan and the outcome as it is observed
01வரம்பு
What the discipline covers
The subtopics its courses and labs teach.
- Department
- Department of Quantitative Risk Analysis
- பாடநெறிகள்
- In preparation
- நிலைகள்
- In development
- ஆய்வகங்கள்
- In development
- Probabilistic and ensemble forecasts
- Proper scoring rules (Brier, CRPS, logarithmic)
- Calibration, sharpness and reliability diagrams
- Discrimination, ROC curves and hit and false alarm rates
- Verification of categorical warnings and triggers
- Forecast value and the cost-loss model
- Post-processing and statistical calibration of ensembles
- Machine-learning weather and hazard forecasts and their verification
- Forecast combination and model averaging
02வரைபடங்கள்
Build a twin of any place
Search any place or open the world; the layers and generators follow probabilistic forecasting and verification on the planet map, the globe and in 3D.
- Terrain 24 to 2,805 m
- 3 reference earthquakes, M6.2 to M6.6
- Earthquakes M4.5 and above, past week: read live when the 3D twin opens
- USGS ShakeMap intensity contours: read live when the 3D twin opens
Drag to move, right drag or Shift with a drag to turn and tilt, Ctrl with the wheel to zoom. Keyboard: arrows move, Shift with arrows turn and tilt, + and – zoom, 0 resets, V steps through the views, Escape clears.
Flat plan of the same data: relief, the three reference epicentres with magnitude and depth, and the numbers below; the sampler's figure shows one precomputed field with its parameters. The 3D twin opens on devices that can draw it, once its data and code have been checked.
The boundaries, names and designations shown do not imply any judgement on the legal status of any territory, or any endorsement or acceptance of such boundaries. Countries: Natural Earth (public domain).
03The subject
The subject, methods and measures and debates and bodies
The discipline
The field and its scope
For forecasters, early warning analysts, modellers and the people who act on forecasts, who need to issue risk as a calibrated distribution and to judge forecasts fairly against what happened.
Probabilistic forecasting and verification study how a future quantity or event is forecast as a probability distribution rather than a single value, and how such forecasts are scored against outcomes so that better forecasts get better scores. The field was built in meteorology, where ensembles of model runs replaced deterministic forecasts in the 1990s, and its methods now serve hydrology, epidemiology, energy, finance, humanitarian early warning and the anticipatory action triggers that release funds before a shock. The principle set out by Gneiting, Balabdaoui and Raftery (2007) is to maximise sharpness subject to calibration: forecasts should be as decisive as possible while the stated probabilities match observed frequencies.
The discipline is organised around proper scoring rules (Brier, logarithmic, continuous ranked probability score), the decomposition of scores into reliability, resolution and uncertainty, diagrams for calibration and discrimination, contingency-table statistics for warnings, statistical post-processing of ensembles, and the economic value of a forecast under a cost-loss decision. Machine learning weather models have made verification the arbiter of a live contest between physical and learned forecasting systems, and the field's open questions are about verifying rare extremes, spatial forecasts and forecasts that change what they predict.
Measures
Quantities and units
| Quantity | Unit or scale | What to know |
|---|---|---|
| Brier score and Brier skill score | Dimensionless, 0 (perfect) to 1 for the score; skill relative to a reference forecast | The mean squared error of a probability forecast for a binary event, decomposable into reliability, resolution and uncertainty. For rare events the score is dominated by the climatological term, so skill scores against climatology are reported and can be unstable with few events. |
| Continuous ranked probability score (CRPS) | Same unit as the forecast variable | The integral of the squared difference between forecast and observed cumulative distributions; it reduces to the absolute error for a point forecast. Sampling uncertainty is reported by bootstrapping over forecast cases. |
| Reliability (calibration) and sharpness | Observed frequency against forecast probability, dimensionless; width of forecast intervals in the variable's unit | Reliability diagrams show whether a 30 per cent forecast verifies about 30 per cent of the time; sharpness is how far the forecasts are from climatology. Both can be read only with enough cases per probability bin. |
| Hit rate, false alarm ratio and the relative operating characteristic | Proportions, dimensionless; area under the ROC curve from 0.5 to 1 | Contingency-table statistics for warnings at a threshold. The ROC measures discrimination and is insensitive to calibration; a well-discriminating system can still be badly calibrated. |
| Relative economic value | Fraction of the value of a perfect forecast, by cost-loss ratio | The value of a forecast to a user who pays a cost to protect against a loss. It depends on the user's cost-loss ratio, so one forecast can be worth much to one user and nothing to another. |
| Lead time and spread-skill relation | Hours or days; ensemble spread against root mean square error | How far ahead skill persists and whether ensemble spread tracks error. Under-dispersion at short lead times is common and is corrected by post-processing. |
முறைகள்
Methods and their limits
Each method is listed with what it assumes and where it fails.
| முறை | Assumes | Fails when |
|---|---|---|
| Ensemble forecasting with perturbed initial conditions and model physics | That the spread of an ensemble represents forecast uncertainty, that members are exchangeable, and that the ensemble size resolves the probabilities of interest. | When the ensemble is under-dispersive because model error is not sampled, when extremes need probabilities below the resolution of the member count, or when all members share a systematic bias. |
| Proper scoring rules | That the score is strictly proper so the forecaster's best strategy is to report their true belief, and that enough independent cases are available for the score to discriminate between systems. | When forecasts are compared across different case sets, when a single score aggregates over regimes where skill differs, or when an improper measure (such as a threshold-based accuracy) is used and rewards hedging. |
| Statistical post-processing (ensemble model output statistics, Bayesian model averaging, quantile mapping) | That the relation between raw ensemble output and observations is stable over the training period and the forecast period, and that the training set covers the conditions of interest. | After a model upgrade, under non-stationary climate, or for extremes outside the training sample where the calibration must be extrapolated. |
| Warning verification with contingency tables | That events and warnings can be matched in space and time, that the event definition is fixed, and that non-events are countable. | When the user's response to a warning prevents the event (so the warning looks like a false alarm), when the observation network misses events, or when the non-event count is arbitrary and inflates the apparent accuracy. |
| Cost-loss analysis of forecast value | That the user faces a known protection cost and a known loss, that decisions are binary, and that the forecast probability is the only information. | When losses are uncertain or non-monetary, when actions are graded rather than binary, or when the user already has other information so that the forecast's marginal value is smaller than the model suggests. |
| Forecast combination and multi-model ensembles | That different models have partly independent errors and that weights can be estimated from past performance. | When the models share data assimilation or parameterisations, when past performance does not predict future performance, or when weighting is fitted to a short period and overfits. |
Debates
What the field debates
- Have machine learning weather models surpassed physical models, and how should the claim be verified?
- Papers such as Lam and co-authors (2023) show lower root mean square error and CRPS than operational physical models on most variables and lead times on reanalysis-based benchmarks. Operational centres reply that reanalysis targets flatter smooth forecasts, that extremes and physical consistency are verified less well, and that verification against observations and in the operational cycle is the fair test; both sides now use shared benchmarks such as WeatherBench 2.
- Should forecasts be judged by scores or by the decisions they support?
- The scoring-rule tradition holds that proper scores are the only incentive-compatible measure and that value follows from calibration and resolution. The decision-analytic view, from Murphy's framework to the anticipatory action community, holds that a forecast with a modest score can be worth more to a user with a particular cost-loss ratio than a better-scoring one, so verification should report value curves and trigger performance as well.
- How should rare extremes be verified when there are few cases?
- One position favours scores restricted to the tail (threshold-weighted CRPS, extreme dependency scores) and pooling across space and time to gain cases, accepting dependence between them. The other holds that any score over a handful of events is dominated by sampling noise, that pooling hides regional failures, and that process-based evaluation and case studies are the honest complement.
Bodies and data
Bodies that set practice, and the data they publish
Bodies and standards are named so learners can find and apply their published work; the Academy teaches their use and links to the publisher, and no affiliation is implied.
| Body | Role in the field |
|---|---|
| World Meteorological Organization (WMO) and its Joint Working Group on Forecast Verification Research | Sets the standard verification practices for operational centres and publishes guidelines on verification of forecasts and warnings. |
| European Centre for Medium-Range Weather Forecasts (ECMWF) | Runs the reference global ensemble, publishes verification scores and reanalyses (ERA5), and makes open forecast data available. |
| National Oceanic and Atmospheric Administration (NOAA) and the National Weather Service | Operates the Global Ensemble Forecast System and the United States verification programmes for weather and flood warnings. |
| Royal Statistical Society and the American Statistical Association | Professional bodies for the statistical theory of scoring rules, calibration and forecast evaluation. |
| Anticipation Hub and the Red Cross Red Crescent Climate Centre | Coordinate practice on forecast-based triggers for anticipatory action and the verification of trigger performance. |
| Dataset | Steward |
|---|---|
| ECMWF open forecast data | European Centre for Medium-Range Weather Forecasts |
| ERA5 reanalysis | Copernicus Climate Change Service, implemented by ECMWF |
| WeatherBench 2 benchmark | Google Research |
| TIGGE global ensemble archive | ECMWF on behalf of the WMO THORPEX programme |
04முறை
From a probability forecast to its value for a user: hits, false alarms and the cost-loss ratio that decides
A probability forecast becomes a warning at a threshold; verified against outcomes it has a hit rate and a false alarm rate; for a user who pays a cost to protect against a loss, those two rates and the climatological frequency give the forecast's economic value, which is positive only for users whose cost-loss ratio lies within a band.
Sources: Richardson, Skill and relative economic value of the ECMWF ensemble prediction system, Quarterly Journal of the Royal Meteorological Society (2000); Gneiting, Balabdaoui and Raftery, Probabilistic forecasts, calibration and sharpness, Journal of the Royal Statistical Society B (2007); Murphy, What is a good forecast? An essay on the nature of goodness in weather forecasting, Weather and Forecasting (1993); Jolliffe and Stephenson (eds), Forecast Verification: A Practitioner's Guide in Atmospheric Science (2012); Brier, Verification of forecasts expressed in terms of probability, Monthly Weather Review (1950).
05At a glance
At a glance, hub, what is changing now and courses
- Build a twin of any placeTwin builderWorked example: the central Apennines: L'Aquila, Amatrice and Norcia · 4 generators · 7 layers · 4 instruments
3D வரைபடம்
இடர் அறிவியல் மற்றும் முறைகள் புலம்
265 nodes1,691 relations9 views
06கட்டுப்பாட்டகம்
How Probabilistic forecasting and verification is taught, as a process you can run
Press Play to run this process from this page's own data.
நிகழ்தகவு முன்னறிவிப்பும் சரிபார்ப்பும்
How Probabilistic forecasting and verification is taught
5 subjects in scope, 3 key terms, courses in development, 1 lab, in the Quantitative Risk Analysis department of Risk Science and Methods: from the scope to the method and its terms, the courses that teach it, the labs on the twin, the assessment and the record.
-
01 Scope
- 01
Probabilistic and…
Start · the discipline
Probabilistic and ensemble forecasts
- 02
Proper scoring rules…
Start · the discipline
Proper scoring rules (Brier, CRPS, logarithmic)
- 03
Calibration, sharpness…
Start · the discipline
Calibration, sharpness and reliability diagrams
- 04
Discrimination, ROC…
Start · the discipline
Discrimination, ROC curves and hit and false alarm rates
- 05
Verification of…
Start · the discipline
Verification of categorical warnings and triggers
- 01
-
02 Method and terms
- 06
From a probability…
Model · the learner
A probability forecast becomes a warning at a threshold; verified against outcomes it has a hit rate and a false alarm rate; for a user…
- 07
Proper scoring rule
Record · the discipline
A score that is optimised in expectation only when the forecaster reports their true probability, such as the Brier score or CRPS.
- 09
Calibration
Record · the discipline
Agreement between forecast probabilities and observed frequencies: events forecast at 30 per cent happen about that often.
- 11
Sharpness
Record · the discipline
How concentrated a forecast distribution is, judged only once calibration holds.
- 06
-
04 Labs
- 08
இரட்டை மாதிரி
Exercise · the learner, on the faculty twin
The method runs on the faculty twin and the admitted map layers.
- 08
-
05 Assess
- 10
Module checks
Gate · the course
Each module ends with a short check against its outcomes.
- 12
Assessed work
Decision · the learner
The assessed work that closes a course: a lab record, a case memo or an assignment, reviewed against the rubric.
- 10
-
06 Record
- 13
நிறைவுப் பதிவு
Outcome · the learner
The record stacks into the discipline achievement.
- 14
Discipline achievement
Record · the Academy
The discipline achievement follows the discipline rule, read from the records.
- 13
Probabilistic forecasting and verification · Quantitative Risk Analysis An outline of how the work is designed to run; it does not report the status of any decision or record.
07கற்றல்
Achievements, where this is practised, practised in and across the ecosystem
Achievements
Achievements in this discipline
Course achievements
One for each course with this as its primary discipline, earned when you pass the course's assessed activities.
Department: Quantitative Risk Analysis
Earn two discipline achievements in the Department of Quantitative Risk Analysis. In development.
An achievement is a record of the learning completed and assessed here. How recognition works.
Where this is practised
Where the subject is practised
A field platform page on this subject is in development.
Practised in
The two Foundry systems where it is practised
- FoundryNXS-EWS systemForecasts and warnings are judged by how they verify
- NXS-LAB systemCompeting forecast methods are compared under transparent tests
Foundry system pages describe what each system and its rooms are designed to do; the courses teach the methods these systems apply.
Across the ecosystem
Across the ecosystem
The science, platforms and standards this discipline draws on and contributes to across the ecosystem.
Foundations in the STEM Commons
- The Global Centre for Risk and Innovation (GCRI Canada)Forecasting and time-series analysisForecasting and time-series analysis predict future values of variables from past observations and explanatory information.
- Probability and statisticsProbability and statistics provide the mathematical framework for describing uncertainty and learning from data.
- Uncertainty quantification and model validationUncertainty quantification and model validation characterise how uncertainty in inputs, parameters and model structure affects predictions.
Pages of the ecosystem, cited so learners can follow the field into research, platforms, standards and national practice.
08பங்கேற்கவும்
Take part in Risk Science and Methods
நீங்கள் யார் என்பதிலிருந்து தொடங்குங்கள்.
Routes
வழித்தடம்
தேசிய நிறுவனங்கள்
Ministries, statistics offices and agencies: a national desk and chapter, and a seat in the working groups. National chapters
வழித்தடம்
Sectors and professions
Operators, regulators, banks, insurers and professional bodies whose work touches risk science and methods. For organisations
வழித்தடம்
கற்றல்
Courses and practice, with a record of every course completed. Free to study. Open learning
வழித்தடம்
தரநிலைகள்
Draft, review and adopt the methods and data standards with the working groups. Working groups
வழித்தடம்
Communities and research
Universities, civil society and individual experts: projects, data and review. Write to the Academy
Open calls
| Open call | Seeks | Closes |
|---|---|---|
| Founding call for experts to the working groups CALL-FOUNDING-EXPERTS | expert, editor, chair | 2026-12-15 |
| Open nomination for the three independent expert seats of the Global Standards Council CALL-GSC-IND | மன்ற உறுப்பினர் | 2026-12-01 |
| Nominations of interim regional members (applies only if the transitional provision is decided) CALL-GSC-REG | மன்ற உறுப்பினர் | 2026-12-15 |
| Review of the wave-1 risk core outlines (C1, C2, C3, C4, C8, C10, C11) CALL-REVIEW-WAVE1 | contributor | 2027-01-31 |
| இடர் அறிவியலின் முதன்மைத் தரநிலைகளுக்கான நிபுணர்கள்: நிச்சயமின்மை மற்றும் சான்றுத் தரப்படுத்தல், அளவுகோல்கள், கட்டமைக்கப்பட்ட நிபுணர் தீர்ப்பு, சகிப்புத்தன்மை, உச்சநிலைகள் மற்றும் கூட்டு நிகழ்வுகள், பேரழிவு மற்றும் இருத்தலியல் இடர், நெறிமுறைகள் மற்றும் சமத்துவம், அலகுகள் மற்றும் அளவீடு (NRS-C17 முதல் NRS-C24 வரை) CALL-CORE-SCIENCE | expert, editor, chair | 2027-01-31 |
| Independent reviewers for the families of the hazard atlas, three from at least two regions for each family (NHS2-R19), priority families first CALL-ATLAS-REVIEW | நிபுணர் | 2027-03-31 |
| இடைக்கால முன்னோடிப் பாதுகாப்பு மறுஆய்வுக் குழுவின் உறுப்பினர்கள் மற்றும் சிவப்புக் குழு மறுஆய்வாளர்கள் (உயிரியல் பாதுகாப்பு, AI பாதுகாப்பு, முக்கிய உள்கட்டமைப்புப் பாதுகாப்பு); விண்ணப்பங்கள் இரகசியமாகக் கையாளப்படும் CALL-FRONTIER-SAFETY-REVIEW | நிபுணர் | 2026-11-30 |
| தேவைப் பதிவேடு, NHS-2 இயந்திரப் பிரதிநிதித்துவம், இணக்கச் சோதனைத் தொகுப்புகள் மற்றும் இணக்க மதிப்பீட்டுத் திட்டத்திற்கான கருத்தியல், அளவியல், மென்பொருள் சரிபார்ப்பு மற்றும் இணக்க மதிப்பீட்டு நிபுணர்கள் CALL-CONFORMANCE-ONTOLOGY | expert, editor | 2027-01-31 |
Calls past their closing date are hidden automatically. Every call, proposal and route: open areas and calls.
Cells and CERTs
Gather the cell
Bring together three to seven people with complementary skills: data, field practice, policy and communication.
Register it
Write to the team with the cell's name, members, area and region. You receive a reference and a contact. Write to the Academy
Choose a charter
Pick one focus: research and evidence, data and maps, standards review, or community preparedness.
Learn together
Follow shared study paths at Risk Academy, with practice and a record of every course completed. Hazards and resilience
Lead and report
Hold regular sessions. Publish what the cell learns through Reports. Bring findings to the working groups. Working groups
Specialise as a Community Emergency Response Team (CERT)
A CERT prepares its neighbourhood for the hazards it faces. It builds a local hazard profile from this site's map. It trains in household and street preparedness. It learns first aid and light search through recognised local providers, and it runs drills. A CERT works alongside the local emergency services and follows their direction. It never replaces them and is not an emergency service.
குறியீடு
What a national team or a local community can do on The Global Risks Index (NXS-GRX) today.
NXS-GRX
Your country's profile
Open risk profiles for 195 countries, with sources and licences on every value. Open the Index
NXS-GRX
Correct or reply
Correct a value or reply for your country. The change is logged with its effect. Open the Index
NXS-GRX
Offer data
Offer a dataset. It enters through the licence gate, with your credit. Open the Index
NXS-GRX
Use the open data
Layers, profiles and tiles through the open API. Open the Index
09References
Core references and where to begin
Start with Murphy, then Gneiting, then Brier.
- Gneiting, T., Balabdaoui, F. and Raftery, A. E., Probabilistic forecasts, calibration and sharpness, Journal of the Royal Statistical Society Series B (2007). DOI
- Gneiting, T. and Raftery, A. E., Strictly proper scoring rules, prediction, and estimation, Journal of the American Statistical Association (2007). DOI
- Brier, G. W., Verification of forecasts expressed in terms of probability, Monthly Weather Review (1950). DOI
- Murphy, A. H., What is a good forecast? An essay on the nature of goodness in weather forecasting, Weather and Forecasting (1993). DOI
- Jolliffe, I. T. and Stephenson, D. B. (eds), Forecast Verification: A Practitioner's Guide in Atmospheric Science, second edition, Wiley (2012). DOI
- Wilks, D. S., Statistical Methods in the Atmospheric Sciences, fourth edition, Elsevier (2019). DOI
- Hersbach, H., Decomposition of the continuous ranked probability score for ensemble prediction systems, Weather and Forecasting (2000). DOI
- Richardson, D. S., Skill and relative economic value of the ECMWF ensemble prediction system, Quarterly Journal of the Royal Meteorological Society (2000). DOI
- Raftery, A. E., Gneiting, T., Balabdaoui, F. and Polakowski, M., Using Bayesian model averaging to calibrate forecast ensembles, Monthly Weather Review (2005). DOI
- Lam, R. et al., Learning skillful medium-range global weather forecasting, Science (2023). DOI
10கேள்விகள்
கேள்விகள், பதில்கள் மற்றும் முக்கியச் சொற்கள்
இங்கே படிப்பு இலவசமா?
ஆம். Nexus Consortium வழிநடத்தும் Risk Academy-இன் ஒவ்வொரு படிப்பையும் கற்றல் பொருளையும் இலவசமாகப் படிக்கலாம், சுய வேகத்திலோ ஒரு குழுவுடனோ. சாதாரணப் பங்கேற்புக்குக் கட்டணம் இல்லை; ஊதியம் பெறும் பணி, பயிற்சிப் பணிகள் மற்றும் ஆய்வு உதவித்தொகைகள் தனி ஏற்பாடுகள்.
ஒரு படிப்பை முடிப்பது எனக்கு என்ன தருகிறது?
A completion record: the course, its version, the date and what was assessed, private by default and shared by you. It stacks into discipline, department and programme achievements. A record states what was studied and assessed and does not certify competence; it is not a licence, a professional title or an employment status.
தரவும் இடங்களும் எங்கிருந்து வருகின்றன?
Every faculty teaches on a twin of a real place built from open data; each layer carries its provider, licence and date, and every result on the Academy states its source. Maps and figures are indicators for teaching, never an assessment of a place.
How do I take part beyond the courses in probabilistic forecasting and verification?
Form a competence cell of three to seven people, study a shared pathway, take a supervised assignment in the National Working Groups, or bring a cohort from your institution. For organisations.
முக்கியச் சொற்கள்
- Proper scoring rule
- A score that is optimised in expectation only when the forecaster reports their true probability, such as the Brier score or CRPS.
- Calibration
- Agreement between forecast probabilities and observed frequencies: events forecast at 30 per cent happen about that often.
- Sharpness
- How concentrated a forecast distribution is, judged only once calibration holds.
- Continuous ranked probability score
- A proper score for a forecast distribution of a continuous quantity, in the units of the quantity.
அறிக்கைகள்பங்களிக்கவும்
இந்தப் பக்கத்திலிருந்து சான்றுகளைப் பங்களிக்கவும்.
Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
Risk Academy
- EmailGateway GW-ACADEMY Send it from this page Evidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen deposit Deposit it and submit to the community Publish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsRisk Academy See every way to contribute The contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.
நிகழ்தகவு முன்னறிவிப்பும் சரிபார்ப்பும்
Prepare with a related course
Open courses in connected disciplines teach the methods this discipline will build on.
படிப்புப் பட்டியலை உலாவுங்கள்Back to Department of Quantitative Risk Analysis