Probabilistic forecasting and verification: foundations and practice
- Level
- L2 Intermediate
- Study time
- about 24 hours (planning estimate)
- অবস্থা
- Planned
- Prerequisites
- An L1 course in the area, or an entry review
Hybrid physical and machine-learning ensembles change how forecast uncertainty is produced and must be validated.
European Centre for Medium-Range Weather Forecasts, ECMWF's ensemble AI forecasts become operational,
Learning outcomes
- Define proper scoring rule and the other key terms of the discipline and use them correctly in a written case
- Explain probabilistic and ensemble forecasts and proper scoring rules (Brier, CRPS, logarithmic) with a worked example and its source
- Apply the discipline's method to a documented case and explain each result with its source
- Analyse a named case against calibration, sharpness and reliability diagrams, stating evidence, date and limits
Modules
-
Probabilistic and ensemble forecasts
- Probabilistic and ensemble forecasts
- Proper scoring rules (Brier, CRPS, logarithmic)
- Calibration, sharpness and reliability diagrams
-
Discrimination, ROC curves and hit and false alarm rates
- Discrimination, ROC curves and hit and false alarm rates
- Verification of categorical warnings and triggers
- Forecast value and the cost-loss model
-
Post-processing and statistical calibration of ensembles
- Post-processing and statistical calibration of ensembles
- Machine-learning weather and hazard forecasts and their verification
- Forecast combination and model averaging
-
Lab: the discipline's method
- A probability forecast becomes a warning at a threshold; verified against outcomes it has a hit rate and a false alarm rate; for a user who pays a cost to protect against a loss, those two rates and the climatological frequency give the forecast's economic value, which is positive only for users whose cost-loss ratio lies within a band.
- Work the method through a documented case and explain each result
- Read the method's sources and stated values
Labs with Academy tools
পদ্ধতি
- From a probability forecast to its value for a user: hits, false alarms and the cost-loss ratio that decidesThe discipline's method on the discipline page, with its cited sources, worked through a documented case
উপকরণ
- Distribution fitterA faculty instrument on the faculty page, run on the faculty's digital twin data
- Return periodA faculty instrument on the faculty page, run on the faculty's digital twin data
Twin builder
- Digital twin builderBuild the faculty's place in 3D, switch layers and read the credits and dates of each
Map layer
- Earthquakes M4.5+, past week (USGS)An open map layer the Academy's licence register admits, read in the map tools
- Natural events (NASA EONET)An open map layer the Academy's licence register admits, read in the map tools
- Population density, 2020 (GPW v4.11)An open map layer the Academy's licence register admits, read in the map tools
Foundry room
- Forecast Verification room, NXS-EWSSkill, reliability and value of probabilistic forecasts by lead time
- Benchmarks room, NXS-LABBenchmarks that score forecast methods on the same events
Assessment
- 40%
Lab record
Work the discipline's method through a documented case and explain each result with its source and date
- 40%
Case memo
Apply the discipline's methods to a named place or organisation, stating evidence and limits
- 20%
Key terms check
Short answers on the key terms, marked against the definitions
Delivery
- Self-paced online, with the labs in the browser
- Cohort study over six to eight weeks, with a weekly lab review
- Co-delivered with a partner institution, inside its own courses
Partner on this course
- Co-teach a module and bring a current case from your work
- Host a cohort for your learners, with your own schedule and review
- Contribute a lab: a dataset your institution publishes openly, or a model
- Review the course content and its references before it opens
- Translate the course and adapt its cases to your region
How institutions and experts take part
In the Academy LMS
- Course on the learning platform
- Intermediate, 4 sections
- Cohort group
- A community group for each cohort of learners
- Achievement
- Recorded on completion of the assessed work
References 4
- European Centre for Medium-Range Weather Forecasts, ECMWF's ensemble AI forecasts become operational,
- Richardson, Skill and relative economic value of the ECMWF ensemble prediction system, Quarterly Journal of the Royal Meteorological Society, 2000
- Gneiting, Balabdaoui and Raftery, Probabilistic forecasts, calibration and sharpness, Journal of the Royal Statistical Society B, 2007
- Murphy, What is a good forecast? An essay on the nature of goodness in weather forecasting, Weather and Forecasting, 1993