Model risk and validation: foundations and practice
- Level
- L2 Intermediate
- Study time
- about 24 hours (planning estimate)
- অবস্থা
- In authoring
- Prerequisites
- An L1 course in the area, or an entry review
It gives model risk functions an international reference for extending validation and monitoring to AI systems.
Learning outcomes
- Define model risk and the other key terms of the discipline and use them correctly in a written case
- Explain model risk governance and model validation with a worked example and its source
- Apply the discipline's method to a documented case and explain each result with its source
- Analyse a named case against model monitoring and drift, stating evidence, date and limits
Modules
-
Model risk governance
- Model risk governance
- Model validation
- Model monitoring and drift
-
Explainability and fairness
- Explainability and fairness
- Third-party models
- Generative AI in finance
-
Human-in-the-loop controls
- Human-in-the-loop controls
- Model inventories
-
Lab: the discipline's method
- A model that claims a confidence level makes a testable promise about how often outcomes will exceed its figure; counting those exceptions over a test window and comparing the count with the expected distribution gives the verdict, and the verdict feeds the model's tier and its permitted use.
- Work the method through a documented case and explain each result
- Read the method's sources and stated values
Labs with Academy tools
পদ্ধতি
- From a model's claim to a back-test verdict: expected exceptions, observed exceptions and the zone the count falls inThe discipline's method on the discipline page, with its cited sources, worked through a documented case
উপকরণ
- Distribution fitterA faculty instrument on the faculty page, run on the faculty's digital twin data
- Return periodA faculty instrument on the faculty page, run on the faculty's digital twin data
Twin builder
- Digital twin builderBuild the faculty's place in 3D, switch layers and read the credits and dates of each
Map layer
- Natural events (NASA EONET)An open map layer the Academy's licence register admits, read in the map tools
- Earthquakes M4.5+, past week (USGS)An open map layer the Academy's licence register admits, read in the map tools
- Built-up and settlement extent, 2010 (HBASE, Landsat)An open map layer the Academy's licence register admits, read in the map tools
Foundry room
- Competence room, NXS-NXQWhether a model is fit for its task and operating envelope
- Benchmarks room, NXS-LABComparison of models under transparent test conditions
Nexus Standards
Teaches
Assessment
- 40%
Lab record
Work the discipline's method through a documented case and explain each result with its source and date
- 40%
Case memo
Apply the discipline's methods to a named place or organisation, stating evidence and limits
- 20%
Key terms check
Short answers on the key terms, marked against the definitions
Delivery
- Self-paced online, with the labs in the browser
- Cohort study over six to eight weeks, with a weekly lab review
- Co-delivered with a partner institution, inside its own courses
Partner on this course
- Co-teach a module and bring a current case from your work
- Host a cohort for your learners, with your own schedule and review
- Contribute a lab: a dataset your institution publishes openly, or a model
- Review the course content and its references before it opens
- Translate the course and adapt its cases to your region
How institutions and experts take part
In the Academy LMS
- Course on the learning platform
- Intermediate, 4 sections
- Cohort group
- A community group for each cohort of learners
- Achievement
- Recorded on completion of the assessed work
References 4
- Financial Stability Board, FSB finalises sound practices for the responsible adoption of artificial intelligence (AI),
- Board of Governors of the Federal Reserve System, SR 11-7: Guidance on Model Risk Management, 2011
- Basel Committee on Banking Supervision, Supervisory framework for the use of backtesting in conjunction with the internal models approach, 1996
- Bank of England PRA, SS1/23 Model risk management principles for banks, 2023