Skip to content

RA · Foundations

Learning model risk and validation: a guide to the field

A learning guide to model risk and validation: effective challenge, validation methods, supervisory texts, an order of study and how expertise is shown.

As of
17 November 2026
阅读
11 minutes
证据
10 sources, cited in order
级别
Introductory to intermediate
学习时长
15 minutes
NEXUS TWIN · WF-1 · SURFACESEED 887f5806RA-F04.DISCIPLINE.MODEL-RISK-VALIDATIONxyzSCHEMATIC · SURFACEST-01 · 0.607 0.544ST-02 · 0.522 0.310
WF-1 · surface Schematic wireframe twin (surface, catastrophe model). An illustration, not data.

What you will be able to do

  • Define a model and model risk.
  • Name the two roots of model risk.
  • Describe effective challenge and its three conditions.
  • List the components of the model lifecycle.
级别
Introductory to intermediate
学习时长
15 minutes
Before you start
The learning guide to the Faculty of Risk Science and Methods, or equivalent background
US guidance
4 April 2011
date of the Federal Reserve guidance on model risk management, SR 11-71
UK statement
17 May 2023
publication date of PRA Supervisory Statement SS1/232
加拿大
1 May 2027
effective date of OSFI Guideline E-23 on model risk management3

01What model risk and validation covers

What model risk and validation covers

Model risk and validation is the Academy discipline of model risk and validation, one of the disciplines of the Faculty of Risk Science and Methods. It covers how organisations find out whether a model is fit for its purpose, and how they manage the risk that it is not.

The Federal Reserve guidance on model risk management, issued on 4 April 2011, uses the term model to refer to a quantitative method, system or approach that applies statistical, economic, financial or mathematical theories, techniques and assumptions to process input data into quantitative estimates 1. The same guidance says that model risk can lead to financial loss, poor business and strategic decision making, or damage to a bank's reputation 1.

The discipline sits beside AI and machine learning for risk analysis, scenario analysis and stress testing and uncertainty and decision making. Its core question is simple to say and hard to answer: is this model good enough for this use?

A short example shows why the question needs care. A model built to rank customers for a marketing campaign may be sound for that task and unsafe for setting a credit limit, because the cost of an error is different and the data may not cover the same people. The model has not changed. The use has. The Federal Reserve guidance makes the same point when it says that a fundamentally sound model may exhibit high model risk if it is misapplied or misused 1. Validation therefore always starts from the stated use.

02Why validation matters

Why validation matters

Models now sit inside many decisions. The Office of the Superintendent of Financial Institutions (OSFI) says that the financial services industry is experiencing a rapid rise in digitalisation and model applications, amplified by the surge in AI and machine learning models. It adds that institutions increasingly rely on models to support or drive decision-making, including in areas that traditionally did not use them 3.

Supervisors treat the subject as a risk discipline in its own right. The PRA states that its SS1/23, published on 17 May 2023, sets out its expectations for banks' management of model risk, and that the desired outcome is a strategic approach to model risk management as a risk discipline in its own right 2. In the euro area, the European Central Bank explains that banks may use internal models to estimate their own funds requirements provided they have prior authorisation from the competent authority 4.

The limits of validation are part of the subject. Oreskes, Shrader-Frechette and Belitz argued in Science in 1994 that verification and validation of numerical models of natural systems is impossible, because natural systems are never closed and model results are always nonunique. They argue that models can be confirmed only partially, and that their predictive value is always open to question 5. A learner should treat validation as building justified confidence, not proving a model true.

03The model lifecycle and its checks

The model lifecycle and its checks

A model has a life, and risk arises at each stage. OSFI defines the model lifecycle as the components that define the life of a model, covering all steps for designing, operating and maintaining it until it is decommissioned. The components are model design, including rationale, data and development, then model review, deployment, monitoring and decommission, and they are not necessarily sequential 3.

The Federal Reserve guidance names ways to check a model. These include ongoing monitoring, with process verification and benchmarking, and outcomes analysis, including back-testing 1. It says a guiding principle is effective challenge of models, which depends on a combination of incentives, competence and influence. Incentives are stronger when challenge is more separate from model development 1. Figure 2 sets out the lifecycle.

Governance sits around the lifecycle. OSFI describes a model risk management framework as a set of policies and procedures that reflects the institution's risk appetite for model risk and defines governance requirements, including roles and responsibilities and processes to identify, assess, manage, monitor and report on model risk 3. The text applies to all federally regulated financial institutions in Canada, to the extent consistent with applicable requirements 3.

Fig. 02The model lifecycle

The model lifecycleFive stacked layers naming the components of a model lifecycle: model design, model review, model deployment, model monitoring and model decommission, with review highlighted as the point of independent challenge.01DesignRationale, data, development02审核Independent challenge03DeploymentPut into use04监测Ongoing checks05DecommissionRetire the model
The lifecycle components named in OSFI Guideline E-23 3. The components are not necessarily sequential.3Sources: OSFI Guideline E-23 (2027). Figure CC BY 4.0. As of 17 November 2026.
Figure 2 in words

Five layers stacked from top to bottom: model design, including rationale, data and development; model review, highlighted; model deployment; model monitoring; model decommission. OSFI notes that the components are not necessarily sequential and may vary with the type of model.

04A suggested learning order in four stages

A suggested learning order in four stages

The Academy describes study at four levels: introductory, intermediate, advanced and integrative. The four stages below follow the same arc for model risk and validation. They are a suggestion, and a learner can enter at any stage.

Stage one: what a model is. Start with the definition and the idea of model risk. Read the opening sections of the Federal Reserve guidance 1. Write a one-sentence definition of a model and of model risk in your own words, and list three models you meet in daily life. For each, say what decision it informs and what would happen if it were wrong. The exercise shows quickly that models matter in proportion to the decisions that rest on them.

Stage two: validation methods. Next, study how models are tested. Learn conceptual review, ongoing monitoring, benchmarking and outcomes analysis with back-testing 1. Learn data leakage, with the scikit-learn guidance on common pitfalls 6. Read Oreskes and colleagues on what validation can and cannot show 5.

Stage three: governance and lifecycle. Then study governance. Read the PRA SS1/23 2 and the OSFI Guideline E-23 3. Compare how each describes roles, the lifecycle and the inventory of models.

Stage four: AI and machine learning models. Finally, study models that learn from data. Read the NIST AI Risk Management Framework 7 and the European Commission summary of the AI Act 8. Read the argument for interpretable models in high-stakes uses 9.

The order is a suggestion. A practitioner may begin at stage three and return to stage one later. What matters is that each new idea has the ideas beneath it in place.

Fig. 01Four stages of study in model risk and validation

Four stages of study in model risk and validationA rising ladder of four steps: what a model is, validation methods, governance and lifecycle, and AI and machine learning models, each with the texts a learner meets at that step.01What a model isDefinitions, model risk02Validation methodsMonitoring, back-testing, limits03Governance and lifecycleSS1/23, E-2304AI and ML modelsNIST, EU AI Act
The suggested learning order of this guide, with the texts that anchor each stage 1,6,5,2,3,7,8,9. The order is a suggestion, not a required curriculum.1,6,5,2,3,7,8,9Sources: Federal Reserve; scikit-learn; Science; PRA; OSFI; NIST; European Commission. Figure CC BY 4.0. As of 17 November 2026.
Figure 1 in words

Four steps rise from left to right. Step one, what a model is: the Federal Reserve guidance definitions. Step two, validation methods: monitoring, outcomes analysis, leakage and the limits of validation. Step three, governance and lifecycle: PRA SS1/23 and OSFI Guideline E-23. Step four, highlighted, AI and machine learning models: the NIST framework, the EU AI Act and interpretable models.

Fig. 03The two roots of model risk

The two roots of model riskA four-stage cascade: a model with possible fundamental errors, use for an intended purpose, possible incorrect or inappropriate use, and a decision based on the output, showing the two roots of model risk.MODELMay contain errorsINTENDED USEDesign objectiveACTUAL USEMay be misusedDECISIONRests on the output
The two primary reasons model risk occurs 1.1Sources: Federal Reserve SR 11-7. Figure CC BY 4.0. As of 17 November 2026.
Figure 3 in words

Four stages run from left to right: model, intended use, actual use and decision. Model risk arises in two ways, according to the Federal Reserve guidance: the model may have fundamental errors and produce inaccurate outputs for its design objective, or it may be used incorrectly or outside its design. Both feed the decision.

05Six core ideas every learner should hold

Six core ideas every learner should hold

Six ideas carry most of the weight in this discipline. Each is simple to state and takes practice to use.

Model risk has two roots. Errors in the model, and wrong or inappropriate use of a sound one 1.

Challenge must be independent enough. Effective challenge depends on incentives, competence and influence 1.

Validation builds confidence, not proof. Oreskes and colleagues state that confirmation is inherently partial and that complete confirmation is logically precluded 5.

The whole lifecycle matters. Design, review, deployment, monitoring and decommission are all points of risk 3.

Models are an inventory and a governance matter. A framework defines roles, risk appetite and reporting 3.

AI models raise the stakes. Supervisors and standard setters address them explicitly 3 7 8.

06Canonical references and data to know

Canonical references and data to know

A learner should know a short list of primary texts. Read the originals first and treat summaries as an aid.

For supervisory expectations: the Federal Reserve guidance of 2011 1, PRA SS1/23 2 and OSFI Guideline E-23 3. For the European setting: the ECB page on internal models 4. For AI: the NIST AI RMF 7, the EU AI Act summary 8 and the FSB report on AI and machine learning in financial services 10. For method and its limits: Oreskes and colleagues 5, the scikit-learn guidance on pitfalls 6 and Rudin on interpretable models 9. These are documents, not datasets. Learners should record the edition and date of each text they use.

07Skills that make the knowledge usable

Skills that make the knowledge usable

Knowledge becomes useful through a small set of skills. The discipline treats these as its practical core.

Scoping a model. State its purpose, users and the decisions it informs. Reviewing design. Check assumptions, data and method against the purpose. Testing outcomes. Compare predictions with results, with back-testing where it applies. Benchmarking. Compare with a simpler or independent model. Monitoring. Set thresholds and triggers for review. Reporting. Tell users what the model can and cannot support, in plain terms.

08Open courses and pathways

Open courses and pathways

No open course on this subject was opened at its provider and confirmed as open to the public for this edition, so none is listed.

The supervisory texts are free to read at their issuers, as confirmed on 17 November 2026 1 2 3. The scikit-learn documentation is a practical guide to avoiding leakage when building models 6. A suggested order is the definitions, then one text on validation methods, then a governance text such as SS1/23 or E-23.

None of these is a course with assessment, and none is described here as leading to a qualification.

09How expertise is shown

How expertise is shown

Expertise in this field is shown by work that others can check. The Academy records learning, not credentials.

Finishing a course gives a completion record. It is not a certificate, an accreditation or a qualification, and it does not state that a person is competent in a profession. Achievements on the Academy recognise assessed learning only.

The Academy catalogue lists two courses for this discipline: Model risk and validation: foundations and practice, at the intermediate level, and Model risk and validation: frontier methods, at the advanced level. On 17 November 2026 the catalogue page shows the first as in authoring and the second as planned. Nothing is open for enrolment until it is published in the NXS Learning catalogue, so this guide does not point to an Academy course as a place to enrol.

Outside the Academy, the strongest evidence of expertise is a body of work: a validation report on a small model with scope, tests and findings, a short comparison of two supervisory texts, or a model inventory with owners and review dates. Learners are encouraged to keep such a record from their first course.

10Emerging areas to watch

Emerging areas to watch

Several areas are changing what a model risk professional needs to know. Each is a place to read on once the core is in place.

AI and machine learning. OSFI states that its guideline responds to the surge in AI and machine learning models 3, and the PRA page announces a model risk management roundtable on artificial intelligence and machine learning technologies 2. New effective dates. OSFI Guideline E-23 takes effect on 1 May 2027 3. Revision of frameworks. NIST says AI RMF 1.0 is being revised 7. Legal regimes. The EU AI Act sets risk-based rules for AI systems 8.

Related work across the Nexus network gives wider context. See models and scenarios research at the Global Centre for Risk and Innovation, AI and model risk in finance at the Global Risks Alliance, digital and AI governance at the Global Risks Forum and the proposed Nexus Standards on AI and models. These pages describe programmes and proposals. A link to them implies no endorsement of the sources this guide cites.

What this guide does not claim: it is suggested learning, not a curriculum that anyone must follow, and it is not professional, legal, medical or technical advice. It names programmes, courses and bodies as places to learn and implies no partnership with any of them. It reports figures as their issuers published them and draws no new estimates from them. Risk Academy does not certify competence, and a completion record is not a professional qualification. The Academy records no partners, cohorts or learner outcomes for this faculty, and this guide implies none. The Nexus Standards are proposed and in draft; nothing here treats them as adopted.

常见问题

问与答

Where should a complete beginner start in model risk and validation?

Start with stage one and the definitions in the Federal Reserve guidance, then compare them with the OSFI Guideline E-23 definition of model lifecycle.

What is effective challenge?

The Federal Reserve guidance names it as a guiding principle. It is critical review of a model by objective, informed parties who can identify limits and assumptions and cause changes.

Can a model ever be fully validated?

Not in the strict sense. Oreskes and colleagues argue that models of natural systems can be confirmed only partially, so validation builds justified confidence.

Do I need to program to follow this guide?

Not for stages one and three. Stages two and four are easier with a scripting language and some statistics.

Do courses in this discipline lead to a qualification?

No. A completion record shows that you finished a course. It is not a certificate or a professional qualification.

条款

术语表

建模
A quantitative method that processes input data into quantitative estimates.
模型风险
The risk of adverse consequences from decisions based on incorrect or misused model outputs.
Effective challenge
Critical review of a model by informed, independent parties who can cause change.
Back-testing
Comparing model predictions with what actually happened.
Benchmarking
Comparing a model with another model or a simpler method.
Model lifecycle
The steps from model design to decommissioning.

Check your understanding

  1. What are the two main reasons model risk occurs?

    Fundamental errors in the model, and incorrect or inappropriate use of the model or its outputs.

  2. What does effective challenge depend on?

    A combination of incentives, competence and influence.

  3. Why can a model never be fully validated?

    Natural systems are never closed and results are nonunique, so confirmation is partial.

Continue with a course

来源

参考文献

10 sources, numbered in the order they are first cited
  1. Board of Governors of the Federal Reserve System (2011-04). Supervisory Guidance on Model Risk Management (SR 11-7, Attachment). www.federalreserve.gov. Accessed 2026-11-17.
  2. Bank of England, Prudential Regulation Authority (2023-05). SS1/23: Model risk management principles for banks. www.bankofengland.co.uk. Accessed 2026-11-17.
  3. Office of the Superintendent of Financial Institutions (Canada) (2025-09). Guideline E-23: Model Risk Management (2027). www.osfi-bsif.gc.ca. Accessed 2026-11-17.
  4. European Central Bank, Banking Supervision (2026). Supervision of internal models. www.bankingsupervision.europa.eu. Accessed 2026-11-17.
  5. Science (Oreskes, Shrader-Frechette and Belitz) (1994-02). Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences. doi.org. Accessed 2026-11-17.
  6. scikit-learn developers (2026). Common pitfalls and recommended practices (data leakage). scikit-learn.org. Accessed 2026-11-17.
  7. U.S. National Institute of Standards and Technology (2026). AI Risk Management Framework. www.nist.gov. Accessed 2026-11-17.
  8. European Commission (2026). AI Act: regulatory framework for AI. digital-strategy.ec.europa.eu. Accessed 2026-11-17.
  9. Nature Machine Intelligence (Rudin) (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. doi.org. Accessed 2026-11-17.
  10. Financial Stability Board (2017-11). Artificial intelligence and machine learning in financial services. www.fsb.org. Accessed 2026-11-17.

记录

Editorial record

Facts and course states are given as of 17 November 2026. This guide is reviewed at least every 270 days and whenever a cited report publishes a new edition or a course of the faculty opens.

Published by the Academy desk of Risk Academy. To report an error or request a correction, write through the Risk Academy contact page; corrections are dated and listed with the page. The Academy's statement on AI use explains how AI assistance is used and reviewed. Text, visuals and data are licensed CC BY 4.0, except where a cited source states other terms.

Cite this article

Academy desk, Risk Academy (2026, November 17). Model risk and validation: a guide to the field. Risk Academy. https://risksacademy.com/risk-science/learning-model-risk-and-validation-guide/

Republish

Republish or translate this article free of charge, in print or online, under CC BY 4.0: keep the byline and Risk Academy, link to the original and note any change. Figures, maps and twins made by the institutions carry the same licence; third-party data keeps its own credit line.

Attribution to paste
<p>Learning model risk and validation: a guide to the field — Academy desk, Risk Academy, Risk Academy. First published 17 November 2026 at <a href="https://risksacademy.com/risk-science/learning-model-risk-and-validation-guide/">https://risksacademy.com/risk-science/learning-model-risk-and-validation-guide/</a>. Republished under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).</p>

占比

EmailLinkedInBlueskyMastodon