RA · Foundations
AI safety and evaluation: what to learn, in what order, and why
A learning guide to AI safety and evaluation: how AI systems are tested, who does it, the references to know, a suggested order of study and recorded learning.
- As of
- 17 November 2026
- वाचन
- 11 minutes
- पुरावा
- 10 sources, cited in order
- स्तर
- Introductory to intermediate
- Study time
- 15 minutes
What you will be able to do
- Explain why a test result is evidence about a test, not about the world.
- Name the main reports, institutions and frameworks in the field.
- Describe evaluation as a loop of testing, red-teaming, monitoring and feedback.
- Say what a safety case is and what it must contain.
- Find the courses of the discipline and read their pages for what is open.
- स्तर
- Introductory to intermediate
- Study time
- 15 minutes
- Before you start
- The learning guide to the Faculty of Technology and Digital Systems, or equivalent background
01What AI safety and evaluation covers
What AI safety and evaluation covers
AI safety and evaluation is the Academy discipline of AI safety and evaluation, one of the disciplines of the Faculty of Technology and Digital Systems. On the Academy's own description it covers how the capabilities, reliability and risks of AI systems are tested before and after deployment. Its themes include evaluation design, red-teaming, benchmarks and safety cases.
It sits in the Department of Artificial Intelligence and Autonomous Systems and borders artificial intelligence and machine learning, AI assurance and audit and robotics and autonomous systems.
02Why testing AI is hard and important
Why testing AI is hard and important
An AI system is not tested the way a bridge is. Its behaviour depends on data, prompts and settings that change from one use to the next. A single test shows a floor, not a ceiling, and a model may do something a test did not look for.
The field has public reference points. The International AI Safety Report describes itself as a comprehensive review of the latest science on the capabilities and risks of general-purpose AI systems. Its Expert Advisory Panel was nominated by over 30 countries and intergovernmental organisations, and it was written by over 100 independent experts 1.
The Bletchley Declaration names the problem. It says particular safety risks arise at the frontier of AI, understood as highly capable general-purpose models, including foundation models, that could perform a wide variety of tasks 2. The Stanford AI Index adds a warning: its 2026 report says the frameworks needed to govern, evaluate and understand AI are falling behind 3.
03A suggested learning order in four stages
A suggested learning order in four stages
The Academy describes study at four levels: introductory, intermediate, advanced and integrative. The four stages below follow the same climb, but they are about subjects, not levels.
Stage one: what can go wrong. Start with a clear list of harms. The NIST AI RMF 1.0 names the characteristics of trustworthy AI: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy enhanced, and fair with harmful bias managed 4. The guide to technology and digital systems gives the wider field.
Stage two: how systems are tested. Learn what an evaluation measures and what it cannot. NIST highlights the importance of test, evaluation, verification and validation processes throughout the AI lifecycle 4. The AI Security Institute of the United Kingdom describes research and infrastructure to understand the capabilities and impacts of advanced AI and to develop and test risk mitigations 5.
Stage three: governance and institutions. Read who sets expectations. The EU AI Act puts in place rules for providers of general-purpose AI models, for example on transparency and copyright, and notes that some of these models could carry systemic risks if they are very capable or widely used 6. The European AI Office enforces the rules for general-purpose AI models 7.
Stage four: safety cases and monitoring. Study how a developer or deployer argues that a system is safe enough for a use, and how problems are found after release. The OECD AI Incidents and Hazards Monitor documents AI incidents and hazards to help policymakers and practitioners gain insight into risks and harms 8. The field of The Global Risks Alliance (GRA) on AI and models and the discipline's courses extend all four stages.
The order is a suggestion. A policy officer may begin at stage three; a machine learning engineer may begin at stage two. What matters is that each new idea has the ideas beneath it in place.
Fig. 01Four stages of study in AI safety and evaluation
Figure 1 in words
Four steps rise from left to right. Step one, what can go wrong: the characteristics of trustworthy AI in the NIST AI RMF. Step two, how systems are tested: test, evaluation, verification and validation, and the UK AI Security Institute. Step three, governance and institutions: the EU AI Act and the European AI Office. Step four, highlighted, safety cases and monitoring: the OECD incidents monitor.
04Before and after release
Before and after release
Figure 2 is a teaching frame that shows evaluation as a loop, not a gate: tests before release, then red-teaming, where testers try to make a system fail, then monitoring and incident reporting after release. What is learned goes back into the design and the next round of tests.
Public bodies describe parts of this loop. The European Commission's AI Act page describes post-market monitoring and the reporting of serious incidents 6. The OECD describes its monitor as an automated media discourse monitor of AI incidents and hazards, in beta, and says its information should not be reported as representing the official views of the OECD or its member countries 8. The UK AI Security Institute shows its own work in the open: its home page features an evaluation of the cyber capabilities of leading open-weight and closed-weight AI models 5.
Fig. 02Before and after release
Figure 2 in words
Five layers stacked from top to bottom. One, pre-release tests. Two, highlighted, red-teaming. Three, monitoring in real use. Four, incident reporting. Five, feedback to design and to the next round of tests.
05Three core ideas every learner should hold
Three core ideas every learner should hold
Three ideas carry most of the weight in this discipline. Each is simple to state and takes practice to use.
Evaluation continues. Test, evaluation, verification and validation run throughout the lifecycle 4.
Frontier models raise special risks. Highly capable general-purpose models are a distinct focus 2.
Incidents are data. Tracking what goes wrong in use informs policy 8.
06Worked example: a benchmark score
Worked example: a benchmark score
Worked example (teaching values): the numbers in this section are invented for teaching and describe no real model. A model scores ninety in a hundred on a public benchmark of a hundred questions. A developer wants to claim that it is strong at the skill the benchmark covers.
Worked example, continued (teaching values): the evaluators write fifty fresh questions that test the same skill but do not appear anywhere public. On these the model scores seventy in a hundred. The gap of twenty points suggests that some benchmark questions or close copies may have been in the training data, so the first score overstates the skill.
The lesson is that a score is evidence about a test, not about the world. Stage two teaches how to build fresh tests and how to report the limit of each one. Stage four teaches how to confirm in real use what the tests suggest.
07Canonical references and bodies to know
Canonical references and bodies to know
A learner in this discipline should know a short list of references, what kind of evidence each holds and how often it is renewed. Primary documents come first; summaries are useful only after them.
For the state of the science: the International AI Safety Report 1. For frontier risk and international commitments: the Bletchley Declaration 2. For national evaluation work: the UK AI Security Institute 5. For risk frameworks: the NIST AI RMF 94. For EU rules and the office that applies them: the AI Act and the European AI Office 67. For incident evidence: the OECD AI Incidents and Hazards Monitor 8. For trend data: the Stanford AI Index 3. For international principles: the OECD AI Principles 10.
08How to read an AI safety report
How to read an AI safety report
A learner meets safety reports early and reads them badly if nobody explains their shape. Three habits help. First, find the issuer and the process. The International AI Safety Report was overseen by an expert panel nominated by over 30 countries and intergovernmental organisations 1. The Stanford AI Index states a mission to provide unbiased, rigorously vetted and globally sourced data 3. A mission statement is a claim by the issuer, so read the method too.
Second, find the date and the edition. The International AI Safety Report published a second edition in February 2026 1. Third, separate what a report finds from what it recommends. Write the finding, the date and the limit beside any figure you take from it. The OECD describes the OECD.AI platform as dedicated to promoting trustworthy, human-centric AI, which is a purpose, not a finding 10.
09Skills that make the knowledge usable
Skills that make the knowledge usable
Knowledge becomes useful through a small set of skills. The discipline treats these as its practical core.
Scoping a test. State the risk, the system and the use. Designing an evaluation. Choose tasks, metrics and controls. Red-teaming. Probe for failures a standard test would miss. Checking test hygiene. Make sure tests are not in the training data. Reading a system card. Find what was tested, what was not and by whom. Writing a safety case. Claim, argument, evidence and limit. Reporting an incident. Describe it clearly so that others can learn from it.
10How expertise is shown at the Academy
How expertise is shown at the Academy
Expertise in this field is shown by work that others can check. The Academy records learning, not credentials.
Finishing a course gives a completion record. It is not a certificate, an accreditation or a qualification, and it does not state that a person is competent in a profession. Achievements on the Academy recognise assessed learning only.
The course catalogue page for this discipline lists two courses: AI safety and evaluation: foundations and practice, at the intermediate level, and AI safety and evaluation: frontier methods, at the advanced level. On 17 November 2026 the page shows the first as in authoring and the second as planned. The discipline page shows no primary course open and two advanced courses in preparation. The catalogue says the NXS Learning catalogue lists every open course and is where you enrol, so this guide does not point to an Academy course as a place to enrol. Check the catalogue before you plan.
Outside the Academy, the strongest evidence of expertise is a body of work: a documented evaluation of a public model that cites its sources and dates, a reproducible test with its limits stated, or a clear safety brief that a reviewer has checked. Keep such a record from your first course.
11Emerging areas to watch
Emerging areas to watch
Several areas are changing what an evaluator needs to know. Each is a place to read on once the core is in place.
General-purpose AI rules. The European AI Office supports governance bodies in Member States and enforces the rules for general-purpose AI models 7. Open and closed models. The UK institute features a comparison of the cyber capabilities of open-weight and closed-weight systems 5. Hazard tracking. The OECD monitor aims over time to show risk patterns in AI incidents and hazards 8. Data transparency. The AI Index describes a field in which data transparency is declining 3.
Related work across the Nexus network gives wider context. See the field page of The Global Centre for Risk and Innovation (GCRI) on artificial intelligence, the page of The Global Risks Forum (GRF) on AI and digital governance and the proposed Nexus Standards for digital and AI systems. These pages describe programmes and proposals. A link to them implies no endorsement of the sources this guide cites.
What this guide does not claim: it is suggested learning, not a curriculum that anyone must follow, and it is not legal, technical or professional advice. It reports figures as their issuers published them and draws no new estimates from them; the only invented numbers are the labelled teaching values. It names institutions and reports for learning without implying any partnership. Risk Academy does not certify competence, and a completion record is not a professional qualification. The Nexus Standards are proposed and in draft, not adopted. The Academy records no partners, cohorts or learner outcomes for this discipline, and this guide implies none.
प्रश्न
Questions and answers
Where should a complete beginner start in AI safety and evaluation?
Start with stage one, what can go wrong. Read the characteristics of trustworthy AI in the NIST framework and the executive material of the International AI Safety Report. Check the course catalogue for the discipline before you plan, because the Academy courses were not yet open for enrolment on 17 November 2026.
What is the difference between AI safety and AI evaluation?
Evaluation is the work of testing what a system can do and how reliably. Safety is the wider aim of keeping harm within acceptable limits. Evaluation provides evidence for safety claims, but a passed test is not a guarantee of safety.
What is red-teaming?
It is a method in which testers deliberately try to make a system fail or misbehave, so that weaknesses are found before users find them. It complements standard benchmarks, which look for known failures on fixed tasks.
Why do benchmark scores need care?
A score describes performance on one test, under one set of conditions. If test questions leak into training data, the score can overstate the skill. Fresh tests, stated limits and repeated runs help a reader judge what a number means.
Does finishing a course in this discipline give me a professional qualification?
No. A completion record shows that you finished a course. It is not a certificate or a professional qualification, and the Academy does not certify competence. If you need a recognised qualification, check the requirements of the body that awards it.
अटी
Glossary
- Evaluation
- Testing what an AI system can do and how reliably.
- Red-teaming
- Testers deliberately try to make a system fail or misbehave.
- Benchmark
- A fixed set of tasks used to compare systems.
- Safety case
- A structured argument, with evidence, that a system is safe enough for a use.
- General-purpose AI
- AI that can perform a wide range of tasks and is the basis for many applications.
- Frontier AI
- Highly capable general-purpose models that could perform a wide variety of tasks, as the Bletchley Declaration describes them.
संबंधित
Further reading across the network
- AI सुरक्षितता आणि मूल्यमापन
- Courses of the discipline
- तंत्रज्ञान आणि डिजिटल प्रणाली विद्याशाखा
- Department of Artificial Intelligence and Autonomous Systems
- Learning guide: technology and digital systems
- Learning guide: AI assurance and audit
- Learning guide: artificial intelligence and machine learning
- रोबोटिक्स आणि स्वायत्त प्रणाली
- GRA: AI and models
- GCRI: Artificial intelligence
- GRF: AI and digital governance
- Nexus Standards (proposed): digital and AI
Check your understanding
Why can a high benchmark score overstate a model's skill?
Because test questions or close copies may have been in the training data. Fresh tests of the same skill show whether the score holds.
What does red-teaming add to benchmarks?
It looks for unknown failures by trying to make the system fail, while benchmarks look for known failures on fixed tasks.
Is a passed test proof that a system is safe?
No. A test shows that particular failures were not found under particular conditions.
Continue with a course
- AI safety and evaluation: foundations and practice ai-safety-evaluation.core
- AI safety and evaluation: frontier methods ai-safety-evaluation.frontier
स्रोत
References
10 sources, numbered in the order they are first cited
- International AI Safety Report (2026). International AI Safety Report. internationalaisafetyreport.org. Accessed 2026-11-17.
- UK Government (2023). The Bletchley Declaration by countries attending the AI Safety Summit. www.gov.uk. Accessed 2026-11-17.
- Stanford Institute for Human-Centered Artificial Intelligence (2026). AI Index. hai.stanford.edu. Accessed 2026-11-17.
- U.S. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. nvlpubs.nist.gov. Accessed 2026-11-17.
- AI Security Institute (United Kingdom) (2026). The AI Security Institute. www.aisi.gov.uk. Accessed 2026-11-17.
- European Commission (2026). AI Act: regulatory framework for AI. digital-strategy.ec.europa.eu. Accessed 2026-11-17.
- European Commission (2026). European AI Office. digital-strategy.ec.europa.eu. Accessed 2026-11-17.
- OECD.AI Policy Observatory (2026). AIM: AI Incidents and Hazards Monitor. oecd.ai. Accessed 2026-11-17.
- U.S. National Institute of Standards and Technology (2026). AI Risk Management Framework. www.nist.gov. Accessed 2026-11-17.
- OECD.AI Policy Observatory (2026). OECD AI Principles overview. oecd.ai. Accessed 2026-11-17.
नोंद
Editorial record
Facts and course states are given as of 17 November 2026. This guide is reviewed at least every 270 days and whenever a cited source publishes a new edition or a course of the faculty opens.
Published by the Academy desk of Risk Academy. To report an error or request a correction, write through the Risk Academy contact page; corrections are dated and listed with the page. The Academy's statement on AI use explains how AI assistance is used and reviewed. Text, visuals and data are licensed CC BY 4.0, except where a cited source states other terms.
Cite this article
Academy desk, Risk Academy (2026, November 17). AI safety and evaluation: a learning guide. Risk Academy. https://risksacademy.com/technology-digital/learning-ai-safety-and-evaluation-guide/
@online{academy2026learningaisafetyande,
author = {{Academy desk, Risk Academy}},
title = {{AI safety and evaluation: a learning guide}},
publisher = {Risk Academy},
year = {2026},
month = {11},
url = {https://risksacademy.com/technology-digital/learning-ai-safety-and-evaluation-guide/},
language = {en}
} TY - ELEC AU - Academy desk, Risk Academy TI - AI safety and evaluation: a learning guide PB - Risk Academy PY - 2026 DA - 2026/11/17 UR - https://risksacademy.com/technology-digital/learning-ai-safety-and-evaluation-guide/ LA - en ER -
Republish
Republish or translate this article free of charge, in print or online, under CC BY 4.0: keep the byline and Risk Academy, link to the original and note any change. Figures, maps and twins made by the institutions carry the same licence; third-party data keeps its own credit line.
Attribution to paste
<p>AI safety and evaluation: what to learn, in what order, and why — Academy desk, Risk Academy, Risk Academy. First published 17 November 2026 at <a href="https://risksacademy.com/technology-digital/learning-ai-safety-and-evaluation-guide/">https://risksacademy.com/technology-digital/learning-ai-safety-and-evaluation-guide/</a>. Republished under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).</p>