Skip to content

RA · Foundations

AI safety and evaluation: what to learn, in what order, and why

A learning guide to AI safety and evaluation: how AI systems are tested, who does it, the references to know, a suggested order of study and recorded learning.

As of
17 November 2026
Lektüre
11 minutes
Evidenz
10 sources, cited in order
Stufe
Introductory to intermediate
Lernzeit
15 minutes
NEXUS TWIN · WF-1 · CLOUDSEED bd7f7427RA-F02.DISCIPLINE.AI-SAFETY-EVALUATIONxyzSCHEMATIC · CLOUDAGENT TRACK
WF-1 · cloud Schematic wireframe twin (cloud, compute and data systems). An illustration, not data.

What you will be able to do

  • Explain why a test result is evidence about a test, not about the world.
  • Name the main reports, institutions and frameworks in the field.
  • Describe evaluation as a loop of testing, red-teaming, monitoring and feedback.
  • Say what a safety case is and what it must contain.
  • Find the courses of the discipline and read their pages for what is open.
Stufe
Introductory to intermediate
Lernzeit
15 minutes
Before you start
The learning guide to the Faculty of Technology and Digital Systems, or equivalent background
International AI Safety Report
100+
independent experts who wrote the 2026 report1
International AI Safety Report
30+
countries and organisations that backed the report1
NIST AI RMF
4
core functions: govern, map, measure and manage4

01What AI safety and evaluation covers

What AI safety and evaluation covers

AI safety and evaluation is the Academy discipline of AI safety and evaluation, one of the disciplines of the Faculty of Technology and Digital Systems. On the Academy's own description it covers how the capabilities, reliability and risks of AI systems are tested before and after deployment. Its themes include evaluation design, red-teaming, benchmarks and safety cases.

It sits in the Department of Artificial Intelligence and Autonomous Systems and borders artificial intelligence and machine learning, AI assurance and audit and robotics and autonomous systems.

02Why testing AI is hard and important

Why testing AI is hard and important

An AI system is not tested the way a bridge is. Its behaviour depends on data, prompts and settings that change from one use to the next. A single test shows a floor, not a ceiling, and a model may do something a test did not look for.

The field has public reference points. The International AI Safety Report describes itself as a comprehensive review of the latest science on the capabilities and risks of general-purpose AI systems. Its Expert Advisory Panel was nominated by over 30 countries and intergovernmental organisations, and it was written by over 100 independent experts 1.

The Bletchley Declaration names the problem. It says particular safety risks arise at the frontier of AI, understood as highly capable general-purpose models, including foundation models, that could perform a wide variety of tasks 2. The Stanford AI Index adds a warning: its 2026 report says the frameworks needed to govern, evaluate and understand AI are falling behind 3.

03A suggested learning order in four stages

A suggested learning order in four stages

The Academy describes study at four levels: introductory, intermediate, advanced and integrative. The four stages below follow the same climb, but they are about subjects, not levels.

Stage one: what can go wrong. Start with a clear list of harms. The NIST AI RMF 1.0 names the characteristics of trustworthy AI: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy enhanced, and fair with harmful bias managed 4. The guide to technology and digital systems gives the wider field.

Stage two: how systems are tested. Learn what an evaluation measures and what it cannot. NIST highlights the importance of test, evaluation, verification and validation processes throughout the AI lifecycle 4. The AI Security Institute of the United Kingdom describes research and infrastructure to understand the capabilities and impacts of advanced AI and to develop and test risk mitigations 5.

Stage three: governance and institutions. Read who sets expectations. The EU AI Act puts in place rules for providers of general-purpose AI models, for example on transparency and copyright, and notes that some of these models could carry systemic risks if they are very capable or widely used 6. The European AI Office enforces the rules for general-purpose AI models 7.

Stage four: safety cases and monitoring. Study how a developer or deployer argues that a system is safe enough for a use, and how problems are found after release. The OECD AI Incidents and Hazards Monitor documents AI incidents and hazards to help policymakers and practitioners gain insight into risks and harms 8. The field of The Global Risks Alliance (GRA) on AI and models and the discipline's courses extend all four stages.

The order is a suggestion. A policy officer may begin at stage three; a machine learning engineer may begin at stage two. What matters is that each new idea has the ideas beneath it in place.

Fig. 01Four stages of study in AI safety and evaluation

Four stages of study in AI safety and evaluationA rising ladder of four steps: what can go wrong, how systems are tested, governance and institutions, and safety cases and monitoring, each with the documents or bodies a learner meets at that step.01What can go wrongNIST trustworthiness characteristics02How systems are testedTEVV, national institutes03GovernanceEU AI Act, AI Office04Safety cases and monitoringIncidents, field evidence
The suggested learning order of this guide, with the documents and bodies that anchor each stage 4,5,6,8. The order is a suggestion, not a required curriculum.4,5,6,8Sources: NIST; UK AI Security Institute; European Commission; OECD.AI. Figure CC BY 4.0. As of 17 November 2026.
Figure 1 in words

Four steps rise from left to right. Step one, what can go wrong: the characteristics of trustworthy AI in the NIST AI RMF. Step two, how systems are tested: test, evaluation, verification and validation, and the UK AI Security Institute. Step three, governance and institutions: the EU AI Act and the European AI Office. Step four, highlighted, safety cases and monitoring: the OECD incidents monitor.

04Before and after release

Before and after release

Figure 2 is a teaching frame that shows evaluation as a loop, not a gate: tests before release, then red-teaming, where testers try to make a system fail, then monitoring and incident reporting after release. What is learned goes back into the design and the next round of tests.

Public bodies describe parts of this loop. The European Commission's AI Act page describes post-market monitoring and the reporting of serious incidents 6. The OECD describes its monitor as an automated media discourse monitor of AI incidents and hazards, in beta, and says its information should not be reported as representing the official views of the OECD or its member countries 8. The UK AI Security Institute shows its own work in the open: its home page features an evaluation of the cyber capabilities of leading open-weight and closed-weight AI models 5.

Fig. 02Before and after release

Before and after releaseFive stacked layers: pre-release tests, red-teaming, monitoring in use, incident reporting, and feedback to design and the next tests.01TestsKnown failures, before release02Red-teamUnknown failures03ÜberwachungBehaviour in real use04IncidentsReports and hazards05RückkopplungInto design and new tests
A teaching frame for evaluation as a loop: tests, red-teaming, monitoring and incident reporting feed back into design. Only two layers rest on sources: the EU AI Act page describes post-market monitoring 6, and the OECD AI Incidents and Hazards Monitor tracks AI incidents and hazards 8.6,8Sources: European Commission; OECD.AI. Figure CC BY 4.0. As of 17 November 2026.
Figure 2 in words

Five layers stacked from top to bottom. One, pre-release tests. Two, highlighted, red-teaming. Three, monitoring in real use. Four, incident reporting. Five, feedback to design and to the next round of tests.

05Three core ideas every learner should hold

Three core ideas every learner should hold

Three ideas carry most of the weight in this discipline. Each is simple to state and takes practice to use.

Evaluation continues. Test, evaluation, verification and validation run throughout the lifecycle 4.

Frontier models raise special risks. Highly capable general-purpose models are a distinct focus 2.

Incidents are data. Tracking what goes wrong in use informs policy 8.

06Worked example: a benchmark score

Worked example: a benchmark score

Worked example (teaching values): the numbers in this section are invented for teaching and describe no real model. A model scores ninety in a hundred on a public benchmark of a hundred questions. A developer wants to claim that it is strong at the skill the benchmark covers.

Worked example, continued (teaching values): the evaluators write fifty fresh questions that test the same skill but do not appear anywhere public. On these the model scores seventy in a hundred. The gap of twenty points suggests that some benchmark questions or close copies may have been in the training data, so the first score overstates the skill.

The lesson is that a score is evidence about a test, not about the world. Stage two teaches how to build fresh tests and how to report the limit of each one. Stage four teaches how to confirm in real use what the tests suggest.

07Canonical references and bodies to know

Canonical references and bodies to know

A learner in this discipline should know a short list of references, what kind of evidence each holds and how often it is renewed. Primary documents come first; summaries are useful only after them.

For the state of the science: the International AI Safety Report 1. For frontier risk and international commitments: the Bletchley Declaration 2. For national evaluation work: the UK AI Security Institute 5. For risk frameworks: the NIST AI RMF 94. For EU rules and the office that applies them: the AI Act and the European AI Office 67. For incident evidence: the OECD AI Incidents and Hazards Monitor 8. For trend data: the Stanford AI Index 3. For international principles: the OECD AI Principles 10.

08How to read an AI safety report

How to read an AI safety report

A learner meets safety reports early and reads them badly if nobody explains their shape. Three habits help. First, find the issuer and the process. The International AI Safety Report was overseen by an expert panel nominated by over 30 countries and intergovernmental organisations 1. The Stanford AI Index states a mission to provide unbiased, rigorously vetted and globally sourced data 3. A mission statement is a claim by the issuer, so read the method too.

Second, find the date and the edition. The International AI Safety Report published a second edition in February 2026 1. Third, separate what a report finds from what it recommends. Write the finding, the date and the limit beside any figure you take from it. The OECD describes the OECD.AI platform as dedicated to promoting trustworthy, human-centric AI, which is a purpose, not a finding 10.

09Skills that make the knowledge usable

Skills that make the knowledge usable

Knowledge becomes useful through a small set of skills. The discipline treats these as its practical core.

Scoping a test. State the risk, the system and the use. Designing an evaluation. Choose tasks, metrics and controls. Red-teaming. Probe for failures a standard test would miss. Checking test hygiene. Make sure tests are not in the training data. Reading a system card. Find what was tested, what was not and by whom. Writing a safety case. Claim, argument, evidence and limit. Reporting an incident. Describe it clearly so that others can learn from it.

10How expertise is shown at the Academy

How expertise is shown at the Academy

Expertise in this field is shown by work that others can check. The Academy records learning, not credentials.

Finishing a course gives a completion record. It is not a certificate, an accreditation or a qualification, and it does not state that a person is competent in a profession. Achievements on the Academy recognise assessed learning only.

The course catalogue page for this discipline lists two courses: AI safety and evaluation: foundations and practice, at the intermediate level, and AI safety and evaluation: frontier methods, at the advanced level. On 17 November 2026 the page shows the first as in authoring and the second as planned. The discipline page shows no primary course open and two advanced courses in preparation. The catalogue says the NXS Learning catalogue lists every open course and is where you enrol, so this guide does not point to an Academy course as a place to enrol. Check the catalogue before you plan.

Outside the Academy, the strongest evidence of expertise is a body of work: a documented evaluation of a public model that cites its sources and dates, a reproducible test with its limits stated, or a clear safety brief that a reviewer has checked. Keep such a record from your first course.

11Emerging areas to watch

Emerging areas to watch

Several areas are changing what an evaluator needs to know. Each is a place to read on once the core is in place.

General-purpose AI rules. The European AI Office supports governance bodies in Member States and enforces the rules for general-purpose AI models 7. Open and closed models. The UK institute features a comparison of the cyber capabilities of open-weight and closed-weight systems 5. Hazard tracking. The OECD monitor aims over time to show risk patterns in AI incidents and hazards 8. Data transparency. The AI Index describes a field in which data transparency is declining 3.

Related work across the Nexus network gives wider context. See the field page of The Global Centre for Risk and Innovation (GCRI) on artificial intelligence, the page of The Global Risks Forum (GRF) on AI and digital governance and the proposed Nexus Standards for digital and AI systems. These pages describe programmes and proposals. A link to them implies no endorsement of the sources this guide cites.

What this guide does not claim: it is suggested learning, not a curriculum that anyone must follow, and it is not legal, technical or professional advice. It reports figures as their issuers published them and draws no new estimates from them; the only invented numbers are the labelled teaching values. It names institutions and reports for learning without implying any partnership. Risk Academy does not certify competence, and a completion record is not a professional qualification. The Nexus Standards are proposed and in draft, not adopted. The Academy records no partners, cohorts or learner outcomes for this discipline, and this guide implies none.

Fragen

Fragen und Antworten

Where should a complete beginner start in AI safety and evaluation?

Start with stage one, what can go wrong. Read the characteristics of trustworthy AI in the NIST framework and the executive material of the International AI Safety Report. Check the course catalogue for the discipline before you plan, because the Academy courses were not yet open for enrolment on 17 November 2026.

What is the difference between AI safety and AI evaluation?

Evaluation is the work of testing what a system can do and how reliably. Safety is the wider aim of keeping harm within acceptable limits. Evaluation provides evidence for safety claims, but a passed test is not a guarantee of safety.

What is red-teaming?

It is a method in which testers deliberately try to make a system fail or misbehave, so that weaknesses are found before users find them. It complements standard benchmarks, which look for known failures on fixed tasks.

Why do benchmark scores need care?

A score describes performance on one test, under one set of conditions. If test questions leak into training data, the score can overstate the skill. Fresh tests, stated limits and repeated runs help a reader judge what a number means.

Does finishing a course in this discipline give me a professional qualification?

No. A completion record shows that you finished a course. It is not a certificate or a professional qualification, and the Academy does not certify competence. If you need a recognised qualification, check the requirements of the body that awards it.

Nutzungsbedingungen

Glossar

Evaluation
Testing what an AI system can do and how reliably.
Red-teaming
Testers deliberately try to make a system fail or misbehave.
Benchmark
A fixed set of tasks used to compare systems.
Sicherheitsnachweis
A structured argument, with evidence, that a system is safe enough for a use.
General-purpose AI
AI that can perform a wide range of tasks and is the basis for many applications.
Frontier AI
Highly capable general-purpose models that could perform a wide variety of tasks, as the Bletchley Declaration describes them.

Check your understanding

  1. Why can a high benchmark score overstate a model's skill?

    Because test questions or close copies may have been in the training data. Fresh tests of the same skill show whether the score holds.

  2. What does red-teaming add to benchmarks?

    It looks for unknown failures by trying to make the system fail, while benchmarks look for known failures on fixed tasks.

  3. Is a passed test proof that a system is safe?

    No. A test shows that particular failures were not found under particular conditions.

Continue with a course

Quellen

Quellen

10 sources, numbered in the order they are first cited
  1. International AI Safety Report (2026). International AI Safety Report. internationalaisafetyreport.org. Accessed 2026-11-17.
  2. UK Government (2023). The Bletchley Declaration by countries attending the AI Safety Summit. www.gov.uk. Accessed 2026-11-17.
  3. Stanford Institute for Human-Centered Artificial Intelligence (2026). AI Index. hai.stanford.edu. Accessed 2026-11-17.
  4. U.S. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. nvlpubs.nist.gov. Accessed 2026-11-17.
  5. AI Security Institute (United Kingdom) (2026). The AI Security Institute. www.aisi.gov.uk. Accessed 2026-11-17.
  6. European Commission (2026). AI Act: regulatory framework for AI. digital-strategy.ec.europa.eu. Accessed 2026-11-17.
  7. European Commission (2026). European AI Office. digital-strategy.ec.europa.eu. Accessed 2026-11-17.
  8. OECD.AI Policy Observatory (2026). AIM: AI Incidents and Hazards Monitor. oecd.ai. Accessed 2026-11-17.
  9. U.S. National Institute of Standards and Technology (2026). AI Risk Management Framework. www.nist.gov. Accessed 2026-11-17.
  10. OECD.AI Policy Observatory (2026). OECD AI Principles overview. oecd.ai. Accessed 2026-11-17.

Eintrag

Editorial record

Facts and course states are given as of 17 November 2026. This guide is reviewed at least every 270 days and whenever a cited source publishes a new edition or a course of the faculty opens.

Published by the Academy desk of Risk Academy. To report an error or request a correction, write through the Risk Academy contact page; corrections are dated and listed with the page. The Academy's statement on AI use explains how AI assistance is used and reviewed. Text, visuals and data are licensed CC BY 4.0, except where a cited source states other terms.

Cite this article

Academy desk, Risk Academy (2026, November 17). AI safety and evaluation: a learning guide. Risk Academy. https://risksacademy.com/technology-digital/learning-ai-safety-and-evaluation-guide/

Republish

Republish or translate this article free of charge, in print or online, under CC BY 4.0: keep the byline and Risk Academy, link to the original and note any change. Figures, maps and twins made by the institutions carry the same licence; third-party data keeps its own credit line.

Attribution to paste
<p>AI safety and evaluation: what to learn, in what order, and why — Academy desk, Risk Academy, Risk Academy. First published 17 November 2026 at <a href="https://risksacademy.com/technology-digital/learning-ai-safety-and-evaluation-guide/">https://risksacademy.com/technology-digital/learning-ai-safety-and-evaluation-guide/</a>. Republished under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).</p>

Anteil

EmailLinkedInBlueskyMastodon