Risk Academy, led by the Nexus Consortium · Course catalogue · Technology and Digital Systems
Ubuhlakani Bokwenziwa Nezinhlelo Ezizisebenzelayo
Ukuphepha nokuhlolwa kwe-AI
AI safety and evaluation, taught twice over: as the working methods of the field and as its frontier since 2024, with labs on the Academy's tools, open data and the Nexus Standards.
AI safety and evaluation: foundations and practice
This area covers how the capabilities, reliability and risks of AI systems are tested before and after deployment: evaluation design, red-teaming, benchmarks and their limits, safety cases for frontier models and agentic systems, and incident reporting.
AI safety and evaluation: foundations and practice
Level
L2 Intermediate
Study time
about 24 hours (planning estimate)
Isimo
In authoring
Prerequisites
An L1 course in the area, or an entry review
Why now
Shows why independent and held-out benchmarks matter when relying on vendor-reported evaluation results.
Define red-teaming and the other key terms of the discipline and use them correctly in a written case
Explain evaluation design and metrics and red-teaming and adversarial testing with a worked example and its source
Apply the discipline's method to a documented case and explain each result with its source
Analyse a named case against benchmarks and data contamination, stating evidence, date and limits
Modules
01
Evaluation design and metrics
Evaluation design and metrics
Red-teaming and adversarial testing
Benchmarks and data contamination
02
Dangerous capability evaluations
Dangerous capability evaluations
Safety cases for frontier models
Agentic system evaluation
03
Monitoring after deployment
Monitoring after deployment
Incident reporting and disclosure
04
Lab: the discipline's method
An attack success rate is measured per attempt against a stated attacker budget; the chance that at least one of k attempts succeeds grows as 1 minus (1 minus s) to the k, and a release gate compares that cumulative figure, after defences, with the level a safety case can tolerate.
Work the method through a documented case and explain each result
Explain what changed in AI safety and evaluation since 2024 and cite the studies, data or rules behind each change
Reproduce independent evaluation finds gap with developer self-reported scores on open data and state the uncertainty of the result
Evaluate a 2024 to 2026 study: its question, data, method, limits and what it changes for practice
Design an analysis that combines a frontier method with the faculty's instruments, and justify each choice
Modules
01
Independent evaluation finds gap with developer self-reported scores
NIST's Center for AI Standards and Innovation found DeepSeek V4 Pro scored better on the developer's own evaluations than on CAISI's, including non-public benchmarks, estimating it trails leading US models by about eight months.
Shows why independent and held-out benchmarks matter when relying on vendor-reported evaluation results.
National Institute of Standards and Technology, 1 May 2026
02
UK AI Security Institute publishes first Frontier AI Trends Report
Drawing on evaluations of over 30 frontier models since 2023, the institute reports fast capability growth in cyber, chemistry and biology tasks, universal jailbreaks found in every system tested, and rising self-replication precursors.
Offers government test results on dangerous capabilities and safeguard strength that evaluators can use to benchmark their own methods.
AI Security Institute (UK), 18 December 2025
03
Key update reviews technical safeguards and frontier risk management
The second key update finds prompt injection attacks still succeed about half the time with ten attempts, small data-poisoning attacks can be effective, and at least 12 companies now publish frontier safety frameworks with mostly implicit thresholds.
Clarifies how much assurance current safeguards provide, which shapes red-teaming scope and residual risk judgements.
International AI Safety Report, 25 November 2025
04
Time-horizon metric measures growth in autonomous task length
Researchers proposed measuring AI capability by the length of software tasks models complete reliably, and found this horizon has doubled roughly every seven months since 2019.
Gives evaluators a trend measure for agentic capability that can inform forecasts and thresholds in safety cases.
arXiv, 18 March 2025
05
UK renames AI Safety Institute as AI Security Institute
The UK government renamed the institute and refocused it on serious AI risks with security implications, such as chemical and biological misuse, cyber-attacks, fraud and child abuse material, adding a criminal misuse team.
Signals how a leading public evaluator now prioritises security-related misuse in its testing agenda.
UK Department for Science, Innovation and Technology, 14 February 2025
06
Reproduction lab
Extend the discipline's method with one frontier method and compare its results
State the uncertainty and the limits of the result
The Academy proposes how each discipline teaches toward the Nexus Standards; the mapping is open for public review. Bodies and standards are named so learners can find and apply their published work; the Academy teaches their use and links to the publisher, and no affiliation is implied.
04Imephu bukhoma
Connected activity in the United Kingdom and Ireland
Night-time lights and population density, the footprint of electrified, connected activity, over the United Kingdom and Ireland.
Connected activity, framed on the United Kingdom and Ireland.Satellite and model products as NASA publishes them, each with its own date. They are indicators, not assessments of any place.
Siyabonga ngokusetshenziswa kwezithombe ezinikezwa yizinsizakalo ze-Global Imagery Browse Services (GIBS) ye-NASA, ingxenye ye-Earth Science Data and Information System (ESDIS) ye-NASA. Imibala yadwetshwa kabusha emephini yemibala eshicilelwe; amanani enziwe isilinganiso emaseleni e-H3 lapho eboniswa khona (kushintshiwe). Idatha evulekile ye-NASA
Siyabonga ngokusetshenziswa kwezithombe ezinikezwa yizinsizakalo ze-Global Imagery Browse Services (GIBS) ye-NASA, ingxenye ye-Earth Science Data and Information System (ESDIS) ye-NASA. Imibala yadwetshwa kabusha emephini yemibala eshicilelwe; amanani enziwe isilinganiso emaseleni e-H3 lapho eboniswa khona (kushintshiwe). Idatha evulekile ye-NASA
Izibani zasebusuku ze-VIIRS Black Marble, NASA Goddard Space Flight Center; zisabalaliswa yi-NASA Level-1 and Atmosphere Archive and Distribution System (LAADS DAAC); izithombe nge-NASA GIBS. Imibala yadwetshwa kabusha emephini yemibala eshicilelwe; amanani enziwe isilinganiso emaseleni e-H3 lapho eboniswa khona (kushintshiwe). Idatha evulekile ye-NASA
Center for International Earth Science Information Network (CIESIN), Columbia University. 2018. Gridded Population of the World, Version 4 (GPWv4): Population Density, Revision 11. NASA Socioeconomic Data and Applications Center (SEDAC). Licensed under CC BY 4.0. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). doi:10.7927/H49C6VHW. CC BY 4.0
Siyabonga ngokusetshenziswa kwezithombe ezinikezwa yizinsizakalo ze-Global Imagery Browse Services (GIBS) ye-NASA, ingxenye ye-Earth Science Data and Information System (ESDIS) ye-NASA.
University, student and national chapter teams. Open
A cell acts within its host's rules and never replaces an authority. A CERT works alongside the local emergency services, follows their direction and is not an emergency service.
Structured attempts to make a system fail or misbehave, to find weaknesses before others do.
Benchmark contamination
When test data have leaked into a model's training data, so scores overstate its ability.
Safety case
A structured argument, supported by evidence, that a system is acceptably safe for a given use.
Frontier model
A general-purpose AI model at or near the most capable level available at a given time.
ImibikoNikela
Nikela ngobufakazi kusuka kuleli khasi.
Reports is the Consortium's quarterly publication, written by six desks. This page sits with the Academy Capability desk. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work. Sent to the wrong desk, it is forwarded, not refused.