Illustrative: layers of units and their weighted connections, an activation front sweeping from input to output

Risk Academy, led by the Nexus Consortium · Course catalogue · Technology and Digital Systems

Artificial Intelligence and Autonomous Systems

AI safety and evaluation

AI safety and evaluation, taught twice over: as the working methods of the field and as its frontier since 2024, with labs on the Academy's tools, open data and the Nexus Standards.

Core course Frontier course

The discipline

01Core course · L2 Intermediate

AI safety and evaluation: foundations and practice

This area covers how the capabilities, reliability and risks of AI systems are tested before and after deployment: evaluation design, red-teaming, benchmarks and their limits, safety cases for frontier models and agentic systems, and incident reporting.

Level
L2 Intermediate
Study time
about 24 hours (planning estimate)
Status
In authoring
Prerequisites
An L1 course in the area, or an entry review

Why now

Shows why independent and held-out benchmarks matter when relying on vendor-reported evaluation results. National Institute of Standards and Technology, CAISI Evaluation of DeepSeek V4 Pro, 1 May 2026

Learning outcomes

  1. Define red-teaming and the other key terms of the discipline and use them correctly in a written case
  2. Explain evaluation design and metrics and red-teaming and adversarial testing with a worked example and its source
  3. Apply the discipline's method to a documented case and explain each result with its source
  4. Analyse a named case against benchmarks and data contamination, stating evidence, date and limits

Modules

  1. Evaluation design and metrics
    • Evaluation design and metrics
    • Red-teaming and adversarial testing
    • Benchmarks and data contamination
  2. Dangerous capability evaluations
    • Dangerous capability evaluations
    • Safety cases for frontier models
    • Agentic system evaluation
  3. Monitoring after deployment
    • Monitoring after deployment
    • Incident reporting and disclosure
  4. Lab: the discipline's method
    • An attack success rate is measured per attempt against a stated attacker budget; the chance that at least one of k attempts succeeds grows as 1 minus (1 minus s) to the k, and a release gate compares that cumulative figure, after defences, with the level a safety case can tolerate.
    • Work the method through a documented case and explain each result
    • Read the method's sources and stated values

Labs with Academy tools

Nexus Standards

Teaches

  • NRS-T1 Artificial intelligence risk
  • NRS-T16 AI agents and autonomous decision systems risk
  • NRS-A9 Evaluation suites for AI in risk work
  • NRS-H95 Hazard family: AI system failure and misalignment
  • NRS-H96 Hazard family: AI security and model supply chain
  • NRS-H97 Hazard family: AI misuse and capability uplift
  • NRS-H98 Hazard family: Agentic AI acting at scale
  • NRS-M19 Method fabric: Cyber, AI and frontier technology

Relies on

  • NS-1 The UNI and the artefact catalogue
  • NS-16 UNI application profile (how articles, documents, code, data, APIs and interfaces carry the UNI)
  • NRS-A1 Standards as code: machine-readable standards, requirements and tests
  • NRS-C1 Risk vocabulary and ontology
  • NRS-C2 Risk assessment method
  • NRS-C3 Hazard, exposure, vulnerability and capacity data
  • NRS-C4 Risk data model, identifiers and exchange
  • NRS-C7 Early warning, triggers and anticipatory action

Assessment

  • 40%Lab recordWork the discipline's method through a documented case and explain each result with its source and date
  • 40%Case memoApply the discipline's methods to a named place or organisation, stating evidence and limits
  • 20%Key terms checkShort answers on the key terms, marked against the definitions

Delivery

  • Self-paced online, with the labs in the browser
  • Cohort study over six to eight weeks, with a weekly lab review
  • Co-delivered with a partner institution, inside its own courses

Partner on this course

  • Co-teach a module and bring a current case from your work
  • Host a cohort for your learners, with your own schedule and review
  • Contribute a lab: a dataset your institution publishes openly, or a model
  • Review the course content and its references before it opens
  • Translate the course and adapt its cases to your region

How institutions and experts take part

In the Academy LMS

Course on the learning platform
Intermediate, 4 topics
Cohort group
A community group for each cohort of learners
Achievement
Recorded on completion of the assessed work

References

  1. National Institute of Standards and Technology, CAISI Evaluation of DeepSeek V4 Pro, 1 May 2026
  2. Shevlane et al., Model evaluation for extreme risks, 2023
  3. Phuong et al., Evaluating Frontier Models for Dangerous Capabilities, 2024
  4. Clymer et al., Safety Cases: How to Justify the Safety of Advanced AI Systems, 2024
02Live map

Connected activity in the United Kingdom and Ireland

Night-time lights and population density, the footprint of electrified, connected activity, over the United Kingdom and Ireland.

The interactive map needs JavaScript; the layer list and sources are below.

Layers 2

Connected activity

Latest available

NASA GIBS / EOSDIS · ASTER GDEM (METI, NASA) · Blue Marble (NASA) · © OpenStreetMap contributors (ODbL) · Natural Earth

Loading the latest NASA imagery.

Drag to move. Ctrl or ⌘ and scroll, pinch or the buttons to zoom; hover for the values under the pointer; reset returns to the region.

Connected activity, framed on the United Kingdom and Ireland.Satellite and model products as NASA publishes them, each with its own date. They are indicators, not assessments of any place.

Data: NASA GIBS · ASTER GDEM · Blue Marble · © OpenStreetMap contributors · Natural Earth · NASA EONETSources and licences

  • We acknowledge the use of imagery provided by services from NASA's Global Imagery Browse Services (GIBS), part of NASA's Earth Science Data and Information System (ESDIS). Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • We acknowledge the use of imagery provided by services from NASA's Global Imagery Browse Services (GIBS), part of NASA's Earth Science Data and Information System (ESDIS). Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • VIIRS Black Marble night-time lights, NASA Goddard Space Flight Center; distributed by the NASA Level-1 and Atmosphere Archive and Distribution System (LAADS DAAC); imagery via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • Center for International Earth Science Information Network (CIESIN), Columbia University. 2018. Gridded Population of the World, Version 4 (GPWv4): Population Density, Revision 11. NASA Socioeconomic Data and Applications Center (SEDAC). Licensed under CC BY 4.0. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). doi:10.7927/H49C6VHW. CC BY 4.0
  • VIIRS near-real-time products (flood, land surface temperature, snow cover, corrected reflectance), NASA LANCE; imagery via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • ASTER GDEM is a product of METI and NASA (ASTER Global Digital Elevation Model V003, NASA LP DAAC); imagery via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). Cite as: NASA/METI/AIST/Japan Spacesystems and U.S./Japan ASTER Science Team (2019). ASTER Global Digital Elevation Model V003. doi:10.5067/ASTER/ASTGTM.003. NASA open data
  • Made with Natural Earth. Free vector and raster map data @ naturalearthdata.com. Public domain
  • © OpenStreetMap contributors (ODbL), via NASA GIBS (coastlines, boundaries, roads and place names). Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). ODbL
  • Blue Marble, NASA Earth Observatory; imagery via NASA GIBS. Colours redrawn from the published colour map; values averaged into H3 cells where shown (modified). NASA open data
  • Context on the globe and flat map: coastlines, boundaries, roads and place names from NASA GIBS reference layers (OpenStreetMap data, ODbL, © OpenStreetMap contributors); Blue Marble imagery as an optional base; 3D terrain and the elevation read-out from the ASTER GDEM elevation colour index (5 m to 200 m classes, exaggerated for legibility). Boundaries are drawn as published and imply no position on any territory.
  • Globe: MapLibre GL JS 6.9.1, BSD-3-Clause, © MapLibre contributors.
  • Live events: NASA Earth Observatory Natural Event Tracker (EONET); events are indicative and keep their original source links.
  • NASA GIBS: We acknowledge the use of imagery provided by services from NASA's Global Imagery Browse Services (GIBS), part of NASA's Earth Science Data and Information System (ESDIS).
  • Optional layers (3 sources in the layer list: live, forecast, observed and context) print their provider credit on the map and here when switched on; forecasts are model output and derived indices are estimates.

Layers and dates

  • Night-time lights, daily: VIIRS Black Marble (VNP46A2), NASA LAADS DAAC · 12 Oct 2026
  • Population density, 2020: Population: CIESIN, Columbia University, Gridded Population of the World v4.11 (2018), NASA SEDAC, CC BY 4.0 · static
  • True colour: VIIRS NOAA-20 corrected reflectance, NASA LANCE · 13 Oct 2026

03How learning runs

How learning runs, as a process you can run

Select a scenario and press Play: course, pathway, practice, faculty, cells and record.

04What is changing

What is changing, with standards and evidence

AI safety and evaluation: frontier methods

The methods, data, tools and studies that changed AI safety and evaluation between 2024 and 2026, taught by reproducing them on open data.

Level
L3 Advanced
Study time
about 28 hours (planning estimate)
Status
Planned
Prerequisites
AI safety and evaluation: foundations and practice; Working knowledge of a scripting language (Python or R) for the labs

Why now

Shows why independent and held-out benchmarks matter when relying on vendor-reported evaluation results. National Institute of Standards and Technology, CAISI Evaluation of DeepSeek V4 Pro, 1 May 2026

Learning outcomes

  1. Explain what changed in AI safety and evaluation since 2024 and cite the studies, data or rules behind each change
  2. Reproduce independent evaluation finds gap with developer self-reported scores on open data and state the uncertainty of the result
  3. Evaluate a 2024 to 2026 study: its question, data, method, limits and what it changes for practice
  4. Design an analysis that combines a frontier method with the faculty's instruments, and justify each choice

Modules

  1. Independent evaluation finds gap with developer self-reported scores
    • NIST's Center for AI Standards and Innovation found DeepSeek V4 Pro scored better on the developer's own evaluations than on CAISI's, including non-public benchmarks, estimating it trails leading US models by about eight months.
    • Shows why independent and held-out benchmarks matter when relying on vendor-reported evaluation results.
    • National Institute of Standards and Technology, 1 May 2026
  2. UK AI Security Institute publishes first Frontier AI Trends Report
    • Drawing on evaluations of over 30 frontier models since 2023, the institute reports fast capability growth in cyber, chemistry and biology tasks, universal jailbreaks found in every system tested, and rising self-replication precursors.
    • Offers government test results on dangerous capabilities and safeguard strength that evaluators can use to benchmark their own methods.
    • AI Security Institute (UK), 18 December 2025
  3. Key update reviews technical safeguards and frontier risk management
    • The second key update finds prompt injection attacks still succeed about half the time with ten attempts, small data-poisoning attacks can be effective, and at least 12 companies now publish frontier safety frameworks with mostly implicit thresholds.
    • Clarifies how much assurance current safeguards provide, which shapes red-teaming scope and residual risk judgements.
    • International AI Safety Report, 25 November 2025
  4. Time-horizon metric measures growth in autonomous task length
    • Researchers proposed measuring AI capability by the length of software tasks models complete reliably, and found this horizon has doubled roughly every seven months since 2019.
    • Gives evaluators a trend measure for agentic capability that can inform forecasts and thresholds in safety cases.
    • arXiv, 18 March 2025
  5. UK renames AI Safety Institute as AI Security Institute
    • The UK government renamed the institute and refocused it on serious AI risks with security implications, such as chemical and biological misuse, cyber-attacks, fraud and child abuse material, adding a criminal misuse team.
    • Signals how a leading public evaluator now prioritises security-related misuse in its testing agenda.
    • UK Department for Science, Innovation and Technology, 14 February 2025
  6. Reproduction lab
    • Extend the discipline's method with one frontier method and compare its results
    • State the uncertainty and the limits of the result
    • Peer review against a shared checklist

Labs with Academy tools

Nexus Standards

Teaches

  • NRS-T1 Artificial intelligence risk
  • NRS-T16 AI agents and autonomous decision systems risk
  • NRS-A9 Evaluation suites for AI in risk work
  • NRS-H95 Hazard family: AI system failure and misalignment
  • NRS-H96 Hazard family: AI security and model supply chain
  • NRS-H97 Hazard family: AI misuse and capability uplift
  • NRS-H98 Hazard family: Agentic AI acting at scale
  • NRS-M19 Method fabric: Cyber, AI and frontier technology

Relies on

  • NS-1 The UNI and the artefact catalogue
  • NS-16 UNI application profile (how articles, documents, code, data, APIs and interfaces carry the UNI)
  • NRS-A1 Standards as code: machine-readable standards, requirements and tests
  • NRS-C1 Risk vocabulary and ontology
  • NRS-C2 Risk assessment method
  • NRS-C3 Hazard, exposure, vulnerability and capacity data
  • NRS-C4 Risk data model, identifiers and exchange
  • NRS-C7 Early warning, triggers and anticipatory action

Assessment

  • 50%Method reproductionReproduce one frontier method on open data and state the uncertainty of the result
  • 30%Study reviewA critical review of a 2024 to 2026 study: question, data, method, limits, what it changes
  • 20%Peer reviewReview two other learners' reproductions against a shared checklist

Delivery

  • Self-paced online, with the labs in the browser
  • Cohort study over six to eight weeks, with a weekly lab review
  • Co-delivered with a partner institution, inside its own courses

Partner on this course

  • Co-teach a module and bring a current case from your work
  • Host a cohort for your learners, with your own schedule and review
  • Contribute a lab: a dataset your institution publishes openly, or a model
  • Review the course content and its references before it opens
  • Translate the course and adapt its cases to your region

How institutions and experts take part

In the Academy LMS

Course on the learning platform
Expert, 6 topics
Cohort group
A community group for each cohort of learners
Achievement
Recorded on completion of the assessed work

References

  1. National Institute of Standards and Technology, CAISI Evaluation of DeepSeek V4 Pro, 1 May 2026
  2. AI Security Institute (UK), Frontier AI Trends Report, 18 December 2025
  3. International AI Safety Report, Second Key Update: Technical Safeguards and Risk Management, 25 November 2025
  4. arXiv, Measuring AI Ability to Complete Long Software Tasks, 18 March 2025
  5. UK Department for Science, Innovation and Technology, Tackling AI security risks to unleash growth and deliver Plan for Change, 14 February 2025
  6. Shevlane et al., Model evaluation for extreme risks, 2023
  7. Phuong et al., Evaluating Frontier Models for Dangerous Capabilities, 2024
  8. Clymer et al., Safety Cases: How to Justify the Safety of Advanced AI Systems, 2024
05Take part

Take part in the work

Start from who you are.

06Questions

Questions, answers and key terms

Is the course free?
Yes. Every course and learning material of Risk Academy, led by the Nexus Consortium, is free to study, self-paced or with a cohort. Ordinary participation has no fee; paid work, internships and fellowships are separate arrangements.
What does completing a course give me?
A completion record: the course, its version, the date and what was assessed, private by default and shared by you. It stacks into discipline, department and programme achievements. A record states what was studied and assessed and does not certify competence; it is not a licence, a professional title or an employment status.
Where do the data and the places come from?
Every faculty teaches on a twin of a real place built from open data; each layer carries its provider, licence and date, and every result on the Academy states its source. Maps and figures are indicators for teaching, never an assessment of a place.
How do I take part beyond the course?
Form a competence cell of three to seven people, study a shared pathway, take a supervised assignment in the National Working Groups, or bring a cohort from your institution. For organisations.

Key terms

Term

Red-teaming

Structured attempts to make a system fail or misbehave, to find weaknesses before others do.

Term

Benchmark contamination

When test data have leaked into a model's training data, so scores overstate its ability.

Term

Safety case

A structured argument, supported by evidence, that a system is acceptably safe for a given use.

Term

Frontier model

A general-purpose AI model at or near the most capable level available at a given time.

ReportsContribute

Contribute evidence from this page.

Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

Risk Academy
Related

Around AI safety and evaluation: courses

Risk Academy

Study AI safety and evaluation with us, or teach it

Every course is free and open. Institutions and experts can co-teach, host a cohort, contribute a lab or review a course before it opens.

Browse the course catalogue Partner with the Academy
Risk Academy

Courses · practice · credentials

Not for profit

Our articles, maps and twins are free to read and free to republish. Readers and partners keep it that way.

Support our work

Who funds us

Risk Academy is a not-for-profit institution. To ask who funds a piece of work, use the funding questions form.

Independence

Editors and the expert reviewers named on each article decide what we publish. Funders and partners do not review or approve articles before publication.

Corrections policy

Licence

Articles: CC BY 4.0. Figures, maps, twins and data: CC BY 4.0. Other uses: ask through the permissions form.

Privacy choices