Risk Academy, led by the Nexus Consortium · Technology and Digital Systems · Department of Artificial Intelligence and Autonomous Systems
Evaluation design, red-teaming, benchmarks and safety cases
Usalama wa AI na tathmini
AI safety and evaluation covers how the capabilities, reliability and risks of AI systems are tested before and after deployment.
Illustrative: layers of units and their weighted connections, an activation front sweeping from input to output
01Wigo
What the discipline covers
The subtopics its courses and labs teach.
- Kozi
- In preparation
- Ngazi
- In development
- Maabara
- In development
- Evaluation design and metrics
- Red-teaming and adversarial testing
- Benchmarks and data contamination
- Dangerous capability evaluations
- Safety cases for frontier models
- Agentic system evaluation
- Monitoring after deployment
- Incident reporting and disclosure
Courses in development extend the scope to further subjects: AI system evaluation methods; Frontier AI risk and safety cases.
02Ramani
Build a twin of any place
Search any place or open the world; the layers and generators follow aI safety and evaluation on the planet map, the globe and in 3D.
- Terrain 0 to 245 m
- 4 satellite dishes
- 74 communication masts and towers
- 12 telephone exchanges
- 7 data centres and cable landing stations
- 12 aerodromes
- 1 telecom offices
- Communication line 161 km
- 729 known exploited vulnerabilities since 2023-10-17
- Night-time lights, VIIRS day-night band: read live when the 3D twin opens
Drag to move, right drag or Shift with a drag to turn and tilt, Ctrl with the wheel to zoom. Keyboard: arrows move, Shift with arrows turn and tilt, + and – zoom, 0 resets, V steps through the views, Escape clears.
Flat plan of the same data: relief as contours from the inline grid, the mapped ground infrastructure by kind, the mapped cables and the numbers below. The 3D twin opens on devices that can draw it, once its data and code have been checked.
The boundaries, names and designations shown do not imply any judgement on the legal status of any territory, or any endorsement or acceptance of such boundaries. Countries: Natural Earth (public domain).
03The subject
The subject, methods and measures and debates and bodies
The discipline
The field and its scope
For evaluators, red teams, safety researchers and the public officers who read their results, who need to design tests of AI systems and know what a result can and cannot support.
AI safety and evaluation is the empirical study of what an AI system can do, how it fails and whether its safeguards hold, before and after deployment. It covers evaluation design and metrics, benchmarks and their contamination, red-teaming and adversarial testing, dangerous-capability evaluations in domains such as cyber operations, biological and chemical knowledge, autonomous replication and persuasion, safety cases that argue a system is acceptably safe, evaluation of agents that act with tools, and the monitoring and incident reporting that follow release. It is distinct from assurance and audit, which take evaluation results as evidence within an organisational and regulatory process.
The field is organised around a small number of frontier developers who run internal evaluations and publish safety frameworks with capability thresholds, government institutes in the United Kingdom, the United States, Japan, the Republic of Korea, Singapore and elsewhere that test models independently, academic groups that build benchmarks and study their validity, and third-party evaluators. Its central difficulty is measurement validity: a score is an estimate on a sample of tasks with a prompt and a scaffold, and the same model scores differently when any of those change. Dangerous capabilities are named here at category level only; the methods that would give uplift are not published.
Measures
Quantities and units
| Quantity | Unit or scale | What to know |
|---|---|---|
| Pass rate on an evaluation suite | percent of tasks solved, with a bootstrap confidence interval | Depends on the number of attempts allowed (pass at 1 versus pass at k), the scaffold, tool access and elicitation effort. Report the interval and the elicitation budget together; without them two scores are not comparable. |
| Attack success rate | percent of adversarial attempts that elicit the prohibited behaviour | Measured for a stated attacker budget (attempts, compute, access level). The second key update of the International AI Safety Report (2025) reports prompt injection succeeding about half the time within ten attempts for current systems; rates fall with defences and rise with adaptive attackers. |
| Task time horizon at 50 percent success | minutes of human expert time | Estimated by logistic fit across tasks of known human duration (Kwa et al., 2025); the doubling time of roughly seven months carries wide uncertainty and the task set is software-heavy. |
| Contamination rate | percent of benchmark items found in training data | Estimated by n-gram overlap or canary strings; underestimates paraphrased leakage and cannot be computed at all when training data are undisclosed. |
| Inter-rater agreement on graded outputs | Cohen's or Fleiss' kappa, minus 1 to 1 | Bounds how much of a human-graded safety score is signal. Kappa below 0.6 on a harm rubric means the rubric, not the model, dominates the result. |
| Incidents reported after deployment | count per period, by severity class | The AI Index (2026) counts 362 documented incidents in 2025 against 233 in 2024 from the AI Incident Database; counts reflect reporting effort and media attention as much as underlying harm. |
Mbinu
Methods and their limits
Each method is listed with what it assumes and where it fails.
| Mbinu | Assumes | Fails when |
|---|---|---|
| Static benchmarks with automatic scoring | The items represent the capability of interest, the answer key is correct and the items were not seen in training. | When items leak into training corpora, when scoring rewards format over substance, when the benchmark saturates, and when the capability of interest is open-ended and has no answer key. |
| Dangerous-capability evaluations | Capability can be elicited with a bounded effort, task performance in a proxy setting predicts uplift in the real setting, and thresholds can be set before the capability appears. | When elicitation is under-resourced, so absence of evidence is read as evidence of absence; when a model sandbags; and when a proxy task is far from the real pathway, which for the gravest categories it must be, for information-hazard reasons. |
| Red teaming, manual and automated | Red teamers cover the attack space well enough that unfound failures are rare, and automated attackers generalise to human ingenuity. | When coverage is unmeasured, when the red team lacks domain expertise, and when the deployed system differs from the tested one in prompts, tools or retrieval. The AI Security Institute reported finding universal jailbreaks in every system tested (2025). |
| Safety cases | A structured argument can connect evidence (evaluations, controls, monitoring) to a top-level claim of acceptable risk, and reviewers can judge the argument's soundness. | When the claim is vague, the evidence is incomplete for the deployment context, or the argument relies on inability claims that a stronger elicitation would overturn; the discipline inherits these problems from safety cases in aviation and nuclear practice. |
| Agent evaluation in sandboxed environments | The environment reproduces the tools, latencies and adversaries of real deployment, and success criteria can be checked automatically. | When the sandbox is cleaner than the world, when agents exploit environment bugs, when long tasks are too few to estimate reliability, and when the evaluated agent scaffold is not the one deployed. |
| Post-deployment monitoring and incident analysis | Harmful uses leave traces in logs that classifiers can detect, and incidents are reported in a comparable form. | When traffic is end-to-end private, when base rates are tiny so false positives swamp detections, and when incident reporting is voluntary and inconsistent across jurisdictions. |
Debates
What the field debates
- Can dangerous-capability thresholds and if-then commitments be set in advance in a way that binds?
- Supporters point to published frontier safety frameworks that tie specific mitigations to capability levels and to precedent in nuclear and aviation safety; the International AI Safety Report counts at least twelve such frameworks. Critics observe that most thresholds are implicit, that developers set and judge their own thresholds, that elicitation is rarely strong enough to show a threshold has not been crossed, and that commitments have been revised when inconvenient. The evidence on both sides is documentary rather than outcome-based.
- Do pre-deployment evaluations predict harm in use?
- The International AI Safety Report (2026) concludes that current tests do not reliably predict real-world risk, and independent evaluations have found gaps against developer-reported scores. The contrary view is that evaluations are not meant to predict harm on their own but to gate capability, that trend lines on standard suites have tracked practical capability, and that the alternative of releasing without tests is worse. The argument turns on what an evaluation is for.
- How much should evaluation methods and results be disclosed?
- Openness advocates argue that unpublished methods cannot be validated and that reproducibility is the foundation of measurement. Those favouring restraint argue that publishing dangerous-capability tasks contaminates future training data and gives uplift in exactly the domains the tests target. Current practice publishes methodology and aggregate results while withholding items in hazardous categories.
Bodies and data
Bodies that set practice, and the data they publish
Bodies and standards are named so learners can find and apply their published work; the Academy teaches their use and links to the publisher, and no affiliation is implied.
| Body | Role in the field |
|---|---|
| AI Security Institute (United Kingdom) | Government evaluator of frontier models; publishes the Inspect evaluation framework, the Frontier AI Trends Report and research on safeguards. |
| Center for AI Standards and Innovation (NIST, United States) | Runs independent evaluations of frontier models and develops measurement science and standards for AI safety and security. |
| International Network of AI Safety Institutes | Coordinates joint testing exercises and shared methods among national institutes. |
| International AI Safety Report | Independent scientific synthesis of the evidence on capabilities, risks and safeguards, with key updates between annual editions. |
| ISO/IEC JTC 1/SC 42 | Standardises AI terminology, risk management (ISO/IEC 23894) and evaluation-related guidance such as ISO/IEC TR 24029 on robustness assessment of neural networks. |
| European AI Office | Specifies evaluation and systemic-risk obligations for general-purpose models under Regulation (EU) 2024/1689 and may commission independent evaluations. |
| Dataset | Steward |
|---|---|
| AI Incident Database | Responsible AI Collaborative |
| OECD AI Incidents Monitor | OECD.AI Policy Observatory |
| Epoch AI benchmarking hub | Epoch AI |
| Holistic Evaluation of Language Models (HELM) | Stanford Center for Research on Foundation Models |
04Mbinu
From one attempt to an attacker's budget: how a per-attempt success rate becomes a release gate
An attack success rate is measured per attempt against a stated attacker budget; the chance that at least one of k attempts succeeds grows as 1 minus (1 minus s) to the k, and a release gate compares that cumulative figure, after defences, with the level a safety case can tolerate.
Sources: Shevlane et al., Model evaluation for extreme risks (2023); Phuong et al., Evaluating Frontier Models for Dangerous Capabilities (2024); Clymer et al., Safety Cases: How to Justify the Safety of Advanced AI Systems (2024); Bengio et al., International AI Safety Report (2025).
05At a glance
At a glance, standards and evidence, research and evidence, what is changing and learning
- Build a twin of any placeTwin builderWorked example: West Cornwall: Goonhilly and the Porthcurno cable landings · 4 generators · 7 layers · 3 instruments
Grafu ya 3D
Kitivo cha Teknolojia na Mifumo ya Kidijitali
295 nodes1,814 relations9 views
06Dashibodi
How AI safety and evaluation is taught, as a process you can run
Press Play to run this process from this page's own data.
Usalama wa AI na tathmini
How AI safety and evaluation is taught
5 subjects in scope, 3 key terms, courses in development, 1 lab, in the Artificial Intelligence and Autonomous Systems department of Technology and Digital Systems: from the scope to the method and its terms, the courses that teach it, the labs on the twin, the assessment and the record.
-
01 Scope
- 01
Evaluation design and…
Start · the discipline
Evaluation design and metrics
- 02
Red-teaming and…
Start · the discipline
Red-teaming and adversarial testing
- 03
Benchmarks and data…
Start · the discipline
Benchmarks and data contamination
- 04
Dangerous capability…
Start · the discipline
Dangerous capability evaluations
- 05
Safety cases for…
Start · the discipline
Safety cases for frontier models
- 01
-
02 Method and terms
- 06
From one attempt to an…
Model · the learner
An attack success rate is measured per attempt against a stated attacker budget; the chance that at least one of k attempts succeeds grows…
- 07
Red-teaming
Record · the discipline
Structured attempts to make a system fail or misbehave, to find weaknesses before others do.
- 09
Benchmark contamination
Record · the discipline
When test data have leaked into a model's training data, so scores overstate its ability.
- 11
Safety case
Record · the discipline
A structured argument, supported by evidence, that a system is acceptably safe for a given use.
- 06
-
04 Labs
- 08
Pacha
Exercise · the learner, on the faculty twin
The method runs on the faculty twin and the admitted map layers.
- 08
-
05 Assess
- 10
Module checks
Gate · the course
Each module ends with a short check against its outcomes.
- 12
Assessed work
Decision · the learner
The assessed work that closes a course: a lab record, a case memo or an assignment, reviewed against the rubric.
- 10
-
06 Record
- 13
Rekodi ya ukamilishaji
Outcome · the learner
The record stacks into the discipline achievement.
- 14
Discipline achievement
Record · the Academy
The discipline achievement follows the discipline rule, read from the records.
- 13
AI safety and evaluation · Artificial Intelligence and Autonomous Systems An outline of how the work is designed to run; it does not report the status of any decision or record.
07Kujifunza
Achievements, where this is practised, practised in and across the ecosystem
Mafanikio
Achievements in this discipline
Course achievements
One for each course with this as its primary discipline, earned when you pass the course's assessed activities.
An achievement is a record of the learning completed and assessed here. How recognition works.
Where this is practised
Where the subject is practised
- The Global Centre for Risk and Innovation (GCRI Canada)Artificial Intelligence and Autonomous SystemsModel evaluation
Field platforms of the ecosystem where the subject is studied and applied.
Practised in
The three Foundry systems where it is practised
- FoundryNXS-NXQ systemFitness of a machine capability for a named task
- NXS-LAB systemEvaluation under transparent test conditions
- NXS-NSF systemAssurance cases and reliance limits
Foundry system pages describe what each system and its rooms are designed to do; the courses teach the methods these systems apply.
Across the ecosystem
Across the ecosystem
The science, platforms and standards this discipline draws on and contributes to across the ecosystem.
Foundations in the STEM Commons
- The Global Centre for Risk and Innovation (GCRI Canada)AI evaluation and assurance engineeringAI evaluation and assurance engineering develops methods to test, monitor and provide justified confidence in AI systems.
- Artificial intelligence and machine learningArtificial intelligence and machine learning develop algorithms that learn patterns from data to classify, predict or generate outputs.
- Uncertainty quantification and model validationUncertainty quantification and model validation characterise how uncertainty in inputs, parameters and model structure affects predictions.
Standards for the field
- Nexus StandardsDigital, Cyber and AIThe subsite that keeps the standards this discipline teaches.
- Open Infrastructure and DevelopersThe subsite that keeps the standards this discipline teaches.
Pages of the ecosystem, cited so learners can follow the field into research, platforms, standards and national practice.
08Shiriki
Take part in Technology and Digital Systems
Anza na wewe ni nani.
Njia
Njia
Taasisi za kitaifa
Wizara, ofisi za takwimu na mashirika: dawati na tawi la kitaifa, na kiti katika vikundi kazi. Sura za kitaifa
Njia
Sekta na taaluma
Operators, regulators, banks, insurers and professional bodies whose work touches technology and digital systems. For organisations
Njia
Kujifunza
Kozi na mazoezi, pamoja na rekodi ya kila kozi iliyokamilishwa. Kusoma ni bure. Ujifunzaji huria
Njia
Viwango
Andaa rasimu, kagua na pitisha mbinu na viwango vya data pamoja na vikundi kazi. Vikundi kazi
Njia
Jumuiya na utafiti
Universities, civil society and individual experts: projects, data and review. Write to the Academy
Miito wazi
| Mwito wazi | Inatafuta | Inafungwa |
|---|---|---|
| Mwito wa uanzishaji kwa wataalamu wa vikundi kazi CALL-FOUNDING-EXPERTS | mtaalamu, mhariri, mwenyekiti | 2026-12-15 |
| Uteuzi wazi kwa viti vitatu vya wataalamu huru vya Baraza la Viwango la Dunia CALL-GSC-IND | mjumbe wa baraza | 2026-12-01 |
| Uteuzi wa wanachama wa kikanda wa muda (unatumika tu ikiwa kifungu cha mpito kitaamuliwa) CALL-GSC-REG | mjumbe wa baraza | 2026-12-15 |
| Ukaguzi wa miongozo ya kiini cha hatari ya wimbi la 1 (C1, C2, C3, C4, C8, C10, C11) CALL-REVIEW-WAVE1 | mchangiaji | 2027-01-31 |
| Wataalamu wa viwango vya msingi vya sayansi ya hatari: kutokuwa na uhakika na upangaji wa ushahidi, vipimo, maamuzi ya wataalamu yaliyopangwa, ustahimilivu, matukio makali na ya mchanganyiko, hatari za maafa makubwa na za kuwepo, maadili na usawa, vipimo na upimaji (NRS-C17 hadi NRS-C24) CALL-CORE-SCIENCE | mtaalamu, mhariri, mwenyekiti | 2027-01-31 |
| Wakaguzi huru wa familia za atlasi ya hatari, watatu kutoka angalau kanda mbili kwa kila familia (NHS2-R19), familia za kipaumbele kwanza CALL-ATLAS-REVIEW | mtaalamu | 2027-03-31 |
| Wajumbe wa Jopo la muda la Ukaguzi wa Usalama wa Mipaka Mipya na wakaguzi wa timu nyekundu (usalama wa kibiolojia na ulinzi wa kibiolojia, usalama na ulinzi wa AI, ulinzi wa miundombinu muhimu); maombi hushughulikiwa kwa usiri CALL-FRONTIER-SAFETY-REVIEW | mtaalamu | 2026-11-30 |
| Wataalamu wa ontolojia, metrolojia, uthibitishaji wa programu na tathmini ya ulinganifu kwa rejista ya mahitaji, uwakilishi wa mashine wa NHS-2, seti za majaribio ya ulinganifu na mpango wa tathmini ya ulinganifu CALL-CONFORMANCE-ONTOLOGY | mtaalamu, mhariri | 2027-01-31 |
Miito iliyopita tarehe ya kufungwa hufichwa kiotomatiki. Kila mwito, pendekezo na njia: maeneo na miito wazi.
Seli na CERT
Kusanya seli
Leta pamoja watu watatu hadi saba wenye ujuzi unaokamilishana: data, mazoezi ya uwandani, sera na mawasiliano.
Isajili
Write to the team with the cell's name, members, area and region. You receive a reference and a contact. Write to the Academy
Chagua hati
Chagua lengo moja: utafiti na ushahidi, data na ramani, ukaguzi wa viwango, au utayari wa jamii.
Jifunzeni pamoja
Fuata njia za pamoja za masomo katika Risk Academy, pamoja na mazoezi na rekodi ya kila kozi iliyokamilishwa. Hatari na ustahimilivu
Ongoza na uripoti
Hold regular sessions. Publish what the cell learns through Reports. Bring findings to the working groups. Working groups
Bobea kama Timu ya Jamii ya Kukabili Dharura (CERT)
A CERT prepares its neighbourhood for the hazards it faces. It builds a local hazard profile from this site's map. It trains in household and street preparedness. It learns first aid and light search through recognised local providers, and it runs drills. A CERT works alongside the local emergency services and follows their direction. It never replaces them and is not an emergency service.
Kielezo
What a national team or a local community can do on The Global Risks Index (NXS-GRX) today.
NXS-GRX
Your country's profile
Wasifu wazi wa hatari kwa nchi 195, pamoja na vyanzo na leseni kwa kila thamani. Fungua Faharasa
NXS-GRX
Sahihisha au jibu
Sahihisha thamani au jibu kwa niaba ya nchi yako. Mabadiliko yanarekodiwa pamoja na athari yake. Fungua Faharasa
NXS-GRX
Toa data
Toa seti ya data. Inaingia kupitia lango la leseni, ikiwa na utajo wako. Fungua Faharasa
NXS-GRX
Tumia data wazi
Tabaka, wasifu na vigae kupitia API wazi. Fungua Faharasa
09References
Core references and where to begin
Start with Bengio, then Shevlane, then Weidinger.
- Shevlane, T., Farquhar, S., Garfinkel, B. et al., Model evaluation for extreme risks (2023). arxiv.org
- Phuong, M., Aitchison, M., Catt, E. et al., Evaluating Frontier Models for Dangerous Capabilities (2024). arxiv.org
- Perez, E., Huang, S., Song, F. et al., Red Teaming Language Models with Language Models (2022). arxiv.org
- Ganguli, D., Lovitt, L., Kernion, J. et al., Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022). arxiv.org
- Wei, A., Haghtalab, N. and Steinhardt, J., Jailbroken: How Does LLM Safety Training Fail? (2023). arxiv.org
- Hendrycks, D., Burns, C., Basart, S. et al., Measuring Massive Multitask Language Understanding (2021). arxiv.org
- Clymer, J., Gabrieli, N., Krueger, D. and Larsen, T., Safety Cases: How to Justify the Safety of Advanced AI Systems (2024). arxiv.org
- Kwa, T., West, B., Becker, J. et al., Measuring AI Ability to Complete Long Tasks (2025). arxiv.org
- Weidinger, L., Rauh, M., Marchal, N. et al., Sociotechnical Safety Evaluation of Generative AI Systems (2023). arxiv.org
- Bengio, Y., Mindermann, S., Privitera, D. et al., International AI Safety Report (2025). arxiv.org
10Maswali
Maswali, majibu na istilahi muhimu
Je, kusoma hapa ni bure?
Ndiyo. Kila kozi na nyenzo ya kujifunzia ya Risk Academy, inayoongozwa na Nexus Consortium, ni bure kusoma, kwa kasi yako mwenyewe au pamoja na kundi. Ushiriki wa kawaida hauna ada. Kazi inayolipwa, mafunzo kazini na ushirika wa kitaaluma ni mipango tofauti.
Kukamilisha kozi kunanipa nini?
Rekodi ya ukamilishaji: kozi, toleo lake, tarehe na kilichotathminiwa. Ni ya faragha kwa chaguo-msingi na huishiriki wewe mwenyewe. Hujumlishwa kuwa mafanikio ya taaluma, idara na programu. Rekodi inaeleza kilichosomwa na kutathminiwa na haithibitishi umahiri; si leseni, cheo cha kitaaluma wala hali ya ajira.
Data na maeneo yanatoka wapi?
Kila kitivo hufundisha kwa kutumia pacha wa mahali halisi uliojengwa kwa data huria. Kila tabaka lina mtoa data, leseni na tarehe yake, na kila matokeo ya Academy yanataja chanzo chake. Ramani na takwimu ni viashiria vya kufundishia, si tathmini ya mahali popote.
How do I take part beyond the courses in aI safety and evaluation?
Unda seli ya umahiri ya watu watatu hadi saba, soma njia ya pamoja, chukua kazi inayosimamiwa katika Vikundi Kazi vya Kitaifa, au leta kundi la wanafunzi kutoka taasisi yako. Kwa mashirika.
Istilahi kuu
- Red-teaming
- Structured attempts to make a system fail or misbehave, to find weaknesses before others do.
- Benchmark contamination
- When test data have leaked into a model's training data, so scores overstate its ability.
- Safety case
- A structured argument, supported by evidence, that a system is acceptably safe for a given use.
- Frontier model
- A general-purpose AI model at or near the most capable level available at a given time.
RipotiChangia
Changia ushahidi kutoka ukurasa huu.
Reports ni chapisho la robo mwaka la Muungano. Unachojua kuhusu mada hii kinaweza kuwa ushahidi, mfano au marekebisho katika toleo lijalo, kikitajwa kwa jina lako na kuchapishwa chini ya CC BY 4.0 pale kinapokuwa kazi yako mwenyewe.
Risk Academy
- EmailGateway GW-ACADEMY Send it from this page Evidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen deposit Deposit it and submit to the community Publish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsRisk Academy See every way to contribute The contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Usitume kamwe data binafsi, maelezo nyeti ya miundombinu wala chochote ambacho huna haki ya kukishiriki. Mchango hupitiwa kabla ya kutumika; kuutuma hakutoi haki za uandishi wala za uhariri, na udhamini haununui matokeo.
Usalama wa AI na tathmini
Prepare with a related course
Open courses in connected disciplines teach the methods this discipline will build on.
Vinjari orodha ya koziBack to Department of Artificial Intelligence and Autonomous Systems