Skip to content

Risk Academy, led by the Nexus Consortium · Technology and Digital Systems · Department of Artificial Intelligence and Autonomous Systems

Evaluation design, red-teaming, benchmarks and safety cases

مصنوعی ذہانت کی حفاظت اور جائزہ

AI safety and evaluation covers how the capabilities, reliability and risks of AI systems are tested before and after deployment.

Map aI safety and evaluation on a real placeSee the courses

Illustrative: layers of units and their weighted connections, an activation front sweeping from input to output

01دائرہ کار

What the discipline covers

The subtopics its courses and labs teach.

کورسز
In preparation
سطحیں
In development
تجربہ گاہیں
In development
  • Evaluation design and metrics
  • Red-teaming and adversarial testing
  • Benchmarks and data contamination
  • Dangerous capability evaluations
  • Safety cases for frontier models
  • Agentic system evaluation
  • Monitoring after deployment
  • Incident reporting and disclosure

Courses in development extend the scope to further subjects: AI system evaluation methods; Frontier AI risk and safety cases.

02نقشے

Build a twin of any place

Search any place or open the world; the layers and generators follow aI safety and evaluation on the planet map, the globe and in 3D.

West Cornwall: Goonhilly and the Porthcurno cable landings: a flat plan of the faculty twin. Terrain 0 to 245 m; 4 satellite dishes; 74 communication masts and towers; 12 telephone exchanges; 7 data centres and cable landing stations; 12 aerodromes; 1 telecom offices; Communication line 161 km; 729 known exploited vulnerabilities since 2023-10-17; Night-time lights, VIIRS day-night band: read live when the 3D twin opens. Contours every 25 metres from the elevation grid; communication cables by class; ground infrastructure as marks.
West Cornwall: Goonhilly and the Porthcurno cable landings, United Kingdom. Flat plan of the same data: relief as contours from the inline grid, the mapped ground infrastructure by kind, the mapped cables and the numbers below. The 3D twin opens on devices that can draw it, once its data and code have been checked.
  • Terrain 0 to 245 m
  • 4 satellite dishes
  • 74 communication masts and towers
  • 12 telephone exchanges
  • 7 data centres and cable landing stations
  • 12 aerodromes
  • 1 telecom offices
  • Communication line 161 km
  • 729 known exploited vulnerabilities since 2023-10-17
  • Night-time lights, VIIRS day-night band: read live when the 3D twin opens
Credits and dates per layer are listed under the stage. Relief in the twin is exaggerated ×1.5 for legibility.

Drag to move, right drag or Shift with a drag to turn and tilt, Ctrl with the wheel to zoom. Keyboard: arrows move, Shift with arrows turn and tilt, + and – zoom, 0 resets, V steps through the views, Escape clears.

Flat plan of the same data: relief as contours from the inline grid, the mapped ground infrastructure by kind, the mapped cables and the numbers below. The 3D twin opens on devices that can draw it, once its data and code have been checked.

The boundaries, names and designations shown do not imply any judgement on the legal status of any territory, or any endorsement or acceptance of such boundaries. Countries: Natural Earth (public domain).

03The subject

The subject, methods and measures and debates and bodies

The discipline

The field and its scope

For evaluators, red teams, safety researchers and the public officers who read their results, who need to design tests of AI systems and know what a result can and cannot support.

AI safety and evaluation is the empirical study of what an AI system can do, how it fails and whether its safeguards hold, before and after deployment. It covers evaluation design and metrics, benchmarks and their contamination, red-teaming and adversarial testing, dangerous-capability evaluations in domains such as cyber operations, biological and chemical knowledge, autonomous replication and persuasion, safety cases that argue a system is acceptably safe, evaluation of agents that act with tools, and the monitoring and incident reporting that follow release. It is distinct from assurance and audit, which take evaluation results as evidence within an organisational and regulatory process.

The field is organised around a small number of frontier developers who run internal evaluations and publish safety frameworks with capability thresholds, government institutes in the United Kingdom, the United States, Japan, the Republic of Korea, Singapore and elsewhere that test models independently, academic groups that build benchmarks and study their validity, and third-party evaluators. Its central difficulty is measurement validity: a score is an estimate on a sample of tasks with a prompt and a scaffold, and the same model scores differently when any of those change. Dangerous capabilities are named here at category level only; the methods that would give uplift are not published.

Measures

Quantities and units

QuantityUnit or scaleWhat to know
Pass rate on an evaluation suitepercent of tasks solved, with a bootstrap confidence intervalDepends on the number of attempts allowed (pass at 1 versus pass at k), the scaffold, tool access and elicitation effort. Report the interval and the elicitation budget together; without them two scores are not comparable.
Attack success ratepercent of adversarial attempts that elicit the prohibited behaviourMeasured for a stated attacker budget (attempts, compute, access level). The second key update of the International AI Safety Report (2025) reports prompt injection succeeding about half the time within ten attempts for current systems; rates fall with defences and rise with adaptive attackers.
Task time horizon at 50 percent successminutes of human expert timeEstimated by logistic fit across tasks of known human duration (Kwa et al., 2025); the doubling time of roughly seven months carries wide uncertainty and the task set is software-heavy.
Contamination ratepercent of benchmark items found in training dataEstimated by n-gram overlap or canary strings; underestimates paraphrased leakage and cannot be computed at all when training data are undisclosed.
Inter-rater agreement on graded outputsCohen's or Fleiss' kappa, minus 1 to 1Bounds how much of a human-graded safety score is signal. Kappa below 0.6 on a harm rubric means the rubric, not the model, dominates the result.
Incidents reported after deploymentcount per period, by severity classThe AI Index (2026) counts 362 documented incidents in 2025 against 233 in 2024 from the AI Incident Database; counts reflect reporting effort and media attention as much as underlying harm.

طریقے

Methods and their limits

Each method is listed with what it assumes and where it fails.

طریقہAssumesFails when
Static benchmarks with automatic scoringThe items represent the capability of interest, the answer key is correct and the items were not seen in training.When items leak into training corpora, when scoring rewards format over substance, when the benchmark saturates, and when the capability of interest is open-ended and has no answer key.
Dangerous-capability evaluationsCapability can be elicited with a bounded effort, task performance in a proxy setting predicts uplift in the real setting, and thresholds can be set before the capability appears.When elicitation is under-resourced, so absence of evidence is read as evidence of absence; when a model sandbags; and when a proxy task is far from the real pathway, which for the gravest categories it must be, for information-hazard reasons.
Red teaming, manual and automatedRed teamers cover the attack space well enough that unfound failures are rare, and automated attackers generalise to human ingenuity.When coverage is unmeasured, when the red team lacks domain expertise, and when the deployed system differs from the tested one in prompts, tools or retrieval. The AI Security Institute reported finding universal jailbreaks in every system tested (2025).
Safety casesA structured argument can connect evidence (evaluations, controls, monitoring) to a top-level claim of acceptable risk, and reviewers can judge the argument's soundness.When the claim is vague, the evidence is incomplete for the deployment context, or the argument relies on inability claims that a stronger elicitation would overturn; the discipline inherits these problems from safety cases in aviation and nuclear practice.
Agent evaluation in sandboxed environmentsThe environment reproduces the tools, latencies and adversaries of real deployment, and success criteria can be checked automatically.When the sandbox is cleaner than the world, when agents exploit environment bugs, when long tasks are too few to estimate reliability, and when the evaluated agent scaffold is not the one deployed.
Post-deployment monitoring and incident analysisHarmful uses leave traces in logs that classifiers can detect, and incidents are reported in a comparable form.When traffic is end-to-end private, when base rates are tiny so false positives swamp detections, and when incident reporting is voluntary and inconsistent across jurisdictions.

Debates

What the field debates

Can dangerous-capability thresholds and if-then commitments be set in advance in a way that binds?
Supporters point to published frontier safety frameworks that tie specific mitigations to capability levels and to precedent in nuclear and aviation safety; the International AI Safety Report counts at least twelve such frameworks. Critics observe that most thresholds are implicit, that developers set and judge their own thresholds, that elicitation is rarely strong enough to show a threshold has not been crossed, and that commitments have been revised when inconvenient. The evidence on both sides is documentary rather than outcome-based.
Do pre-deployment evaluations predict harm in use?
The International AI Safety Report (2026) concludes that current tests do not reliably predict real-world risk, and independent evaluations have found gaps against developer-reported scores. The contrary view is that evaluations are not meant to predict harm on their own but to gate capability, that trend lines on standard suites have tracked practical capability, and that the alternative of releasing without tests is worse. The argument turns on what an evaluation is for.
How much should evaluation methods and results be disclosed?
Openness advocates argue that unpublished methods cannot be validated and that reproducibility is the foundation of measurement. Those favouring restraint argue that publishing dangerous-capability tasks contaminates future training data and gives uplift in exactly the domains the tests target. Current practice publishes methodology and aggregate results while withholding items in hazardous categories.

Bodies and data

Bodies that set practice, and the data they publish

Bodies and standards are named so learners can find and apply their published work; the Academy teaches their use and links to the publisher, and no affiliation is implied.

BodyRole in the field
AI Security Institute (United Kingdom)Government evaluator of frontier models; publishes the Inspect evaluation framework, the Frontier AI Trends Report and research on safeguards.
Center for AI Standards and Innovation (NIST, United States)Runs independent evaluations of frontier models and develops measurement science and standards for AI safety and security.
International Network of AI Safety InstitutesCoordinates joint testing exercises and shared methods among national institutes.
International AI Safety ReportIndependent scientific synthesis of the evidence on capabilities, risks and safeguards, with key updates between annual editions.
ISO/IEC JTC 1/SC 42Standardises AI terminology, risk management (ISO/IEC 23894) and evaluation-related guidance such as ISO/IEC TR 24029 on robustness assessment of neural networks.
European AI OfficeSpecifies evaluation and systemic-risk obligations for general-purpose models under Regulation (EU) 2024/1689 and may commission independent evaluations.
DatasetSteward
AI Incident DatabaseResponsible AI Collaborative
OECD AI Incidents MonitorOECD.AI Policy Observatory
Epoch AI benchmarking hubEpoch AI
Holistic Evaluation of Language Models (HELM)Stanford Center for Research on Foundation Models

04طریقہ

From one attempt to an attacker's budget: how a per-attempt success rate becomes a release gate

An attack success rate is measured per attempt against a stated attacker budget; the chance that at least one of k attempts succeeds grows as 1 minus (1 minus s) to the k, and a release gate compares that cumulative figure, after defences, with the level a safety case can tolerate.

Sources: Shevlane et al., Model evaluation for extreme risks (2023); Phuong et al., Evaluating Frontier Models for Dangerous Capabilities (2024); Clymer et al., Safety Cases: How to Justify the Safety of Advanced AI Systems (2024); Bengio et al., International AI Safety Report (2025).

05At a glance

At a glance, standards and evidence, research and evidence, what is changing and learning

3D گراف

فیکلٹی برائے ٹیکنالوجی اور ڈیجیٹل نظام

295 nodes1,814 relations9 views

Explore the 3D graph: Faculty of Technology and Digital Systems

06کنسول

How AI safety and evaluation is taught, as a process you can run

Press Play to run this process from this page's own data.

مصنوعی ذہانت کی حفاظت اور جائزہ

How AI safety and evaluation is taught

5 subjects in scope, 3 key terms, courses in development, 1 lab, in the Artificial Intelligence and Autonomous Systems department of Technology and Digital Systems: from the scope to the method and its terms, the courses that teach it, the labs on the twin, the assessment and the record.

  1. 01 Scope

    1. 01

      Evaluation design and…

      Start · the discipline

      Evaluation design and metrics

    2. 02

      Red-teaming and…

      Start · the discipline

      Red-teaming and adversarial testing

    3. 03

      Benchmarks and data…

      Start · the discipline

      Benchmarks and data contamination

    4. 04

      Dangerous capability…

      Start · the discipline

      Dangerous capability evaluations

    5. 05

      Safety cases for…

      Start · the discipline

      Safety cases for frontier models

  2. 02 Method and terms

    1. 06

      From one attempt to an…

      Model · the learner

      An attack success rate is measured per attempt against a stated attacker budget; the chance that at least one of k attempts succeeds grows…

    2. 07

      Red-teaming

      Record · the discipline

      Structured attempts to make a system fail or misbehave, to find weaknesses before others do.

    3. 09

      Benchmark contamination

      Record · the discipline

      When test data have leaked into a model's training data, so scores overstate its ability.

    4. 11

      Safety case

      Record · the discipline

      A structured argument, supported by evidence, that a system is acceptably safe for a given use.

  3. 04 Labs

    1. 08

      جڑواں ماڈل

      Exercise · the learner, on the faculty twin

      The method runs on the faculty twin and the admitted map layers.

  4. 05 Assess

    1. 10

      Module checks

      Gate · the course

      Each module ends with a short check against its outcomes.

    2. 12

      Assessed work

      Decision · the learner

      The assessed work that closes a course: a lab record, a case memo or an assignment, reviewed against the rubric.

  5. 06 Record

    1. 13

      تکمیل کا ریکارڈ

      Outcome · the learner

      The record stacks into the discipline achievement.

    2. 14

      Discipline achievement

      Record · the Academy

      The discipline achievement follows the discipline rule, read from the records.

AI safety and evaluation · Artificial Intelligence and Autonomous Systems An outline of how the work is designed to run; it does not report the status of any decision or record.

07تعلیم

Achievements, where this is practised, practised in and across the ecosystem

Achievements

Achievements in this discipline

  1. Course achievements

    One for each course with this as its primary discipline, earned when you pass the course's assessed activities.

An achievement is a record of the learning completed and assessed here. How recognition works.

Where this is practised

Where the subject is practised

Field platforms of the ecosystem where the subject is studied and applied.

Practised in

The three Foundry systems where it is practised

Foundry system pages describe what each system and its rooms are designed to do; the courses teach the methods these systems apply.

Across the ecosystem

Across the ecosystem

The science, platforms and standards this discipline draws on and contributes to across the ecosystem.

Foundations in the STEM Commons

Standards for the field

Pages of the ecosystem, cited so learners can follow the field into research, platforms, standards and national practice.

08حصہ لیں

Take part in Technology and Digital Systems

اپنی شناخت سے شروع کریں۔

Routes

  • راستہ

    قومی ادارے

    Ministries, statistics offices and agencies: a national desk and chapter, and a seat in the working groups. National chapters

  • راستہ

    Sectors and professions

    Operators, regulators, banks, insurers and professional bodies whose work touches technology and digital systems. For organisations

  • راستہ

    تعلیم

    Courses and practice, with a record of every course completed. Free to study. Open learning

  • راستہ

    معیارات

    Draft, review and adopt the methods and data standards with the working groups. Working groups

  • راستہ

    Communities and research

    Universities, civil society and individual experts: projects, data and review. Write to the Academy

Open calls

Open callSeeksCloses
Founding call for experts to the working groups CALL-FOUNDING-EXPERTSexpert, editor, chair2026-12-15
Open nomination for the three independent expert seats of the Global Standards Council CALL-GSC-INDکونسل کا رکن2026-12-01
Nominations of interim regional members (applies only if the transitional provision is decided) CALL-GSC-REGکونسل کا رکن2026-12-15
Review of the wave-1 risk core outlines (C1, C2, C3, C4, C8, C10, C11) CALL-REVIEW-WAVE1contributor2027-01-31
Experts for the core standards of risk science: uncertainty and evidence grading, scales, structured expert judgement, tolerability, extremes and compound events, catastrophic and existential risk, ethics and equity, units and measurement (NRS-C17 to NRS-C24) CALL-CORE-SCIENCEexpert, editor, chair2027-01-31
Independent reviewers for the families of the hazard atlas, three from at least two regions for each family (NHS2-R19), priority families first CALL-ATLAS-REVIEWماہر2027-03-31
Members of the interim Frontier Safety Review Panel and red-team reviewers (biosafety and biosecurity, AI safety and security, critical-infrastructure security); applications treated in confidence CALL-FRONTIER-SAFETY-REVIEWماہر2026-11-30
Experts in ontology, metrology, software verification and conformity assessment for the requirement register, the NHS-2 machine representation, the conformance test suites and the conformity assessment scheme CALL-CONFORMANCE-ONTOLOGYexpert, editor2027-01-31

Calls past their closing date are hidden automatically. Every call, proposal and route: open areas and calls.

Cells and CERTs

  1. Gather the cell

    Bring together three to seven people with complementary skills: data, field practice, policy and communication.

  2. Register it

    Write to the team with the cell's name, members, area and region. You receive a reference and a contact. Write to the Academy

  3. Choose a charter

    Pick one focus: research and evidence, data and maps, standards review, or community preparedness.

  4. Learn together

    Follow shared study paths at Risk Academy, with practice and a record of every course completed. Hazards and resilience

  5. Lead and report

    Hold regular sessions. Publish what the cell learns through Reports. Bring findings to the working groups. Working groups

  • Specialise as a Community Emergency Response Team (CERT)

    A CERT prepares its neighbourhood for the hazards it faces. It builds a local hazard profile from this site's map. It trains in household and street preparedness. It learns first aid and light search through recognised local providers, and it runs drills. A CERT works alongside the local emergency services and follows their direction. It never replaces them and is not an emergency service.

انڈیکس

What a national team or a local community can do on The Global Risks Index (NXS-GRX) today.

  • NXS-GRX

    Your country's profile

    Open risk profiles for 195 countries, with sources and licences on every value. Open the Index

  • NXS-GRX

    Correct or reply

    Correct a value or reply for your country. The change is logged with its effect. Open the Index

  • NXS-GRX

    Offer data

    Offer a dataset. It enters through the licence gate, with your credit. Open the Index

  • NXS-GRX

    Use the open data

    Layers, profiles and tiles through the open API. Open the Index

09References

Core references and where to begin

Start with Bengio, then Shevlane, then Weidinger.

  1. Shevlane, T., Farquhar, S., Garfinkel, B. et al., Model evaluation for extreme risks (2023). arxiv.org
  2. Phuong, M., Aitchison, M., Catt, E. et al., Evaluating Frontier Models for Dangerous Capabilities (2024). arxiv.org
  3. Perez, E., Huang, S., Song, F. et al., Red Teaming Language Models with Language Models (2022). arxiv.org
  4. Ganguli, D., Lovitt, L., Kernion, J. et al., Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022). arxiv.org
  5. Wei, A., Haghtalab, N. and Steinhardt, J., Jailbroken: How Does LLM Safety Training Fail? (2023). arxiv.org
  6. Hendrycks, D., Burns, C., Basart, S. et al., Measuring Massive Multitask Language Understanding (2021). arxiv.org
  7. Clymer, J., Gabrieli, N., Krueger, D. and Larsen, T., Safety Cases: How to Justify the Safety of Advanced AI Systems (2024). arxiv.org
  8. Kwa, T., West, B., Becker, J. et al., Measuring AI Ability to Complete Long Tasks (2025). arxiv.org
  9. Weidinger, L., Rauh, M., Marchal, N. et al., Sociotechnical Safety Evaluation of Generative AI Systems (2023). arxiv.org
  10. Bengio, Y., Mindermann, S., Privitera, D. et al., International AI Safety Report (2025). arxiv.org

10سوالات

سوالات، جوابات اور کلیدی اصطلاحات

کیا یہاں تعلیم مفت ہے؟

جی ہاں۔ Risk Academy، جس کی قیادت Nexus Consortium کرتا ہے، کا ہر کورس اور تعلیمی مواد مفت ہے، اپنی رفتار سے یا کسی گروپ کے ساتھ۔ عام شرکت کی کوئی فیس نہیں؛ معاوضے والا کام، انٹرن شپس اور فیلوشپس الگ انتظامات ہیں۔

کورس مکمل کرنے سے مجھے کیا ملتا ہے؟

A completion record: the course, its version, the date and what was assessed, private by default and shared by you. It stacks into discipline, department and programme achievements. A record states what was studied and assessed and does not certify competence; it is not a licence, a professional title or an employment status.

ڈیٹا اور مقامات کہاں سے آتے ہیں؟

Every faculty teaches on a twin of a real place built from open data; each layer carries its provider, licence and date, and every result on the Academy states its source. Maps and figures are indicators for teaching, never an assessment of a place.

How do I take part beyond the courses in aI safety and evaluation?

Form a competence cell of three to seven people, study a shared pathway, take a supervised assignment in the National Working Groups, or bring a cohort from your institution. For organisations.

کلیدی اصطلاحات

Red-teaming
Structured attempts to make a system fail or misbehave, to find weaknesses before others do.
Benchmark contamination
When test data have leaked into a model's training data, so scores overstate its ability.
Safety case
A structured argument, supported by evidence, that a system is acceptably safe for a given use.
Frontier model
A general-purpose AI model at or near the most capable level available at a given time.

رپورٹیںتعاون کریں

اس صفحے سے شواہد فراہم کریں۔

Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

Risk Academy

Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.

مصنوعی ذہانت کی حفاظت اور جائزہ

Prepare with a related course

Open courses in connected disciplines teach the methods this discipline will build on.

کورس کیٹلاگ دیکھیںBack to Department of Artificial Intelligence and Autonomous Systems