AI safety and evaluation: foundations and practice
- Level
- L2 Intermediate
- Study time
- about 24 hours (planning estimate)
- অবস্থা
- In authoring
- Prerequisites
- An L1 course in the area, or an entry review
Shows why independent and held-out benchmarks matter when relying on vendor-reported evaluation results.
National Institute of Standards and Technology, CAISI Evaluation of DeepSeek V4 Pro,
Learning outcomes
- Define red-teaming and the other key terms of the discipline and use them correctly in a written case
- Explain evaluation design and metrics and red-teaming and adversarial testing with a worked example and its source
- Apply the discipline's method to a documented case and explain each result with its source
- Analyse a named case against benchmarks and data contamination, stating evidence, date and limits
Modules
-
Evaluation design and metrics
- Evaluation design and metrics
- Red-teaming and adversarial testing
- Benchmarks and data contamination
-
Dangerous capability evaluations
- Dangerous capability evaluations
- Safety cases for frontier models
- Agentic system evaluation
-
Monitoring after deployment
- Monitoring after deployment
- Incident reporting and disclosure
-
Lab: the discipline's method
- An attack success rate is measured per attempt against a stated attacker budget; the chance that at least one of k attempts succeeds grows as 1 minus (1 minus s) to the k, and a release gate compares that cumulative figure, after defences, with the level a safety case can tolerate.
- Work the method through a documented case and explain each result
- Read the method's sources and stated values
Labs with Academy tools
পদ্ধতি
- From one attempt to an attacker's budget: how a per-attempt success rate becomes a release gateThe discipline's method on the discipline page, with its cited sources, worked through a documented case
উপকরণ
- Orbit propagatorA faculty instrument on the faculty page, run on the faculty's digital twin data
- Evaluation ledgerA faculty instrument on the faculty page, run on the faculty's digital twin data
Twin builder
- Digital twin builderBuild the faculty's place in 3D, switch layers and read the credits and dates of each
Map layer
- Built-up and settlement extent, 2010 (HBASE, Landsat)An open map layer the Academy's licence register admits, read in the map tools
- Night lights tonight, outages (VIIRS Day/Night Band)An open map layer the Academy's licence register admits, read in the map tools
- Population density, 2020 (GPW v4.11)An open map layer the Academy's licence register admits, read in the map tools
Foundry room
- Competence room, NXS-NXQWhether a model is fit for a task, population and operating envelope
- Benchmarks room, NXS-LABEvaluation and benchmark design
Nexus Standards
Teaches
- NRS-T1কৃত্রিম বুদ্ধিমত্তার ঝুঁকি
- NRS-T16এআই এজেন্ট ও স্বয়ংক্রিয় সিদ্ধান্ত ব্যবস্থার ঝুঁকি
- NRS-A9ঝুঁকির কাজে এআইয়ের মূল্যায়ন সংকলন
- NRS-H95Hazard family: AI system failure and misalignment
- NRS-H96Hazard family: AI security and model supply chain
- NRS-H97Hazard family: AI misuse and capability uplift
- NRS-H98Hazard family: Agentic AI acting at scale
- NRS-M19পদ্ধতির বুনন: সাইবার, এআই ও অগ্রসর প্রযুক্তি
Relies on 8
- NS-1The UNI and the artefact catalogue
- NS-16UNI application profile (how articles, documents, code, data, APIs and interfaces carry the UNI)
- NRS-A1কোড হিসেবে মানদণ্ড: যন্ত্রপাঠ্য মানদণ্ড, প্রয়োজনীয়তা ও পরীক্ষা
- NRS-C1ঝুঁকির শব্দভান্ডার ও অন্টোলজি
- NRS-C2ঝুঁকি মূল্যায়ন পদ্ধতি
- NRS-C3বিপদ, সংস্পর্শ, ঝুঁকিপ্রবণতা ও সক্ষমতার ডেটা
- NRS-C4ঝুঁকির ডেটা মডেল, শনাক্তকারী ও বিনিময়
- NRS-C7আগাম সতর্কতা, ট্রিগার ও আগাম পদক্ষেপ
Assessment
- 40%
Lab record
Work the discipline's method through a documented case and explain each result with its source and date
- 40%
Case memo
Apply the discipline's methods to a named place or organisation, stating evidence and limits
- 20%
Key terms check
Short answers on the key terms, marked against the definitions
Delivery
- Self-paced online, with the labs in the browser
- Cohort study over six to eight weeks, with a weekly lab review
- Co-delivered with a partner institution, inside its own courses
Partner on this course
- Co-teach a module and bring a current case from your work
- Host a cohort for your learners, with your own schedule and review
- Contribute a lab: a dataset your institution publishes openly, or a model
- Review the course content and its references before it opens
- Translate the course and adapt its cases to your region
How institutions and experts take part
In the Academy LMS
- Course on the learning platform
- Intermediate, 4 sections
- Cohort group
- A community group for each cohort of learners
- Achievement
- Recorded on completion of the assessed work
References 4
- National Institute of Standards and Technology, CAISI Evaluation of DeepSeek V4 Pro,
- Shevlane et al., Model evaluation for extreme risks, 2023
- Phuong et al., Evaluating Frontier Models for Dangerous Capabilities, 2024
- Clymer et al., Safety Cases: How to Justify the Safety of Advanced AI Systems, 2024