Skip to content

Risk Academy, led by the Nexus Consortium · Technology and Digital Systems · Department of Artificial Intelligence and Autonomous Systems

Foundation models, agentic systems, failure modes and robustness

Ubukrelekrele bokuzenzela nokufunda koomatshini

Artificial intelligence and machine learning covers how models are built from data and training pipelines, how they are deployed in science and public services, and the reliability, security and societal effects that follow.

Map artificial intelligence and machine learning on a real placeOpen RAT-204

Illustrative: layers of units and their weighted connections, an activation front sweeping from input to output

01Ubungakanani bomsebenzi

What the discipline covers

The subtopics its courses and labs teach.

Izifundo
In preparation
Amanqanaba
In development
Iilabhoratri
GeoAI lab, Network lab, Geo quiz
  • Machine learning for risk professionals
  • Foundation models and generative AI
  • Data and training pipelines
  • AI failure modes and robustness
  • Agentic systems
  • AI security
  • AI in public services
  • AI in science
  • Labour and social effects of AI

Courses in development extend the scope to one further subject: AI literacy for risk professionals.

02Iimephu

Build a twin of any place

Search any place or open the world; the layers and generators follow artificial intelligence and machine learning on the planet map, the globe and in 3D.

West Cornwall: Goonhilly and the Porthcurno cable landings: a flat plan of the faculty twin. Terrain 0 to 245 m; 4 satellite dishes; 74 communication masts and towers; 12 telephone exchanges; 7 data centres and cable landing stations; 12 aerodromes; 1 telecom offices; Communication line 161 km; 729 known exploited vulnerabilities since 2023-10-17; Night-time lights, VIIRS day-night band: read live when the 3D twin opens. Contours every 25 metres from the elevation grid; communication cables by class; ground infrastructure as marks.
West Cornwall: Goonhilly and the Porthcurno cable landings, United Kingdom. Flat plan of the same data: relief as contours from the inline grid, the mapped ground infrastructure by kind, the mapped cables and the numbers below. The 3D twin opens on devices that can draw it, once its data and code have been checked.
  • Terrain 0 to 245 m
  • 4 satellite dishes
  • 74 communication masts and towers
  • 12 telephone exchanges
  • 7 data centres and cable landing stations
  • 12 aerodromes
  • 1 telecom offices
  • Communication line 161 km
  • 729 known exploited vulnerabilities since 2023-10-17
  • Night-time lights, VIIRS day-night band: read live when the 3D twin opens
Credits and dates per layer are listed under the stage. Relief in the twin is exaggerated ×1.5 for legibility.

Drag to move, right drag or Shift with a drag to turn and tilt, Ctrl with the wheel to zoom. Keyboard: arrows move, Shift with arrows turn and tilt, + and – zoom, 0 resets, V steps through the views, Escape clears.

Flat plan of the same data: relief as contours from the inline grid, the mapped ground infrastructure by kind, the mapped cables and the numbers below. The 3D twin opens on devices that can draw it, once its data and code have been checked.

The boundaries, names and designations shown do not imply any judgement on the legal status of any territory, or any endorsement or acceptance of such boundaries. Countries: Natural Earth (public domain).

03The subject

The subject, methods and measures and debates and bodies

The discipline

The field and its scope

For analysts, engineers and public servants who must judge where a machine learning system can be relied upon, where it fails, and what its deployment changes for the people and institutions around it.

Artificial intelligence and machine learning study systems that learn statistical structure from data and use it to classify, predict, generate or act. The field now centres on foundation models: large neural networks trained once on broad corpora and adapted to many tasks, including language models that write and reason, multimodal models that read images and audio, and agentic systems that call tools, browse and execute code over long horizons. It is practised by research laboratories, a small number of model developers with access to large compute, and a far larger population of deployers who build products and public services on top of models they did not train.

For risk work the subject is organised around three questions. How capable is a given system, measured on tasks that resemble its use. How reliable is it, given that the same model can hallucinate, follow injected instructions, drift when the world changes or fail silently on groups under-represented in training data. And what does deployment change: in labour markets, in the integrity of information, in scientific practice where models now predict protein structures and weather, and in security where the same capabilities serve attackers and defenders. The chapter treats evaluation, assurance and governance as neighbouring disciplines and keeps to the systems themselves.

Measures

Quantities and units

QuantityUnit or scaleWhat to know
Benchmark accuracy or pass ratepercent of items correct, with a 95 percent confidence intervalA sample estimate on a fixed item set: a model scoring 82 percent on 500 items carries an interval of roughly plus or minus 3.4 points, and contamination of the item set by training data biases it upwards. Comparable between models only on the same items, prompts and scoring rules.
Training computefloating-point operations (FLOP), reported in powers of tenEstimated from parameter count, tokens and hardware utilisation, so published figures carry about a factor-of-two uncertainty. Used as a proxy for capability and as a regulatory threshold, though compute buys different capability in different years.
Calibration errorexpected calibration error, 0 to 1The gap between stated confidence and observed accuracy, binned over confidence levels. Sensitive to the binning scheme and to the distribution of the test set; a model well calibrated on one domain is often over-confident on another.
Hallucination or unsupported-claim ratepercent of responses, graded by annotators or a judge modelDepends on the task, the grading rubric and inter-annotator agreement, which should be reported as a kappa statistic. Judge models share failure modes with the models they grade, so a human-graded subset is needed to bound the error.
Task time horizonminutes of human-equivalent work completed at 50 percent reliabilityA measure of agentic capability from task suites with known human completion times; its confidence interval widens sharply at long horizons because few long tasks exist, and the suite is dominated by software tasks.
Inference cost and latencyUSD per million tokens; seconds to first token and per tokenOperational quantities that set which deployments are feasible. Prices move quickly and are quoted without the energy or water used, which depend on data centre location and hardware.

Iindlela zokusebenza

Methods and their limits

Each method is listed with what it assumes and where it fails.

Indlela yokusebenzaAssumesFails when
Supervised and self-supervised learning with held-out evaluationTraining and deployment data are drawn from the same distribution, the labels are correct and the held-out set is independent of training.Under distribution shift, label noise, data leakage between splits, and whenever the deployment population differs from the training population, which is the usual case in public services.
Scaling-law analysisLoss falls as a smooth power law in compute, data and parameters, and downstream capability follows loss.At predicting specific downstream abilities, which can appear abruptly or not at all; when data of adequate quality runs out; and for post-training methods whose returns do not follow the pre-training curves.
Reinforcement learning from human feedback and related alignment trainingHuman raters can judge response quality, the reward model generalises, and the preferences being optimised are the ones intended.When raters reward confident or sycophantic answers, when the policy exploits reward-model errors, and when a jailbreak or injected instruction reaches a model whose training only reduced, rather than removed, the behaviour.
Adversarial robustness testingAn attacker's perturbations can be characterised by a threat model, such as a bounded change to an input or a text prefix.When the real attacker is not bounded in that way; results from one threat model transfer poorly, and defences that look sound against known attacks are often broken by adaptive ones within months.
Model and system cardsDevelopers report training data, intended use, evaluation results and known limitations truthfully and in enough detail to be checked.When the card describes a model that has since been updated, omits evaluations that were unfavourable, or when deployers compose the model with retrieval, tools and prompts the card never covered.
Monitoring in deploymentLogged inputs, outputs and user feedback are representative and that drift can be detected before harm accumulates.When feedback is sparse or biased towards complainers, when privacy rules prevent logging, and when harms occur outside the system, such as a decision taken on an unverified summary.

Debates

What the field debates

Should frontier model weights be released openly or held behind an interface?
One position holds that open weights let researchers find failures, reduce dependence on a few providers and allow local control of data; it rests on the history of open-source security and on the large volume of safety research performed on open models. The other holds that released weights cannot be withdrawn and that safety training is removable by fine-tuning at low cost, so each release should be judged against the marginal uplift it offers a misuser; it rests on fine-tuning studies and on the absence of a recall mechanism. Both sides accept that the gap between open and closed models has narrowed.
Does performance on benchmarks reflect capability in use?
Sceptics point to contamination, to models that pass examinations and fail simple variants, and to the finding of the International AI Safety Report that pre-deployment tests predict real-world behaviour poorly. Defenders note that aggregate benchmark trends have tracked practical capability over years, that held-out and private test sets reduce contamination, and that no better instrument exists at scale. The disagreement is over how much weight a single score should carry in a decision.
Are labour effects of AI mainly substitution or augmentation?
Field studies of customer support and software work found productivity gains concentrated among less experienced workers, supporting augmentation. Task-exposure studies find a large share of occupations with some tasks exposed, and firm-level evidence of hiring slowdowns in exposed roles supports substitution in specific segments. The evidence is young, mostly from high-income countries, and sensitive to how a task is defined.

Bodies and data

Bodies that set practice, and the data they publish

Bodies and standards are named so learners can find and apply their published work; the Academy teaches their use and links to the publisher, and no affiliation is implied.

BodyRole in the field
National Institute of Standards and Technology (NIST)Publishes the AI Risk Management Framework (AI 100-1) and the Generative AI Profile (AI 600-1), and houses the Center for AI Standards and Innovation, which runs independent model evaluations.
ISO/IEC JTC 1/SC 42The international committee for AI standards: ISO/IEC 22989 (concepts and terminology), 23053 (machine learning framework), 23894 (AI risk management) and 42001 (AI management systems).
OECD AI Policy Observatory (OECD.AI)Keeps the OECD AI Principles, a database of national AI policies and the AI Incidents Monitor used for cross-country comparison.
European Commission AI OfficeOversees general-purpose AI obligations under Regulation (EU) 2024/1689 and the General-Purpose AI Code of Practice.
International AI Safety Report secretariatCoordinates the annual scientific synthesis on capabilities, risks and safeguards of general-purpose AI, written by independent experts with an advisory panel nominated by governments and international organisations.
Partnership on AIA multi-stakeholder body that publishes guidance on safe foundation model deployment and synthetic media, and launched the AI Incident Database now run by the Responsible AI Collaborative.
DatasetSteward
OECD AI Incidents MonitorOECD.AI Policy Observatory
AI Incident DatabaseResponsible AI Collaborative
Epoch AI data on models, compute and hardwareEpoch AI
Holistic Evaluation of Language Models (HELM)Stanford Center for Research on Foundation Models
AI Index annual dataStanford Institute for Human-Centered Artificial Intelligence

04Indlela

From a benchmark score to a deployment call: what a pass rate can and cannot say

A benchmark score is a sample estimate on a fixed item set: the item count sets its sampling interval, contamination of the items by training data inflates it, and only the corrected lower bound can be compared with the threshold a deployment requires.

Sources: Hendrycks et al., Measuring Massive Multitask Language Understanding (2021); Mitchell et al., Model Cards for Model Reporting (2019); Bommasani et al., On the Opportunities and Risks of Foundation Models (2021); NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (2023).

05At a glance

At a glance, with 5 related topics

Igrafu ye-3D

IFakhalthi yeTekhnoloji neeNkqubo zeDijithali

295 nodes1,814 relations9 views

Explore the 3D graph: Faculty of Technology and Digital Systems

06Iilabhoratri

How the subject is practised in labs

LabWhat you do in itCourses using it
GeoAI labApply Earth observation and machine learning to open imagery2
Network labBuild dependency graphs, find critical nodes, trace cascades1
Geo quizCheck spatial reasoning on where hazards and systems meet1

07Ikhonsoli

How Artificial intelligence and machine learning is taught, as a process you can run

Press Play to run this process from this page's own data.

Ubukrelekrele bokuzenzela nokufunda koomatshini

How Artificial intelligence and machine learning is taught

5 subjects in scope, 3 key terms, courses in development, 1 lab, in the Artificial Intelligence and Autonomous Systems department of Technology and Digital Systems: from the scope to the method and its terms, the courses that teach it, the labs on the twin, the assessment and the record.

  1. 01 Scope

    1. 01

      Machine learning for…

      Start · the discipline

      Machine learning for risk professionals

    2. 02

      Foundation models and…

      Start · the discipline

      Foundation models and generative AI

    3. 03

      Data and training…

      Start · the discipline

      Data and training pipelines

    4. 04

      AI failure modes and…

      Start · the discipline

      AI failure modes and robustness

    5. 05

      Agentic systems

      Start · the discipline

      Agentic systems

  2. 02 Method and terms

    1. 06

      From a benchmark score…

      Model · the learner

      A benchmark score is a sample estimate on a fixed item set: the item count sets its sampling interval, contamination of the items by…

    2. 07

      Model evaluation

      Record · the discipline

      Testing an AI model's capabilities, reliability and risks against defined tasks and criteria.

    3. 09

      Distribution shift

      Record · the discipline

      A change between the data a model was trained on and the data it meets in use.

    4. 11

      Hallucination

      Record · the discipline

      Output from a generative model that is fluent but false or unsupported.

  3. 04 Labs

    1. 08

      Iwele

      Exercise · the learner, on the faculty twin

      The method runs on the faculty twin and the admitted map layers.

  4. 05 Assess

    1. 10

      Module checks

      Gate · the course

      Each module ends with a short check against its outcomes.

    2. 12

      Assessed work

      Decision · the learner

      The assessed work that closes a course: a lab record, a case memo or an assignment, reviewed against the rubric.

  5. 06 Record

    1. 13

      Irekhodi yokugqiba

      Outcome · the learner

      The record stacks into the discipline achievement.

    2. 14

      Discipline achievement

      Record · the Academy

      The discipline achievement follows the discipline rule, read from the records.

Artificial intelligence and machine learning · Artificial Intelligence and Autonomous Systems An outline of how the work is designed to run; it does not report the status of any decision or record.

08Ukufunda

Learning, practice and connections

Achievements

Achievements in this discipline

  1. Course achievements

    One for each course with this as its primary discipline, earned when you pass the course's assessed activities.

  2. Discipline: Artificial intelligence and machine learning

    Complete 3 assessed courses in artificial intelligence and machine learning, at least 2 with it as their primary discipline and at least 1 at Intermediate (L2) or above, totalling at least 45 measured learning hours. In development.

  3. Programme: Data and GeoAI specialist

    Complete every core course of the Data and GeoAI specialist under assessed completion, ending with its capstone. In development.

  4. Cross-faculty: AI, information integrity and governance

    Courses in connected disciplines and a passed integrative task. In development.

An achievement is a record of the learning completed and assessed here. How recognition works.

Iindlela

Pathways that teach it

Where this is practised

Where the subject is practised

Field platforms of the ecosystem where the subject is studied and applied.

Practised in

The three Foundry systems where it is practised

Foundry system pages describe what each system and its rooms are designed to do; the courses teach the methods these systems apply.

Connected disciplines

Disciplines it depends on

Across the ecosystem

Across the ecosystem

The science, platforms and standards this discipline draws on and contributes to across the ecosystem.

Foundations in the STEM Commons

Standards for the field

Pages of the ecosystem, cited so learners can follow the field into research, platforms, standards and national practice.

09Thatha inxaxheba

Take part in Technology and Digital Systems

Qala ngokuba ungubani.

Routes

  • Indlela

    Amaziko esizwe

    Ministries, statistics offices and agencies: a national desk and chapter, and a seat in the working groups. National chapters

  • Indlela

    Sectors and professions

    Operators, regulators, banks, insurers and professional bodies whose work touches technology and digital systems. For organisations

  • Indlela

    Ukufunda

    Courses and practice, with a record of every course completed. Free to study. Open learning

  • Indlela

    Imigangatho

    Draft, review and adopt the methods and data standards with the working groups. Working groups

  • Indlela

    Communities and research

    Universities, civil society and individual experts: projects, data and review. Write to the Academy

Open calls

Open callSeeksCloses
Founding call for experts to the working groups CALL-FOUNDING-EXPERTSexpert, editor, chair2026-12-15
Open nomination for the three independent expert seats of the Global Standards Council CALL-GSC-INDilungu-lebhunga2026-12-01
Nominations of interim regional members (applies only if the transitional provision is decided) CALL-GSC-REGilungu-lebhunga2026-12-15
Uphononongo loshwankathelo olungundoqo lomngcipheko lwegagasi-1 (C1, C2, C3, C4, C8, C10, C11) CALL-REVIEW-WAVE1contributor2027-01-31
Iingcali zemigangatho engundoqo yenzululwazi yomngcipheko: ukungaqiniseki nokuhlelwa kobungqina, izikali, isigwebo seengcali esicwangcisiweyo, ukunyamezeleka, iziganeko ezigqithisileyo nezidibeneyo, umngcipheko weentlekele nowobukho, iinqobo ezisesikweni nokulingana, iiyunithi nomlinganiselo (NRS-C17 ukuya ku-NRS-C24) CALL-CORE-SCIENCEexpert, editor, chair2027-01-31
Independent reviewers for the families of the hazard atlas, three from at least two regions for each family (NHS2-R19), priority families first CALL-ATLAS-REVIEWingcali2027-03-31
Amalungu ePhaneli yethutyana yoPhononongo loKhuseleko lweTekhnoloji eNtsha nabaphononongi beqela elibomvu (ukhuseleko lwebhayoloji, ukhuseleko lwe-AI, ukhuseleko lweziseko ezibalulekileyo); izicelo ziphathwa ngemfihlo CALL-FRONTIER-SAFETY-REVIEWingcali2026-11-30
Iingcali zeontoloji, inzululwazi yemilinganiselo, ukuqinisekiswa kwesoftware novavanyo lokuhambelana zerejista yeemfuno, umfanekiso weNHS-2 woomatshini, iiseti zovavanyo lokuhambelana nesikimu sovavanyo lokuhambelana CALL-CONFORMANCE-ONTOLOGYexpert, editor2027-01-31

Calls past their closing date are hidden automatically. Every call, proposal and route: open areas and calls.

Cells and CERTs

  1. Gather the cell

    Bring together three to seven people with complementary skills: data, field practice, policy and communication.

  2. Register it

    Write to the team with the cell's name, members, area and region. You receive a reference and a contact. Write to the Academy

  3. Choose a charter

    Pick one focus: research and evidence, data and maps, standards review, or community preparedness.

  4. Learn together

    Follow shared study paths at Risk Academy, with practice and a record of every course completed. Hazards and resilience

  5. Lead and report

    Hold regular sessions. Publish what the cell learns through Reports. Bring findings to the working groups. Working groups

  • Specialise as a Community Emergency Response Team (CERT)

    A CERT prepares its neighbourhood for the hazards it faces. It builds a local hazard profile from this site's map. It trains in household and street preparedness. It learns first aid and light search through recognised local providers, and it runs drills. A CERT works alongside the local emergency services and follows their direction. It never replaces them and is not an emergency service.

Isalathiso

What a national team or a local community can do on The Global Risks Index (NXS-GRX) today.

  • NXS-GRX

    Your country's profile

    Open risk profiles for 195 countries, with sources and licences on every value. Open the Index

  • NXS-GRX

    Correct or reply

    Correct a value or reply for your country. The change is logged with its effect. Open the Index

  • NXS-GRX

    Offer data

    Offer a dataset. It enters through the licence gate, with your credit. Open the Index

  • NXS-GRX

    Use the open data

    Layers, profiles and tiles through the open API. Open the Index

10References

Core references and where to begin

Start with Bommasani, then Bengio, then National Institute of Standards.

  1. Vaswani, A., Shazeer, N., Parmar, N. et al., Attention Is All You Need (2017). arxiv.org
  2. Bommasani, R., Hudson, D. A., Adeli, E. et al., On the Opportunities and Risks of Foundation Models (2021). arxiv.org
  3. Kaplan, J., McCandlish, S., Henighan, T. et al., Scaling Laws for Neural Language Models (2020). arxiv.org
  4. Amodei, D., Olah, C., Steinhardt, J. et al., Concrete Problems in AI Safety (2016). arxiv.org
  5. Bender, E. M., Gebru, T., McMillan-Major, A. and Shmitchell, S., On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021). DOI
  6. Mitchell, M., Wu, S., Zaldivar, A. et al., Model Cards for Model Reporting (2019). DOI
  7. Ji, Z., Lee, N., Frieske, R. et al., Survey of Hallucination in Natural Language Generation (2023). DOI
  8. Jumper, J., Evans, R., Pritzel, A. et al., Highly accurate protein structure prediction with AlphaFold (2021). DOI
  9. Goodfellow, I. J., Shlens, J. and Szegedy, C., Explaining and Harnessing Adversarial Examples (2015). arxiv.org
  10. National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (2023). nist.gov
  11. Bengio, Y., Mindermann, S., Privitera, D. et al., International AI Safety Report (2025). arxiv.org

11Imibuzo

Imibuzo, iimpendulo namagama aphambili

Ingaba ukufunda apha kusimahla?

Ewe. Yonke ikhosi nezixhobo zokufunda ze-Risk Academy, ekhokelwa yi-Nexus Consortium, zisimahla ukuzifunda, ngesantya sakho okanye neqela. Ukuthatha inxaxheba okuqhelekileyo akunamrhumo; umsebenzi ohlawulwayo, ukuqeqeshelwa umsebenzi neefelowshiphu ngamalungiselelo ahlukileyo.

Ukugqiba ikhosi kundinika ntoni?

A completion record: the course, its version, the date and what was assessed, private by default and shared by you. It stacks into discipline, department and programme achievements. A record states what was studied and assessed and does not certify competence; it is not a licence, a professional title or an employment status.

Idatha neendawo zivela phi?

Every faculty teaches on a twin of a real place built from open data; each layer carries its provider, licence and date, and every result on the Academy states its source. Maps and figures are indicators for teaching, never an assessment of a place.

How do I take part beyond the courses in artificial intelligence and machine learning?

Form a competence cell of three to seven people, study a shared pathway, take a supervised assignment in the National Working Groups, or bring a cohort from your institution. For organisations.

Amagama aphambili

Model evaluation
Testing an AI model's capabilities, reliability and risks against defined tasks and criteria.
Distribution shift
A change between the data a model was trained on and the data it meets in use.
Hallucination
Output from a generative model that is fluent but false or unsupported.
Agentic system
An AI system that plans and takes actions toward goals with limited human direction.

IingxeloYiba negalelo

Faka ubungqina obuvela kweli phepha.

Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

Risk Academy

Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.

Ubukrelekrele bokuzenzela nokufunda koomatshini

Start with an open course

The catalogue lists each course with its outcomes, labs, hours and level.

Open RAT-204Back to Department of Artificial Intelligence and Autonomous Systems