Skip to content
Novandri.

How to Get AI Safety Cases Right

A safety case argues why a system is acceptable for a use. Read what it must contain, a worked example, and why documentation alone is not safety.

Rachmadwipa Novandri4 min read

A safety case is a structured argument that a system is acceptable for a specific use, with evidence for the claims. The idea comes from safety-critical engineering and is increasingly applied to AI, where the risk is that a generic assurance document is mistaken for evidence about a particular deployment. A good case is bounded, falsifiable, and alive, as the AI RMF frames trustworthiness as a property of design, development, use, and evaluation.

Direct answer: Build a safety case that states the system, the use, the hazards, the controls, and the evidence that the controls work, with an explicit residual risk and a review trigger. The NIST AI RMF and the adversarial-ML taxonomy provide the frameworks. The boundary: a case is an argument, not a guarantee, and it covers the conditions it states, as the risk-assessment article requires for honest residual risk, as AI Red Teaming: A Practical Methodology explains why untested failures remain.

A case that lists controls without evidence is a document, as the evaluation article warns about activity measures.

A four-step flow: state use and hazards, name controls, show evidence, then name residual risk.
Conceptual: bounded, falsifiable and alive. A case that lists controls without evidence is a document.

What it contains

Scope and use. The specific system, the decision it supports, and the conditions excluded, which the clinical-AI article treats as intended use, as the decision-memo article requires for a bounded claim.

Hazards and controls. What can go wrong and what reduces it, which the threat-modeling article structures by assets, boundaries, and controls, as the risk-assessment article requires likelihood and impact.

Evidence. Evaluation results, red-team findings, and monitoring data that support the claim, which the evaluation-rubric article requires for concrete criteria, as the data-governance article requires for provenance and dates.

Residual risk and review. What remains and what would trigger reassessment, which the risk-assessment article treats as an explicit decision, as the monitoring article advises for changed conditions.

Three consequences follow. The case should be specific to a deployment, which the governance checklist requires an owner, as the AI Act article treats deployer obligations as role-specific. It should be falsifiable, as the evaluation article argues for tests that could fail, as the benchmark article warns about untested conditions. And it should be reviewed after a change, which the monitoring article treats as a trigger, as AI Red Teaming: A Practical Methodology recommends repeated adversarial testing.

A worked example

The following is hypothetical and illustrative. A support assistant drafts replies and can call a refund tool.

Three checks decide whether the case is credible. First, does it name the hazards — a wrong policy citation, an unauthorised refund, a missed escalation — as the threat-modeling article requires? Second, is there evidence for the controls, such as citation checks and tool authorisation, as the legal-research and tool-calling articles advise, with results from a fixed test set, as the evaluation-rubric article requires? Third, does it state the residual risk and a review trigger, as the risk-assessment article requires, after a model or policy change, as the monitoring article advises?

Three conclusions follow. And the case should be recorded with a date and an owner, as the decision-memo article requires.

What a safety case cannot do

It cannot prove the absence of failure, which the red-teaming article treats as a property of testing, as the publication standard forbids guarantees. It cannot replace monitoring, which the observability article treats as the signal that a system has drifted, as the monitoring article advises. And it does not create accountability — a named owner does, which the governance checklist requires, as the AI-governance article argues for clear responsibility.

The counterargument that documentation is bureaucracy is fair when a case is generic, which the worked example avoids by tying every claim to evidence and a specific use, as the evaluation-rubric article requires.

A decision test

  1. State the system and its use with exclusions, as the intended-use guidance requires.
  2. List hazards and controls, as the threat-modeling article advises.
  3. Attach evidence with dates and sources, as the data-governance article requires.
  4. State residual risk explicitly, as the risk-assessment article requires.
  5. Set a review trigger tied to change, as the monitoring article advises.
  6. Test the controls adversarially, as AI Red Teaming: A Practical Methodology recommends.
  7. Record the case with an owner, as the decision-memo article requires.

For the framework, see AI Safety; for the adversarial test, see AI Red Teaming.

References & Further Reading