How to Get AI Safety Cases Right
A safety case argues why a system is acceptable for a use. Read what it must contain, a worked example, and why documentation alone is not safety.
A safety case is a structured argument that a system is acceptable for a specific use, with evidence for the claims. The idea comes from safety-critical engineering and is increasingly applied to AI, where the risk is that a generic assurance document is mistaken for evidence about a particular deployment. A good case is bounded, falsifiable, and alive, as the AI RMF frames trustworthiness as a property of design, development, use, and evaluation.
Direct answer: Build a safety case that states the system, the use, the hazards, the controls, and the evidence that the controls work, with an explicit residual risk and a review trigger. The NIST AI RMF and the adversarial-ML taxonomy provide the frameworks. The boundary: a case is an argument, not a guarantee, and it covers the conditions it states, as the risk-assessment article requires for honest residual risk, as AI Red Teaming: A Practical Methodology explains why untested failures remain.
A case that lists controls without evidence is a document, as the evaluation article warns about activity measures.

What it contains
Scope and use. The specific system, the decision it supports, and the conditions excluded, which the clinical-AI article treats as intended use, as the decision-memo article requires for a bounded claim.
Hazards and controls. What can go wrong and what reduces it, which the threat-modeling article structures by assets, boundaries, and controls, as the risk-assessment article requires likelihood and impact.
Evidence. Evaluation results, red-team findings, and monitoring data that support the claim, which the evaluation-rubric article requires for concrete criteria, as the data-governance article requires for provenance and dates.
Residual risk and review. What remains and what would trigger reassessment, which the risk-assessment article treats as an explicit decision, as the monitoring article advises for changed conditions.
Three consequences follow. The case should be specific to a deployment, which the governance checklist requires an owner, as the AI Act article treats deployer obligations as role-specific. It should be falsifiable, as the evaluation article argues for tests that could fail, as the benchmark article warns about untested conditions. And it should be reviewed after a change, which the monitoring article treats as a trigger, as AI Red Teaming: A Practical Methodology recommends repeated adversarial testing.
A worked example
The following is hypothetical and illustrative. A support assistant drafts replies and can call a refund tool.
Three checks decide whether the case is credible. First, does it name the hazards — a wrong policy citation, an unauthorised refund, a missed escalation — as the threat-modeling article requires? Second, is there evidence for the controls, such as citation checks and tool authorisation, as the legal-research and tool-calling articles advise, with results from a fixed test set, as the evaluation-rubric article requires? Third, does it state the residual risk and a review trigger, as the risk-assessment article requires, after a model or policy change, as the monitoring article advises?
Three conclusions follow. And the case should be recorded with a date and an owner, as the decision-memo article requires.
What a safety case cannot do
It cannot prove the absence of failure, which the red-teaming article treats as a property of testing, as the publication standard forbids guarantees. It cannot replace monitoring, which the observability article treats as the signal that a system has drifted, as the monitoring article advises. And it does not create accountability — a named owner does, which the governance checklist requires, as the AI-governance article argues for clear responsibility.
The counterargument that documentation is bureaucracy is fair when a case is generic, which the worked example avoids by tying every claim to evidence and a specific use, as the evaluation-rubric article requires.
A decision test
- State the system and its use with exclusions, as the intended-use guidance requires.
- List hazards and controls, as the threat-modeling article advises.
- Attach evidence with dates and sources, as the data-governance article requires.
- State residual risk explicitly, as the risk-assessment article requires.
- Set a review trigger tied to change, as the monitoring article advises.
- Test the controls adversarially, as AI Red Teaming: A Practical Methodology recommends.
- Record the case with an owner, as the decision-memo article requires.
For the framework, see AI Safety; for the adversarial test, see AI Red Teaming.
References & Further Reading
- NIST, 2023. AI Risk Management Framework | NIST — The governance framework referenced in AI risk and compliance work.
- NIST, 2025. AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations | CSRC — The adversarial machine learning taxonomy that treats abuse and large-language-model attacks as a formal category.