Skip to content
Novandri.

AI Red Teaming Metrics that Matter

A red team finds failures before users do. Read the metrics, a worked exercise, and why a finding count is a weak result.

Rachmadwipa Novandri6 min read

Red teaming is often reported as a number of findings, which rewards volume over relevance. The NIST adversarial ML taxonomy describes categories of attack and mitigation, and the OWASP LLM list identifies risks for language-model applications. The useful question is whether the exercise changed a decision about the system.

Direct answer: Measure red teaming by the coverage of defined attack categories, the severity of confirmed failures, and the rate at which fixes are verified. The sources provide the categories; the measurement rule is a synthesis.

Delta: This article adds a claim-status layer and a worked scoring method, so the distinction between documented categories and locally chosen thresholds is explicit.

The mechanism

Three ideas from the sources do the work:

  • Categories are defined. The NIST taxonomy names attack classes. Using it as a checklist makes coverage visible instead of implicit.
  • Risks are application-specific. The OWASP list describes risks for LLM applications. A generic test suite cannot tell you which of them matters to your deployment.
  • A finding is not a fix. A confirmed failure is evidence, but it becomes useful only when the mitigation is tested again.

The practical sequence is: define the system's boundaries, choose relevant categories, run probes with recorded conditions, confirm failures, then retest after mitigation. Choosing categories is judgement, which makes it inferred; the existence of the categories is source-reported.

A cycle of choosing attack categories, recording probe conditions, ranking severity, then mitigating and retesting the same probe.
Conceptual: a red-team finding is useful only after the same probe is retested after mitigation.

A worked exercise

Illustrative. A support assistant can read documents and issue refunds.

Three checks decide. First, which attack categories apply to a system that reads untrusted text and can act on refunds? Second, for each confirmed failure, was the condition recorded precisely enough to reproduce, including the input and the tool state? Third, after a mitigation, was the same probe re-run, and did it stop producing the failure?

The outcome is a severity-ranked list with verified fixes, not a raw count. The thresholds are a local decision; the method is the contribution.

Why the scoring method matters

A security exercise produces evidence only if the conditions are recorded. A failure that cannot be reproduced is a story, not a finding. The condition record should include the input, the tool state, and the configuration that was active at the time. Without those details, a mitigation cannot be verified, and the team repeats the same discovery in the next exercise under a slightly different name.

Severity is the second problem.

A severity scale works only when it is defined before the exercise and applied without adjustment afterwards. Otherwise the scale becomes a way of making the report look better rather than a way of making risk visible.

Coverage answers a different question from severity. A system can have few findings because it is robust, or because only one category was tested. Reporting coverage makes that distinction visible. It also exposes the reverse case: many findings concentrated in one category may indicate that one area was examined deeply while others were ignored.

Retesting closes the loop.

A confirmed failure that has been mitigated is not resolved until the same probe is run again.

Teams that skip this step accumulate a backlog of findings marked fixed on the strength of a code change rather than evidence. A small retest suite, kept from the original exercise, is usually enough.

Taken together, these practices turn a red team from a reporting exercise into a control. The categories come from the sources; the decision about what to test, how to score, and when to retest belongs to the team running the system.

Evidence status

ClaimStatusSourceVerifiedRecheck when
Adversarial ML attacks can be taxonomisedsource-reportedNIST AI 100-2 E20252026-11-18New taxonomy released
LLM applications have identified risk categoriessource-reportedOWASP LLM Top 102026-11-18New edition published
Retest confirmed failures after mitigationinferredDerived from the risk lifecycle2026-11-18Process changes

Why counting findings misleads

A count mixes a harmless anomaly with a critical failure. It also grows with the effort spent looking, which means a more thorough team can appear worse. Severity weighting helps, but only if the severity scale is defined before the exercise and applied consistently. Reproduction is the stronger test: a finding that cannot be reproduced cannot be verified as fixed.

Coverage matters for the opposite reason. A small number of findings concentrated in one category may indicate that other categories were never tested. Reporting coverage alongside severity prevents a team from mistaking an easy category for a secure system.

What it cannot do

Red teaming cannot prove the absence of failures. It does not replace monitoring, which observes the deployed system. And it cannot assign risk ownership, which remains a governance decision.

The counterargument that a clean exercise means a safe system is fair for the tested cases and incomplete for everything outside them.

A decision test

  1. Define the system boundary.
  2. Choose relevant categories from the taxonomy.
  3. Record probe conditions.
  4. Confirm and severity-rank failures.
  5. Mitigate and retest the same probes.
  6. Report coverage and verified fixes.
  7. Record what remains untested.

For the method, see AI Red Teaming.

A worked example

The following is hypothetical and illustrative. A hypothetical support assistant can read customer documents and issue refunds; a red team is asked to test it before launch. Three checks define the exercise. First, which categories apply? Using the NIST adversarial ML taxonomy as a checklist makes coverage visible, and the OWASP LLM list identifies which risks matter to an application that reads untrusted text and can act, rather than to a generic suite. Second, is each confirmed failure recorded precisely enough to reproduce, including the input, the tool state, and the active configuration? A failure without that record is a story, not a finding, and cannot later be verified as fixed. Third, after a mitigation, was the same probe re-run and did it stop producing the failure? The outcome is bounded: the report is a severity-ranked list with verified fixes and a note on what remains untested, not a finding count, so a more thorough team does not appear worse simply for looking harder.

References & Further Reading