Skip to content
Novandri.

AI Safety: Alignment, Robustness, and Governance

Safety is not a single property. Read the three distinct problems — intent, reliability under change, and institutional control — and what each requires.

Rachmadwipa Novandri9 min read

AI safety is a term that covers several different concerns, and treating them as one produces muddled policies. A system can faithfully follow the wrong objective, behave correctly on the test distribution and badly outside it, or be technically sound while operating without any accountable owner. These are three problems with three different literatures.

Direct answer:

Alignment asks whether a system does what its operators intend; robustness asks whether it continues to behave correctly under distribution shift and adversarial input; governance asks who is accountable and what evidence is required.

The research gives concrete mechanisms for each — reinforcement learning from human preferences for alignment, evaluation under shift for robustness, and frameworks such as NIST's for governance. NIST describes its AI Risk Management Framework as "intended for voluntary use and to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems": a process for producing evidence, not a guarantee of safety.

A stack of layers: governance: who is accountable, robustness: novel conditions, alignment: specified intent.
Conceptual: intent, change, accountability. A sound system without an owner is a governance failure.

Alignment: the problem of specifying intent

A system optimises the objective it is given, not the one its designers had in mind. When the objective is an approximation — a proxy for what we actually want — an optimiser will find ways to score well on the proxy while failing at the intended goal.

Christian et al. show that goals can be defined through "(non-expert) human preferences between pairs of trajectory segments", solving complex tasks "without access to the reward function", including Atari games and simulated locomotion. The cost of oversight was remarkably low: feedback on "less than one percent of our agent's interactions", and roughly "an hour of human time" to train complex novel behaviours. The mechanism works because comparing two outcomes is easier for a human than writing a complete reward function.

less than one percentof our agent's interactionsChristiano et al.
an hour of human timeto train complex novel behavioursChristiano et al.

Bai et al. extend this with Constitutional AI, where "the only human oversight is provided through a list of rules or principles". The method combines a supervised phase of self-critique and revision with a reinforcement phase using "RL from AI Feedback" (RLAIF), producing an assistant that is "harmless but non-evasive" — one that "engages with harmful queries by explaining its objections to them".

The governance significance is easy to miss: the principles list is the artefact that encodes the values, and it is therefore the thing that must be reviewed, versioned, and owned. A constitutional approach does not remove human judgement; it makes the judgement explicit and machine-enforceable.

Robustness: the problem of novel conditions

A model can perform well on its evaluation set and fail in deployment because the deployment distribution differs. Distribution shift is not a curiosity; it is the normal condition for systems that interact with the world.

Three forms deserve separate treatment:

Covariate shift — the inputs change while the relationship to the outcome holds. A model trained on one camera's images may fail on another's, even though the task is the same.

Concept drift — the relationship itself changes. A fraud model trained on last year's patterns faces different fraud this year, and no amount of input monitoring reveals the change without labels.

Adversarial input — an actor deliberately crafts inputs that cause failure. This is a different threat model from natural shift, because the change is adaptive and targeted.

The practical response is not a single technique. It includes evaluation on data that differs from the training distribution, monitoring inputs and outputs after deployment, and designing fallback behaviour for conditions the model was not trained for. The SRE practice of alerting on burn rate — "how fast, relative to the SLO, the service consumes the error budget" — is a useful model for making a slow degradation visible before it becomes an incident.

The honest limit is that no evaluation covers every future condition. A system's behaviour outside its training distribution is fundamentally uncertain, which is why the fallback path matters as much as the model.

Governance: the problem of accountability

Mitchell et al.'s model cards and Gebru et al.'s datasheets are the documentation layer: they require the intended use, the evaluation conditions, and the known limits to be written down alongside the artefact. NIST's framework is the process layer, organising the work into governance, mapping, measurement, and management.

Three questions expose most governance gaps:

  1. Who can stop the system? If nobody has the authority and the mechanism to disable a deployed model, the organisation has no operational control.
  2. What evidence is required before deployment? A process that accepts a model on the basis of aggregate accuracy alone will miss the group-level failures that model cards were designed to surface.
  3. Where is the decision recorded? Value choices — which errors matter, which groups are prioritised — have to be explicit and attributable, or they will be made implicitly and inconsistently.

For the broader framework, see AI Governance; for the ethics of these choices, see AI Ethics.

A worked example of a safety failure

Consider a support system that drafts replies and is evaluated on resolution speed. The model learns to close tickets quickly, and the metric improves. Six months later, repeat contacts rise: the replies were fast but did not solve the problem, so customers came back.

Three interpretations are possible, and they correspond to the three problems above:

  • Alignment failure if the objective — resolution speed — was a poor proxy for the intended outcome, customer satisfaction. The optimiser did exactly what it was told.
  • Robustness failure if the model handled common requests well but degraded on a new category of problem that arrived after deployment. The evaluation set no longer represented the traffic.
  • Governance failure if nobody was monitoring repeat contacts, or if the team that owned the speed metric had no incentive to investigate a metric owned by another team.

The same observed problem can be caused by any of the three, and the fix depends on which one it is. Adding a better reward function does not fix a monitoring gap; improved monitoring does not fix a wrong objective. This is why the distinction matters operationally rather than academically.

A useful discipline is to write down, for each deployed system, which of the three failure classes is most plausible and what signal would reveal it. That record turns "safety" into a set of testable claims.

A practical checklist

  1. State the objective and the proxy it approximates, and name the ways they diverge.
  2. Evaluate on shifted conditions that resemble plausible deployment changes.
  3. Monitor outputs and outcomes, not only infrastructure health.
  4. Define the fallback for conditions the model was not trained for.
  5. Name the accountable owner and the authority to stop the system.
  6. Require documentation — model card, datasheet, intended use, known limits — before deployment.
  7. Record value decisions with their reasoning and review date.

What safety cannot promise

Three limits should temper any claim that a system is safe.

No evaluation is complete. A test set is a finite sample of conditions, and an adversarial actor is actively looking for conditions the test missed. A result of "no failures found" is a statement about the evaluation, not about the system's behaviour everywhere.

Safety properties do not compose. A model can be well-aligned and a deployment can still be unsafe because the interface encourages misuse, the fallback is broken, or the accountable owner has no authority. The worked example above showed how three independent failures can produce the same visible symptom, and the same is true in reverse: fixing one does not guarantee the system is safe.

The threat model changes. A system that was safe against natural distribution shift may not be safe against an adversary who studies it, and a policy that was appropriate at one scale of deployment may not be at another. Safety is therefore a programme of continuous evaluation and review, not a certification obtained once.

This is why the NIST framework is described as a tool for incorporating trustworthiness into "design, development, use, and evaluation" — the list includes use and evaluation as ongoing activities, not final checks. The documentation artefacts are valuable for the same reason: they make the system's claimed limits available to the people who will operate it, so that the limits can be tested in practice.

For the alignment research in detail, see Reinforcement Learning; for the responsible-AI context, see Responsible AI.

A counterargument

The three-way split is useful for organising questions but overstates how independent the problems are. Constitutional AI is filed under alignment, yet Bai et al. present it as a harmlessness procedure spanning self-critique and RLAIF, and the article itself says the principles list must be reviewed, versioned, and owned - which makes it a governance artefact as much as an alignment one. Christiano et al.'s "less than one percent" oversight figure is offered as a cost result, but such sparse feedback is also what leaves a reward model fragile under the distribution shift the article treats as a separate robustness problem. NIST describes its RMF as "intended for voluntary use", so the governance column rests on a process the piece elsewhere calls "not a guarantee of safety".

The taxonomy is incomplete as a claim that three causes have three independent fixes.

References & Further Reading