Skip to content
Novandri.

Site Reliability Engineering: SLOs, Error Budgets, and Incident Response

SRE replaced vague reliability goals with measured objectives and a budget for failure. Read how the mechanism actually works.

Rachmadwipa Novandri7 min read

Most organisations say they want high availability, and almost none can say what that means in a number, over what window, measured how. Site reliability engineering starts by making that sentence precise, and the precision changes the conversation — because once reliability has a number, the organisation can decide how much unreliability it can afford.

Direct answer: An SLO is a target for a service level indicator — the Google SRE book defines an SLI as "a carefully defined quantitative measure of some aspect of the level of service that is provided" — and an error budget is the amount of failure the SLO permits. When the budget is intact, the team can ship; when it is exhausted, reliability work takes priority until it recovers. That is the mechanism that stops reliability and velocity from being opposing slogans.

The budget is not a ceiling to hide under; it is a currency that funds change.

A four-step flow: define the SLI, set the SLO, spend the budget, then alert on burn rate.
Conceptual: reliability becomes a number. The budget is not a ceiling to hide under; it is a currency that funds change.

The terms do real work

The SRE book is unusually careful about language, and the distinction matters. An SLI is a measurement — availability, latency, error rate, whatever can be quantified from the system's behaviour. An SLO is the target for that measurement. An SLA is an agreement with consequences, usually external.

The book notes that "the term SLA is overloaded and has taken on a number of meanings depending on context", which is why the vocabulary has to be fixed before a team can reason about it. Saying "we have a 99.9% SLA" when what exists is an internal target with no contractual consequence confuses the three, and the confusion shows up in incident discussions.

Choosing an SLI is the first substantive decision. It has to measure what users experience, not what is convenient to instrument: a server's CPU load is not an availability signal, and a synthetic check that polls a healthy endpoint is not the same as a user completing a purchase. The book's guidance is to use "intuition, experience, and an understanding of what users want" — which is a reminder that the definition is a product decision, not a monitoring default.

The error budget is the actual invention

An SLO of 99.9% over a month permits roughly 43 minutes of unavailability. That number is an error budget, and its value is not that it measures failure but that it allocates it.

The allocation works in both directions. A team that has used little of its budget can take risks — deploy more often, run experiments, accept that a change might cause a short incident. A team that has exhausted it stops feature work and spends the next period on reliability, because another incident would breach the objective. This is a pre-agreed decision rule, and it removes the argument about whether "now" is a good time to be careful.

The budget also makes a uncomfortable question answerable: is 99.9% the right target? If the team never spends the budget, the objective is too loose and users are getting less reliability than they could. If it is permanently exhausted, the objective is too strict for the system's current design, and the answer is engineering work or a renegotiation — not a heroic effort to avoid incidents.

Alerting on symptoms and burn rate

A budget that is reviewed monthly is too slow to prevent a breach. The SRE Workbook's chapter on alerting introduces burn rate: "Burn rate is how fast, relative to the SLO, the service consumes the error budget." A burn rate of 1 exhausts the budget exactly at the end of the window; a burn rate of 10 exhausts it in a tenth of the time.

The point is behavioural. A slow, low-level failure should not page anyone — it will show up in the monthly review. A fast failure should page immediately, because at that rate the budget will be gone in hours. Alerting on burn rate rather than on a fixed error threshold gives detection time and precision at the same time, which is the stated goal of the approach.

The principle is to alert on symptoms — what users experience — and on the rate at which the agreed budget is being spent.

Chaos engineering tests the assumption

An SLO describes what should happen; chaos engineering tests what actually happens when something fails. The Principles of Chaos Engineering describe the discipline as experimenting "to uncover systemic weaknesses", and list failure modes that include "retry storms from improperly tuned timeouts" and "cascading failures when a single point of failure crashes".

The method starts by defining "steady state as some measurable output of a system that indicates normal behavior", then introduces a real failure and checks whether steady state holds. That is an experiment with a hypothesis and an observable result, not a game day with a script.

The relevance to SLOs is direct: a chaos experiment tells you whether your measured SLI actually captures the failure mode you care about, and whether the fallback you assumed works under realistic conditions.

Many organisations discover that their availability metric stayed green during a test while users could not complete a transaction — which means the SLI was wrong, not the system.

A sequence that works

  1. Choose one SLI per user journey that can be measured from real requests.
  2. Set an SLO over a defined window, and write down why that target.
  3. Derive the error budget and make it visible to the whole team.
  4. Agree the policy in advance: what the team may do while budget remains, and what it must do when the budget is gone.
  5. Alert on burn rate, with a short-window alert for fast failures and a long-window one for slow ones.
  6. Run chaos experiments against the assumptions behind the SLI and the fallback paths.
  7. Review the target periodically. A budget that is never spent is a target that should be tighter; one that is always spent is a design problem.

For the automation side, see DevOps; for the statistical idea of alerting quality, see Model Monitoring.

A worked example

The following is hypothetical and illustrative. A team runs a checkout service with a stated availability target of 99.9 percent over a month, roughly 43 minutes of permitted unavailability. Check one tests whether the SLI measures the user journey: a purchase completing, not CPU load on the application servers or a synthetic probe of a healthy endpoint, since the SRE book calls the indicator a quantitative measure of service and leaves its choice to intuition and experience. Check two examines burn rate rather than a fixed error threshold: a fast failure that would spend the whole budget in hours pages immediately, while a slow one is left to the monthly review, following the Workbook's alerting guidance. Check three runs a chaos experiment around the assumed fallback, injecting a failure of a single dependency and checking whether the measured steady state holds while a transaction can still complete. The outcome is bounded: if the indicator stayed green during the test while purchases failed, the metric is wrong rather than the system, so the team revises the SLI before judging the budget.

A counterargument

A critic would argue that the precision is borrowed, not inherent. The SRE book calls the definition of an SLI a matter of "intuition, experience, and an understanding of what users want," and warns that "SLA" is overloaded - so the number inherits whatever a team chose to measure, including a metric that stays green while users cannot complete a transaction, as the article itself notes. The error-budget rule also presumes an organisation that can halt feature work and whose reliability work will recover the budget; the cited SRE material describes the mechanism but does not guarantee that governance. Burn-rate alerting from the Workbook still needs enough signal to compute a rate, and a budget never spent may reflect a coarse SLI rather than a tight target.

References & Further Reading