Skip to content
Novandri.

Incident Response at Scale

At scale, incidents are a coordination problem. Read how to respond, a worked example, and why a hero culture is a warning sign.

Rachmadwipa Novandri5 min read

Incident response at scale is less about individual brilliance and more about clear roles, signals, and rehearsal. The NIST incident-handling guide treats preparation as part of the lifecycle, as the SRE alerting workbook ties detection to service objectives, as the rollback article treats recovery as a design choice, as the observability article treats signals as evidence, as the risk article requires residual risk.

Direct answer:

Respond with a named lead, a containment step, a tested recovery path, and a record, then review without blame.

NIST SP 800-61 covers the lifecycle, and the SRE workbook covers detection. The boundary: no process guarantees a clean recovery, which the publication standard forbids claiming, as the risk article requires residual risk, as the red-teaming article treats testing as partial.

A four-step flow: detect against an objective, name a lead, contain first, review without blame.
Conceptual: roles and rehearsal beat heroics.

The structure

Detect against an objective. An alert should be tied to a service objective, which the SRE workbook requires, as Site Reliability Engineering: SLOs, Error Budgets, and Incident Response explains, as the observability article treats signals as evidence, as the alerting article warns about noise, as the rubric article requires concrete criteria, as the risk article requires residual risk, as the decision-memo article requires for a record.

Name a lead. One person coordinates, which the NIST guide treats as part of handling, as the governance checklist requires an owner, as the small-team decision article advises, as the crisis article treats a public record, as the decision-memo article requires for a record, as the monitoring article advises for review, as the risk article requires residual risk, as Site Reliability Engineering: SLOs, Error Budgets, and Incident Response requires containment.

Contain before fixing. The first step is to limit impact, which the NIST guide treats, as Site Reliability Engineering: SLOs, Error Budgets, and Incident Response requires, as the rollback article advises on reverting a bad change, as Disaster Recovery: a Decision Framework treats a tested path, as the data-governance article requires for records, as the decision-memo article requires for a judgement, as the risk article requires residual risk, as the monitoring article advises for review, as the observability article treats signals as evidence.

Review without blame. A post-incident review should find the cause, which the root-cause article advises, as the NIST guide treats the lifecycle, as the monitoring article advises for review, as the decision-memo article requires a reversal condition, as the data-governance article requires dated figures, as the risk article requires residual risk, as the team-culture article warns about norms, as the decision-memo article requires for a record.

Three consequences follow. The signal comes before the response, as the SRE workbook and observability article advise, as the alerting article warns about noise, as the rubric article requires criteria, as the risk article requires residual risk, as the decision-memo article requires for a record, as the NIST guide treats the lifecycle, as the governance checklist requires an owner, as the small-team decision article advises, as the crisis article treats a public record, as the rollback and disaster-recovery articles require a tested path, as the data-governance article requires provenance, as the root-cause article advises, as the team-culture article warns, as the monitoring article advises for review.

A worked example

The following is hypothetical and illustrative. A deploy causes a sustained rise in errors across several services.

Three checks decide. First, is the alert tied to an objective and sustained, as the SRE workbook requires, with evidence, as the observability article advises? Second, is there a named lead and a containment step, as the NIST guide and incident article require, with a tested rollback, as the rollback and disaster-recovery articles advise? Third, was the cause recorded, as the root-cause and decision-memo articles require, with a reversal condition, as the decision-memo article requires, as the risk article requires residual risk?

Three conclusions follow. A lead prevents duplicated work, which the checks reveal. Recovery should be tested, as the rollback and disaster-recovery articles advise. And the review should be recorded, as the decision-memo article requires.

What incident response cannot do

It does not replace monitoring, which the observability and SRE articles require, as the alerting article warns about noise, as the rubric article requires criteria, as the monitoring article advises for review. And it cannot remove the need for design changes, which the rollback and disaster-recovery articles advise, as the root-cause article treats cause, as the decision-memo article requires a reversal condition, as the risk article requires residual risk.

The counterargument that a hero culture is acceptable during a crisis is fair for short bursts and incomplete as a strategy, which the worked example shows, as the NIST guide and governance checklist require clear roles, as the team-culture article warns, as the risk article requires residual risk.

A decision test

  1. Tie alerts to objectives, as the SRE workbook requires.
  2. Name a lead, as the governance checklist requires.
  3. Contain first, as the NIST guide requires.
  4. Use a tested recovery path, as the rollback article advises.
  5. Record the decision, as the decision-memo article requires.
  6. Find the cause, as the root-cause article advises.
  7. Review and state residual risk, as the monitoring and risk articles require.

For the lifecycle, see Incident Response; for the design, see Disaster Recovery.

References & Further Reading

  • Paul Cichonski, Thomas Millar, Tim Grance, Karen Scarfone (NIST), 2012. SP 800-61 Rev. 2, Computer Security Incident Handling Guide — The incident-response capability lifecycle: preparation, detection and analysis, containment/eradication/recovery, and post-incident activity.
  • Google, 2016. Alerting on SLOs — How to alert on symptoms and burn rate rather than on causes.