[ SRE_ARENA ]
INCIDENT MANAGEMENT

Stop Debugging and Start Mitigating.

Visualization: 1. The Incident Command System (ICS)
DOWNLOADING CONCEPT: 1. THE INCIDENT COMMAND SYSTEM (ICS)...
  • ↳When the database crashes during Black Friday, chaos erupts.
  • ↳Ten engineers jump into Slack, all trying different fixes simultaneously, accidentally stepping on each other's toes.

↳To establish order, SREs use the Incident Command System (ICS) (adapted from firefighters):

1. Incident Commander (IC)
  • ↳The boss of the incident.
  • ↳The IC's golden rule is *keep your hands off the keyboard*.
  • ↳They don't look at logs.
  • ↳They coordinate the SMEs, make the hard calls ('Drop 50% of traffic to save the cluster'), and ensure progress.
  • ↳2. Subject Matter Experts (SMEs): The engineers executing the commands.
  • ↳They report their findings to the IC.
  • ↳3. Scribe: Documents every command run and exact timestamps.
  • ↳Critical for the postmortem.
  • ↳4. Communications Lead: Writes status page updates and emails executives so they stop bothering the IC.
▲ LIVE INCIDENTSite is down. 15 engineers are in Slack shouting ideas. Someone just ran a database migration.
sre@arena:~/incident-management
sre@arena:~$ cat slack_transcript.txt