[ SRE_ARENA ]
Roadmap
System Design
SREdle
⚡ 1420 ELO
OP
☰
←
BACK TO ROADMAP
INCIDENT MANAGEMENT
Stop Debugging and Start Mitigating.
Visualization: 1. The Incident Command System (ICS)
Subject Matter Expert
Investigates & Executes
Incident Commander (IC)
Makes decisions. Hands off keyboard.
Communications Lead
Updates Execs & Customers
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
1. The Incident Command System (ICS)
2. Mitigation vs Resolution
3. Severity Levels (SEV)
DOWNLOADING CONCEPT: 1. THE INCIDENT COMMAND SYSTEM (ICS)...
↳
When the database crashes during Black Friday, chaos erupts.
↳
Ten engineers jump into Slack, all trying different fixes simultaneously, accidentally stepping on each other's toes.
↳
To establish order, SREs use the
Incident Command System (ICS)
(adapted from firefighters):
1. Incident Commander (IC)
↳
The boss of the incident.
↳
The IC's golden rule is *keep your hands off the keyboard*.
↳
They don't look at logs.
↳
They coordinate the SMEs, make the hard calls ('Drop 50% of traffic to save the cluster'), and ensure progress.
↳
2. Subject Matter Experts (SMEs):
The engineers executing the commands.
↳
They report their findings to the IC.
↳
3. Scribe:
Documents every command run and exact timestamps.
↳
Critical for the postmortem.
↳
4. Communications Lead:
Writes status page updates and emails executives so they stop bothering the IC.
▲ LIVE INCIDENT
Site is down. 15 engineers are in Slack shouting ideas. Someone just ran a database migration.
sre@arena:~/incident-management
sre@arena
:
~
$
cat slack_transcript.txt
▶ EXECUTE COMMAND