Visualization: 1. The Psychology of Alerting (Why We Alert)
Noisy Alerts
CPU high, Disk 60%
Alert Fatigue
Ignored by brain
Major Outage
Sev-1 Missed
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
DOWNLOADING CONCEPT: 1. THE PSYCHOLOGY OF ALERTING (WHY WE ALERT)...
↳The fastest way to burn out an engineering team is a poorly configured alerting system.
↳If an engineer's phone buzzes 50 times a night for non-actionable issues (e.g., a background job took 2 seconds longer than usual), psychological Alert Fatigue sets in.
↳The human brain naturally learns to ignore the pager.
↳When a real, catastrophic Sev-1 outage occurs the next week, the engineer will subconsciously assume it's just 'another noisy alert' and roll over to go back to sleep.
↳Alerting is a psychological tool: every single page must represent an immediate, undeniable threat to the business that requires human intervention.
▲ LIVE INCIDENTAn engineer was paged 45 times last night for the same database issue.