[ SRE_ARENA ]
Roadmap
System Design
SREdle
⚡ 1420 ELO
OP
☰
←
BACK TO ROADMAP
METRICS & MONITORING
Metrics: You Can't Fix What You Can't Measure.
Visualization: 1. The Four Golden Signals
Latency
Time to serve
Traffic
Requests/sec
Errors
5xx / Failures
Saturation
CPU/RAM/Disk usage
System Health
Holistic Overview
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
1. The Four Golden Signals
2. Instrumentation (How it's implemented)
3. The Push vs Pull Model
4. The Cardinality Explosion
DOWNLOADING CONCEPT: 1. THE FOUR GOLDEN SIGNALS...
Why do we monitor?
↳
To know if our system is broken before the customers complain on Twitter.
↳
Google's SRE book defines the *Four Golden Signals* as the minimum viable metrics you must track:
■
Latency:
The time it takes to service a request. (e.g., 99th percentile response time is 200ms).
■
Traffic:
A measure of how much demand is being placed on your system. (e.g., HTTP requests per second).
■
Errors:
The rate of requests that fail, either explicitly (HTTP 500s) or implicitly (returning the wrong content).
■
Saturation:
How "full" your service is. (e.g., 90% CPU utilization, or a database connection pool that is maxed out).
↳
If you only measure these four things, you can detect 95% of all outages.
▲ LIVE INCIDENT
Prometheus keeps crashing with OOMKilled. The server has 32GB of RAM.
sre@arena:~/metrics-monitoring
sre@arena
:
~
$
curl -s http://prometheus:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName | limit(5;.[])'
▶ EXECUTE COMMAND