Logs Are Just Text. Until You Have 5 Terabytes of Them.
Visualization: 1. The Distributed Logging Nightmare
SRE
SSH & Grep
Server A
/var/log/app.log
Server B
/var/log/app.log
Failure
Too Slow!
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
DOWNLOADING CONCEPT: 1. THE DISTRIBUTED LOGGING NIGHTMARE...
↳When you have one server, debugging is simple: ssh web-01 followed by tail -f /var/log/nginx/access.log.
↳But what happens when you auto-scale to 500 servers? If a user reports an error, you cannot SSH into 500 servers and grep 500 files to find their specific request.
↳Even worse, if a server crashes and terminates (e.g., an AWS Spot Instance or a Kubernetes Pod), its local disk—and all the logs on it—are instantly deleted forever.
↳To solve this, SREs decouple log *generation* from log *storage*.
↳We use Centralized Logging.
↳The application writes logs locally, but a background agent instantly ships those logs over the network to a permanent, searchable database.
▲ LIVE INCIDENTThe Elasticsearch cluster went red. No new logs are being ingested.
sre@arena:~/centralized-logging
sre@arena:~$ curl -X GET "localhost:9200/_cluster/health?pretty"