[ SRE_ARENA ]
← BACK TO ROADMAP
APACHE KAFKA DEEP DIVE

Apache Kafka Deep Dive: The Nervous System of the Modern Internet

DOWNLOADING CONCEPT: 1. WHAT EXACTLY IS KAFKA?...
  • ↳Most developers think Kafka is a 'Message Queue' like RabbitMQ.
  • ↳It is not.
  • ↳Kafka is a distributed, immutable, append-only commit log.
  • ↳When a message is written to Kafka, it is appended to the end of a file on disk.
  • ↳It is immutable (you can never edit it) and it is persistent (it is not deleted when a consumer reads it).
  • ↳Consumers simply act as 'cursors' reading through a massive text file.
  • ↳Because reads are sequential and writes are append-only, Kafka can max out the physical limits of the hard drive (often processing millions of messages per second).
▲ LIVE INCIDENTOur Fraud Detection consumer group has 15 containers running, but 10 of them are doing absolutely zero work.
sre@arena:~/kafka-deep-dive
sre@arena:~$ kafka-topics.sh --describe --topic transactions