[ SRE_ARENA ]
CONSENSUS ALGORITHMS (RAFT/PAXOS)

Getting Machines to Agree is Hard.

Visualization: 1. The Two Generals Problem
DOWNLOADING CONCEPT: 1. THE TWO GENERALS PROBLEM...
  • ↳Distributed consensus is based on an unsolvable thought experiment: Two generals are on opposite sides of a valley, trying to coordinate an attack on a city in the middle.
  • ↳If they attack together, they win.
  • ↳If one attacks alone, they die.
  • ↳General A sends a messenger: 'Attack at dawn.
  • ↳Acknowledge.' General B receives it and sends a messenger back: 'I acknowledge.' But General B now thinks: 'What if my messenger gets captured? General A won't attack if he doesn't get my acknowledgement.' So General A must send a messenger back acknowledging the acknowledgement.
  • ↳This loops infinitely.
  • ↳Over an unreliable network, it is mathematically impossible to guarantee 100% agreement between two nodes.
▲ LIVE INCIDENTYour 5-node etcd cluster is experiencing a massive network partition.
sre@arena:~/consensus
sre@arena:~$ etcdctl endpoint status --cluster