[ SRE_ARENA ]
Roadmap
System Design
SREdle
⚡ 1420 ELO
OP
☰
←
BACK TO ROADMAP
RESILIENCY PATTERNS
Assume everything will fail. Build Bulkheads.
Visualization: 1. Cascading Failures & Circuit Breakers
Service A
Checkout
Circuit Breaker
OPEN (Tripped)
Service B
Payments (Down)
Fallback
Fast Failure / Cache
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
1. Cascading Failures & Circuit Breakers
2. Retries & Jitter (The Thundering Herd)
3. Bulkheads (Resource Isolation)
DOWNLOADING CONCEPT: 1. CASCADING FAILURES & CIRCUIT BREAKERS...
↳
In a distributed system, if Service A calls Service B, and Service B is slow, Service A will wait.
↳
As more requests come in, Service A exhausts all of its threads waiting for B.
↳
Service A crashes.
↳
Now Service C, which calls Service A, crashes.
↳
This is a
Cascading Failure
.
↳
We stop this using a
Circuit Breaker
.
↳
If Service B fails or times out 5 times in a row, the circuit 'opens'.
↳
Service A stops trying to call Service B entirely, immediately returning an error (or cached fallback data).
↳
This protects Service A from thread starvation and gives Service B time to recover without being hammered by traffic.
▲ LIVE INCIDENT
The payment gateway is returning 503 Service Unavailable, and the entire checkout flow is timing out.
sre@arena:~/resiliency-patterns
sre@arena
:
~
$
istioctl proxy-config cluster checkout-service | grep payment
▶ EXECUTE COMMAND