Reliability Patterns
At scale, something is always failing: a disk, a network link, a slow dependency. Reliable systems aren't the ones where nothing breaks; they're the ones where a failure stays small. These patterns come up in every deep-dive: "what happens when this service is slow?"
- C4.1Timeouts and cascading failuresWithout a timeout, a call to a slow dependency waits forever. Each waiting request holds a thread or connection. When they're all used up,…
- C4.2Retries with exponential backoff and jitterMany failures are transient (a dropped packet, a restarting pod, a 503). Retrying fixes them, but naive retries make outages worse: if…
- C4.3Idempotency keysA client sends POST /payments. The server charges the card, but the response is lost on the way back (timeout). Did it work? The client…
- C4.4Circuit breakersIf a dependency is down, retrying every request just wastes time and piles load onto it. A circuit breaker wraps the call and tracks…
- C4.5Bulkheads, load shedding and graceful degradationBulkheads: give each dependency its own connection pool / thread pool (like watertight compartments in a ship), so one slow dependency can…