A distributed system is a system designed to fail. The question is not whether a dependency will be slow or unavailable, but when. Circuit breakers and bulkheads are the two patterns that contain failures: the circuit breaker stops sending requests to a failing dependency, and the bulkhead prevents that failure from consuming resources needed by the rest of the system.
The circuit breaker: fail fast
A circuit breaker wraps a downstream call. It has three states:
Closed — requests flow normally. The breaker counts successes and failures. If the failure rate exceeds a threshold (e.g., 50% of requests fail within 10 seconds), the breaker trips.
Open — the breaker stops sending requests to the failing dependency. Every call returns an error immediately, without network cost. This is the "fail fast" state: the caller knows the dependency is down before waiting for a timeout.
Half-open — after a cooldown period (e.g., 30 seconds), the breaker allows a single probe request through. If it succeeds, the breaker closes. If it fails, the breaker reopens. This allows the system to recover automatically when the dependency comes back.
The circuit breaker is not a retry mechanism. It does not retry the request; it short-circuits it. Retries happen at a higher level, with backoff and jitter, only after the breaker has confirmed the dependency is healthy again.
The bulkhead: resource isolation
A bulkhead divides system resources — thread pools, connection pools, memory — into independent compartments. If the payment-processing compartment is overloaded, it does not consume threads from the user-profile compartment. Each compartment has its own circuit breaker, failure budget, and alerting.
The name comes from ship design: a breach in one compartment does not sink the ship. In software, the compartments are thread pools, connection pools, or even separate processes. Without bulkheads, a slow dependency exhausts the entire thread pool, and every request — including health checks and unrelated endpoints — starts to fail. That is a cascade.
Timeouts: the first line of defense
Every call to a dependency must have a timeout. The timeout must be shorter than the caller's own deadline — if the API gateway allows 5 seconds, the database call should time out at 2 seconds, leaving headroom for the circuit breaker to react and the caller to fall back.
Timeouts that are too short cause unnecessary failures; timeouts that are too long cause cascades. The right timeout is not a guess — it is the 99th percentile of the dependency's response time under load, plus a margin for variance.
The connection to queues
The distributed queues article describes asynchronous communication, which is the complementary approach to circuit breakers: instead of failing fast on a slow dependency, queue the request and process it later. Queues and circuit breakers are not alternatives — a queue with a bounded depth and a circuit breaker that trips when the queue fills is a robust combination. The cross-region failover trace shows how circuit breakers drive the failover decision: when the circuit opens, traffic shifts to the backup region.
The latency and throughput primer explains why the coordination cost of circuit breakers and retries matters: each retry multiplies the load on the failing dependency, which is why good circuit breakers use jittered backoff and why bulkheads bound the number of concurrent retries.