A load balancer sits in front of a pool of identical service instances and decides, for each incoming request, which instance should handle it. Simple in concept, subtle in practice. The choice of algorithm and the health-checking strategy determine whether the system stays fast, balanced, and available.
The algorithms
Round-robin sends each request to the next instance in rotation. It assumes instances are equally capable and stateless. It is the default for a reason: it is simple, cheap, and fair when conditions are uniform. When they are not, it produces visible imbalance.
Least connections sends the request to the instance with the fewest active connections. It adapts to uneven request cost — a slow endpoint won't accumulate extra load the way round-robin would. But it can amplify hotspots if many connections arrive at once before the balancer observes the load.
IP hash maps the client's source IP to an instance, pinning a client to a server. It provides session affinity without sticky cookies, but a few large IP ranges (corporate NATs, mobile carriers) can overload a single instance.
Consistent hashing places instances and keys on a hash ring. A request maps to the next instance clockwise from its hash. When an instance joins or leaves, only the keys in the affected arc move — the rest stay put. This is the algorithm behind sharded caches, partitioned databases, and any system that needs to move work incrementally rather than reshuffling everything.
Health checks: the silent failure detector
The balancer must know which instances are alive. Active checks open a connection or send a request on a schedule and mark the instance down if it fails. Passive checks observe real traffic — a stream of 5xx responses or timeouts trips the circuit. Active checks detect failure faster but add traffic; passive checks are free but only react to real user impact.
The hard case is the flapping instance — one that accepts connections but returns errors. A naive active check that only tests connectivity marks it healthy, and half the traffic succeeds while half fails. The check path must exercise the real dependency stack, not just the port.
A common mistake is to assume health is binary. A server might be up but degraded — slow database, exhausted thread pool, throttled upstream. Good load balancers support graduated responses: weight reduction, circuit-breaking, or draining, rather than a hard up/down flip.
Termination modes
- Layer 4 (TCP): the balancer forwards bytes without inspecting HTTP. Lowest latency, can route any TCP protocol, but no path-based routing or header manipulation.
- Layer 7 (HTTP): the balancer terminates TLS, reads headers, and routes by path, host, or cookie. Enables canary deployments, A/B testing, and fine-grained routing — at the cost of one extra TLS handshake unless TLS passthrough is used.
- Direct-server-return (DSR): responses bypass the balancer, saving a hop. More complex to configure; connections can be asymmetric.
The choice depends on what you need to inspect. If you only need to distribute TCP, stay at Layer 4. If you need path-based routing, canary deploys, or response rewriting, Layer 7 is required.
Connection management
A balancer holds a pool of upstream connections. Keep-alive reuses connections to reduce handshake overhead. But connection pool exhaustion is silent: when all slots are taken, new requests queue and eventually time out. The queue depth is a load indicator the load balancer itself should expose.
Drain on shutdown: when an instance is removed, the balancer should stop sending new requests immediately but allow in-flight requests to complete. Killing connections in flight causes partial failures that look like bugs to the user.
The request in context
See the end-to-end path in the browser request flow, and trace how a request flows through a load-balanced service with health checks and cache lookups in the load-balanced service trace. The design tradeoff — when to use round-robin vs. consistent hashing — is tested in the rate limiter practice.