A strong answer
I would first clarify the scope: is this about a hardware load balancer, an L7 service mesh, or an in-application client-side load balancer? The mechanisms differ, but the principles are the same.
1. Active health checks — The load balancer periodically sends a request to a health endpoint (e.g., GET /health). The endpoint must exercise the real dependency stack, not just return 200. A check that only tests the port being open will never catch a slow database or an exhausted thread pool.
2. Passive health checks — The balancer observes real traffic for error rates, latency spikes, and connection failures. Passive checks are free (no extra traffic) but reactive — the first users see the failure. They complement active checks by catching problems that synthetic checks miss.
3. Flapping detection — A backend that oscillates between healthy and unhealthy causes request failures and cache churn. Good load balancers dampen this with hysteresis: multiple consecutive failures before removing, multiple consecutive successes before re-adding.
4. Graduated responses — Instead of a binary in/out of the pool, a smart system can reduce weight, drain connections, or route around a degraded instance. This matters because a backend can be "up" but "degraded" — accepting connections but serving errors.
5. The check endpoint must be meaningful. A /health endpoint that always returns 200 is worse than useless — it creates a false sense of availability. It should check the database connection, the cache, and any critical upstream dependency. But it must also fail fast — a health check that hangs for 10 seconds blocks the entire health-check cycle.
The trace in request-through-load-balanced-service shows how health-check state feeds into the backend selection step. Pair this with the load balancer article and the circuit-breaker concepts in the database write trace.