Design a rate limiter for an HTTP API. The product needs 1,000 requests per second average with bursts up to 5,000 per second. The limiter must run across multiple service instances.
What to decide
-
Algorithm: A token bucket allows burst absorption up to capacity while constraining the average rate. A fixed-window counter is simpler but has edge effects at window boundaries. A sliding log is precise but memory-expensive.
-
State location: In-process state is fast but per-instance — a client can exceed the global limit by hitting different instances. A shared store (Redis) provides a global limit but adds a network round-trip per request. A hybrid — local burst allowance with global coordination — is common in practice.
-
Where to place it: At the edge (API gateway) catches traffic before it reaches application servers. In the application, it is more flexible but adds cost to every request on the critical path.
Implementation hints
A token bucket has two parameters: the refill rate (tokens per second) and the capacity (maximum burst). On each request, atomically check if a token is available and decrement it. In Redis, this is a Lua script that reads the current count and timestamp, computes refill, and decrements atomically.
The hard part is clock drift across instances. Use a monotonic clock for refill computation, and never trust a timestamp sent by the client. For multi-region deployments, consider allowing each region its own bucket and coordinating only when the total budget is shared.
Read the load-balancing article to see how the limiter sits behind the load balancer, and the token bucket diagram for the visual flow.