System design is measured in time. How long does a request take? How many requests can the system handle? How much slower does it get when you need two components to agree on something? The answers come down to three quantities: latency, throughput, and coordination cost.
Latency: the cost of one operation
Latency is how long a single request takes from the caller's perspective. It is not an average. It is not even a single number — it is a distribution. A service that responds in 10 ms on average but sometimes takes 5 seconds has a tail-latency problem that averages cannot hide.
For web services, latency is dominated by the slowest hop in the request path. A database query that takes 2 ms is irrelevant if the network round-trip between your service and the database takes 1 ms and the cache lookup on the hot path takes 8 ms. Every component on the critical path adds its own tail, and the worst tail of each component multiplies the overall tail. That is why reducing the number of components on the critical path matters more than shaving milliseconds off any single hop.
Throughput: how often you can do it
Throughput is how many operations the system can complete per unit of time. A single-threaded service processing requests in 10 ms each can handle about 100 requests per second — regardless of how fast the downstream database or cache is. Throughput is limited by the slowest bottleneck in the pipeline, which may not be the component you think it is.
Throughput and latency are coupled. At low load, latency is low because requests flow smoothly. As load approaches capacity, queues build up, latency rises, and eventually the system spends more time waiting than working. The operating point — the fraction of capacity you sustain — determines both your latency and your ability to absorb bursts.
The coordination tax
Most system design problems are not about doing work faster. They are about doing work together. Coordination — making multiple components agree — is where every millisecond goes that you did not plan for.
A write to a single database instance is a network round-trip. A write to a replicated database that requires majority acknowledgment is a network round-trip multiplied by the slowest replica in the quorum, plus the time to resolve any conflicts. A distributed transaction is coordination multiplied by the number of participants, each of which can fail independently.
The rule of thumb: every consensus step costs roughly one network round-trip, and every network round-trip costs at least the speed of light across the physical distance between the participants. Between New York and London that is about 30 ms. Between New York and San Francisco it is about 40 ms. You cannot make the speed of light slower, and you cannot make it irrelevant when nodes must agree.
This is why single-region systems are faster than multi-region systems, why caches that avoid coordination are faster than writes that require it, and why the hardest system design decisions are the ones that reduce the need to coordinate at all.
Putting a number on it
Before you draw any architecture, estimate the numbers your system will live with:
- Local memory access: 100 nanoseconds
- Local disk seek: 10 milliseconds
- Cross-AZ network round-trip: 1-2 milliseconds
- Cross-region network round-trip: 50-100 milliseconds
- A single write to a strongly-consistent leader: one round-trip to the quorum
- A read from a cache hit: sub-millisecond
- A read that misses the cache and hits the database: one round-trip to the database
These are orders of magnitude apart. A cache hit that avoids a database round-trip saves not milliseconds but tens of milliseconds — enough to change from a fast-feeling service to a slow one. Every factor of ten on this scale is a user-experience cliff.
See the end-to-end path in the browser-to-server flow diagram, and trace how a cache hit or miss plays out in the cache trace.