Most system design problems are about doing work in one place. Distributed consensus is about doing work in many places and keeping them in agreement. The cost of that agreement — round-trips, quorums, failure detection — is the price of surviving node failures. The design question is whether the system can tolerate that price, or whether it should avoid consensus entirely by accepting weaker guarantees.
When do you need consensus?
You need consensus when two or more nodes must agree on a value, and it is better to stop making progress than to disagree. Classic cases:
- Configuration changes — "which replica is the leader?" If two nodes disagree, both may accept writes, and the system splits in two.
- Transaction commit — "did this write succeed?" A two-phase commit asks every participant to vote; a majority quorum decides.
- Metadata changes — "what is the schema?" If nodes have different schemas, they will produce different results.
You do not need consensus when you can tolerate temporary inconsistency. A cache that is eventually consistent, a queue that delivers at-least-once, or a service that degrades to stale data during a partition — these avoid the coordination cost entirely. The system design choice is whether the product needs the stronger guarantee or can live with the weaker one.
Raft's approach: terms, votes, and quorums
Raft reduces consensus to three understandable mechanisms:
Leader election: Each node tracks a monotonically increasing term (a logical clock). When a follower stops hearing from its leader (election timeout, typically 150-300 ms), it increments its term, votes for itself, and requests votes from others. A candidate wins by collecting a majority of votes. A majority is the quorum — for N nodes, you need ⌊N/2⌋ + 1.
Log replication: Once elected, the leader receives all writes, appends them to its own log, and pushes them to followers via AppendEntries RPCs. An entry is considered committed once replicated to a majority. This means a minority of followers can be down, and the system continues — but if a majority is lost, writes stop.
Safety rules: A candidate's log must be at least as up-to-date as the voter's log before it receives a vote. This prevents a stale node from winning an election and overwriting newer data. The leader can only commit entries from its own term, not older terms, unless a later entry from its term has been replicated.
The core tension: Raft trades write availability for consistency. A network partition that splits the cluster into two minorities means neither can reach majority, and no writes succeed. This is the CAP theorem in action — you cannot have consistency and availability simultaneously across a partition.
The leader election trace
The Raft leader election trace follows the exact state changes: a leader stops sending heartbeats, followers time out, a candidate emerges, votes are exchanged, and a new leader is chosen. The existing consensus trace covers the normal write path through a stable leader. Together they show both the steady state and the failure recovery.
The cost of consensus is paid in round-trips. As the latency and throughput primer explains, each round-trip between data centers adds tens of milliseconds. A Raft write that requires a majority ack across three regions costs at least two cross-region round-trips. That is the fundamental ceiling that no amount of caching can remove.