This trace follows a leader failure and recovery in a 5-node Raft cluster. The normal write path trace covers the steady state under a stable leader; this trace covers the failure case. Read the consensus article first for the model.
1. The leader stops sending heartbeats
The leader normally sends AppendEntries heartbeats every 50-100 ms to assert liveness. When the leader crashes or the network between leader and followers fails, no heartbeats arrive. Followers do not know whether the leader is dead or the network is partitioned — they can only observe silence.
2. A follower's election timer fires
Each follower has an election timer, randomized between 150-300 ms to reduce the chance that multiple followers time out simultaneously and cause a split vote. When the timer fires, the follower concludes the leader is dead (or unreachable) and begins a new election.
3. The follower becomes a candidate
The follower increments its current term, votes for itself, and transitions to the candidate state. It immediately sends RequestVote RPCs to all other nodes, each carrying its term, its last log index, and its last log term (for the log-up-to-date check).
4. The candidate requests votes
Each RequestVote RPC includes the candidate's term, the index and term of the last log entry. A recipient grants its vote only if:
- The candidate's term is at least as large as the recipient's current term.
- The candidate's log is at least as up-to-date as the recipient's.
A node grants at most one vote per term.
5. A majority of votes are collected
The candidate wins when it receives votes from a majority of the cluster (3 of 5). The majority requirement is what prevents split-brain: two candidates cannot both reach majority, because the majorities must overlap on at least one node, and that node can only vote for one candidate per term.
6. The candidate becomes the new leader
Upon winning, the candidate transitions to leader. It initializes its nextIndex and matchIndex state for each follower, and immediately begins sending AppendEntries heartbeats to establish its authority. Any pending client requests that were sent to the old leader will be retried by clients, who will be redirected to the new leader.
7. The new leader sends heartbeats
The leader's first responsibility is to assert liveness. Followers that have not yet heard from the new leader will time out and start a new election — but the leader's heartbeats reset their timers, keeping the system stable under a single leader.
A split vote (two candidates each get 2 of 5 votes) triggers another election after a randomized backoff. The randomization is what prevents repeated split votes. The trace is complete when the leader is stable, sending heartbeats, and accepting client requests again.