System Design Advanced: Consensus, Sharding, and Failover

Lesson promise

By the end of this lesson, the learner should be able to: explain how Raft uses terms, votes, and quorums to elect a leader, design a consistent hashing scheme with virtual nodes, and trace how a circuit breaker trips and triggers a cross-region failover.

Narration draft

Start with consensus: why distributed systems need agreement, and why it costs a majority quorum. Walk through Raft's leader election — a follower times out, becomes a candidate, requests votes, and wins when it collects a majority. Show how the randomized election timer prevents split votes, and how the log-up-to-date rule prevents stale leaders.

Then explain consistent hashing: the ring, clockwise key assignment, and why adding one server moves only 1/N of the keys. Show virtual nodes smoothing out uneven distribution, and explain why this same technique appears in load balancers and distributed queues.

Finish with resilience: the circuit breaker's closed/open/half-open states, how health checks feed the breaker, and how a tripped breaker triggers failover. Walk the cross-region path: health check failure → breaker trips → failover controller promotes the passive region → DNS/routing updates → traffic shifts. End with the recovery: the primary region comes back, and failback happens safely.

Visual sequence

1. Animate a 5-node Raft cluster: leader stops heartbeats, follower times out, candidate requests votes, wins majority, becomes new leader.
2. Show the consistent hashing ring: add a 6th node, animate only the keys in the new arc moving. Highlight virtual nodes.
3. Show the circuit breaker state machine and the cross-region failover flow side by side.

Companion material

Use the consensus article, sharding article, circuit breaker article, Raft election trace, cross-region failover trace, multi-region diagram, and the System Design Advanced learning path as written references. The video remains planned until media and transcript are produced.