This trace follows a failover from the perspective of the routing layer and circuit breaker. Read the circuit breaker article first for the failure-detection model. The multi-region architecture diagram shows the same path visually.
1. Health checks pass in the primary region
The healthy region serves all traffic. Health checkers poll /health endpoints every few seconds. They observe normal response rates and latencies. The circuit breaker for the primary region is closed — traffic flows normally.
2. Health checks start failing
An incident occurs — a database outage, a network partition, or a cascading failure. Health checks begin returning 5xx errors or timing out. The failure rate climbs above the circuit breaker threshold. The circuit breaker records the degradation.
3. The circuit breaker trips
After the failure threshold is breached (e.g., 50% of checks fail within 10 seconds), the circuit breaker transitions from closed to open. New requests to the primary region are rejected immediately without network cost. This is the fail-fast state — the system has decided the primary is unavailable.
4. The failover controller promotes the passive region
A failover controller (external to the application, often a Kubernetes operator or a cloud route-53 routing policy) receives the signal that the primary circuit is open. It promotes the passive region: read replicas are allowed to accept writes, the standby database is brought to a writable state, and the region's capacity is scaled up if needed.
The promotion step is the most dangerous — it can only happen if the passive region's data is at least as fresh as the primary's last committed write. If replication lag exceeds the acceptable data-loss window, the failover is delayed or aborted.
5. DNS and routing update
The failover controller updates DNS records, load balancer target groups, or service mesh routing rules to point traffic to the backup region. Depending on the TTL, this can take seconds to minutes to propagate globally.
6. Traffic shifts to the backup region
Requests now arrive at the backup region. The backup's application servers are healthy and ready. The failover is complete — from the user's perspective, the service remained available (or degraded briefly), but did not go down entirely.
7. The primary region begins recovery
The incident in the primary region is resolved. Health checks return to normal. The circuit breaker begins its half-open cooldown. When the primary region is healthy again, the system can either shift traffic back (failback) or keep traffic on the backup until the next decision point. Failback is more complex than failover — it requires a second data-sync and a second traffic shift, and must not cause a brief outage during the transition.
The trace is complete when traffic is flowing to a healthy region and the failed region has either recovered or been decommissioned.