Consider a user updating their profile. The write goes through a leader node, which must persist the change to its write-ahead log, apply it to the in-memory page, replicate to followers, and acknowledge to the client. Each step carries a latency cost and a failure mode.
1. The application sends a write
The service begins a transaction: UPDATE users SET name = 'New Name' WHERE id = 42. This is not the same as committing — the change is buffered, not visible to other transactions yet. The transaction's isolation level (read committed, serializable) determines when other readers see it.
2. The leader appends to the WAL and fsyncs
Before the change is durable, the leader writes the transaction's intent to the write-ahead log and calls fsync. This is a synchronous disk operation — 1-10 ms on most hardware. Until this completes, a crash could lose the transaction entirely. This fsync is the floor for synchronous write latency.
3. The leader applies the change
The WAL entry is applied to the in-memory data page. If the page is full, a page split may occur, which is more expensive. The change is now visible to queries on the leader, even before replication completes.
4. The leader streams the log entry to followers
The leader pushes the WAL record over a streaming replication connection. This is a network hop — fast within a region (1-2 ms), slower across regions (50+ ms). The follower receives and queues the record.
5. The follower applies and acks
The follower writes the record to its own WAL, fsyncs, applies it, and sends an acknowledgment back. Until the follower applies it, reads from that follower will return stale data.
6. Client acknowledgment
If the client requested synchronous replication, the leader waits for at least one follower's ack before confirming commit. If asynchronous, the leader returns immediately and the follower catches up in the background. The choice determines whether a follower lag of 5 seconds means 5 seconds of potentially lost data.
This write path is the other half of the system: the request-through-load-balanced-service trace shows the read path; this trace shows the write path. The deadlock trace shows what happens when transactions conflict.