In a single process, a function call either returns or raises an error that the caller can observe. Across a network, a request can reach a server and its response can be lost. The client sees a timeout but cannot infer whether the operation happened. This uncertainty is one of the basic facts of distributed systems.
Failure detectors are suspicions
A timeout is a local decision to stop waiting. It does not prove that a remote machine is dead; the machine may be slow, partitioned, overloaded, or replying along a delayed route. Systems use timeouts to make progress, then design retries, reconciliation, or compensation around uncertain outcomes.
Messages may be delayed, duplicated, reordered, or dropped depending on the transport and application. Reliable delivery mechanisms still need application-level rules for duplicate operations and durable state. Use idempotency keys or deduplication where repeating a logical action must not repeat its effect.
Replication introduces consistency choices
Replicas can improve availability and read capacity, but changes must move between copies. A read from a lagging replica can return an older value. Quorums, leaders, and conflict-resolution policies offer different guarantees and costs. State the guarantee in terms a caller can observe, such as whether a successful write is immediately visible to subsequent reads.
Consensus protocols coordinate a replicated log despite some failures under stated assumptions. Raft separates leader election, log replication, and safety rules so a cluster can agree on an ordered history. It does not make arbitrary network partitions disappear or guarantee progress when the required quorum is unavailable.
Design for the uncertainty
Specify operation semantics, timeout budgets, retry rules, ordering requirements, and recovery behavior. Test crashes between steps, lost acknowledgements, duplicate delivery, and stale reads. The Raft paper is a primary reference for one replicated-state-machine consensus design.