This trace follows the actual state transitions behind the companion Distributed Calls Can Fail Halfway Through. It describes a common execution path; implementation details can vary, so keep the contract separate from the mechanism.
Step 1: Send a remote operation
In a distributed system, a caller may lose contact with a process that is still running, or receive no response after the remote side completed its work. A network partition, overloaded queue, crashed process, and slow dependency can look similar from the caller’s vantage point.
Step 2: Allow remote work to continue
A client starts a request with a deadline. If the deadline expires, it cancels local waiting and reports uncertainty; cancellation may or may not stop remote work. Propagating the remaining deadline downstream prevents each hop from independently consuming a full timeout budget.
Step 3: Reach the local deadline
The caller may time out after the remote operation commits; treat the result as unknown until a deduplication record or status query resolves it.
At this point, record the state that changed and check the invariant before advancing. If the operation repeats, make clear which values persist and which are recomputed.
Step 4: Retry with an idempotency key
No timeout means resources can remain tied up indefinitely; an overly short timeout creates false failures and duplicate retries. Logs need request or trace identifiers to correlate attempts, but identifiers do not guarantee exactly-once execution. Design operations to tolerate ambiguous completion.
Step 5: Reconcile the final outcome
A worker times out while charging a card, then retries. Explain why the first charge may have succeeded and how an idempotency record can resolve the ambiguity.
The trace is complete when the result satisfies the stated contract. Compare this model with the concrete runtime or system you are studying before making a performance claim.