The Runtime Theory
Backend Engineering

Retries Need an Operation Contract

Design timeouts, retries, idempotency keys, and server-side state so a lost response does not silently duplicate an operation.

The Runtime Theory Team7 min read#api-design#retries#idempotency#reliability
▸ On this page

A timeout tells a client it did not receive a response in time. It does not tell the client whether the server performed the operation. The server may have completed a payment, written a row, or started a job just before the response was lost.

Retry safety follows semantics

Repeating a read is often safe, though its result may have changed. Repeating a write can create duplicate effects unless the API defines what a retry means. HTTP method semantics provide useful guidance, but an application must still design its operation contract around real side effects.

An idempotency key lets a client identify retries of one logical operation. The server scopes the key to an account or caller, records enough request identity to reject accidental reuse with different parameters, and stores the outcome so a duplicate can receive a consistent result. Recording the key and applying the effect must be coordinated; otherwise a crash between them can reintroduce duplication or lose work.

Timeouts and backoff are load controls

Set timeouts for the whole operation and for individual dependency calls. Retry only failures that may recover, cap attempts, use backoff with jitter, and avoid synchronized retry storms. If several layers retry independently, a single user action can amplify into many downstream requests. Circuit breakers, bulkheads, and queue limits can prevent one unhealthy dependency from consuming the whole service.

Keep the boundary visible

Backend reliability also depends on input validation, authorization before side effects, transaction boundaries, and useful logs. A trace ID helps connect a client request to downstream work, but it is not an idempotency key: tracing identifies an execution path; idempotency identifies the logical action across retries.

The AWS Builders' Library guide to idempotent APIs describes the ambiguity caused by a lost response and the server-side contract needed for safe retries.

Not started

Sign in to save your learning progress.

Sign in to save