A queue sits between a producer and a consumer, decoupling them in time. The producer sends a message and moves on; the consumer processes it when ready. This simple idea hides a spectrum of delivery guarantees, each with a cost. The system design question is which guarantee the product actually needs.
The three delivery semantics
At-most-once: the producer sends the message once; if the consumer does not receive it, it is lost. No retries, no deduplication. Cheap and fast, but unreliable. Useful for telemetry where losing a metric is acceptable.
At-least-once: the producer retries until the consumer acknowledges. The consumer may receive the same message multiple times. This is the default for most message queues — reliability over perfection, with the understanding that consumers must be idempotent.
Exactly-once: the system guarantees the message is processed exactly once, even in the face of failures. In practice, this means idempotent processing combined with de-duplication at the consumer. True exactly-once across independent systems is impossible without consensus — what looks like exactly-once is usually at-least-once with idempotent consumers and deduplication IDs.
Visibility timeout and consumer leases
When a consumer picks up a message, the queue must hide it from other consumers for a window — the visibility timeout. If the consumer finishes before the timeout, it acknowledges and the message is deleted. If it crashes, the message reappears after the timeout and another consumer picks it up.
The timeout is a tension: too short, and the message is redelivered while the consumer is still working — duplicate processing. Too long, and recovery from a crashed consumer is delayed — reduced throughput. Sizing the timeout to the slowest plausible processing time is a design decision.
Dead-letter queues and poison pills
A poison pill is a message that always fails — perhaps malformed, perhaps referencing a dependency that is permanently broken. Without intervention, a consumer will pick it up, fail, and the message will reappear, forever. A dead-letter queue catches messages that fail N times and moves them aside so the main queue can progress.
The dead-letter queue is also a debugging surface: messages land there with their failure history, a structured record of what went wrong.
Ordering guarantees
A queue that preserves order across partitions is a queue that serializes all messages — the antithesis of scaling. A partitioned queue can deliver messages in order within a partition but not across partitions. If the consumer cares about order, it must route related messages (same user, same account) to the same partition.
This is the same consistent-hashing problem from the load balancer article: the routing key determines how work is partitioned. A cache key that ignores tenant context leaks data between users; a queue partition key that ignores request ordering loses sequence guarantees.
For a hands-on exercise in rate limiting — a sibling problem in traffic control — see the rate limiter practice.