Why Retries Are Inevitable — Idempotency & Retry-Safe Systems

In distributed systems a request can fail in three ways: it never reached the server, the server failed while processing, or the server succeeded but the response was lost (timeout, connection reset, crashed load balancer). From the client's side the last two look identical — the outcome is unknown. Clients, SDKs, proxies, message brokers and users (double-clicks, refreshes) therefore retry, and every unsafe operation must expect to be executed more than once.

What a client knows after a failure

Failure	Did the server act?	Safe to retry blindly?
Connection refused / DNS failure	No	Yes
Request timeout	Unknown	Only if idempotent
502/503/504 from proxy	Unknown (often no)	Only if idempotent
500 from the service	Maybe partially	Only if idempotent
Response lost after 201	Yes	Only if idempotent
Broker redelivery	Maybe	Consumer must be idempotent

Common pitfalls

Assuming a timeout means the operation didn't happen.
Disabling retries to 'avoid duplicates' (you just lose operations instead).
Treating double-clicks as a UI problem only.

Key points

Timeouts produce unknown outcomes
Retries come from clients, proxies, brokers and humans
Design unsafe operations to tolerate repeats

Further reading: AWS Builders' Library: Making retries safe with idempotent APIs

Practice questions

Why can't a client know whether a timed-out POST was applied?

Answer

The request may have been processed with only the response lost; without a way to look up or deduplicate the operation, the client can't distinguish 'not applied' from 'applied but unacknowledged'.

Name four sources of retries besides application code.

Answer

HTTP client libraries/SDKs, proxies and load balancers, message brokers redelivering unacked messages, and users double-clicking or refreshing.