The Anatomy of a Request

Scenario

Example app: a reporting endpoint runs behind a cloud load balancer, then nginx, then an application server (PHP-FPM, Gunicorn or Node.js), then the framework. Some reports take more than a minute to build.

What you see in production

• Users get 504 Gateway Timeout pages after exactly 60 seconds.
• Application logs show the same requests finishing successfully after 65 to 90 seconds, with status 200.
• Users click "Generate" again, so the server builds the same report two or three times, making everything slower.

The naive code

# load balancer idle timeout: 60 s (cloud default)
location / {
    proxy_pass http://app;
    proxy_read_timeout 300s;     # "give reports plenty of time"
}
# application server / framework: no request timeout at all

Reproduce it

A request passes through several layers, each with its own timeout. The simulation sends a 65-second request through two configurations.

function travel(layers, workSeconds) {
  const lines = [];
  // the outermost layer that times out first answers the user; inner layers keep working
  const firstTimeout = layers.filter((l) => l.timeout < workSeconds).sort((a, b) => a.timeout - b.timeout)[0];
  for (const l of layers) {
    const gaveUp = l.timeout < workSeconds;
    lines.push(`  ${l.name.padEnd(18)} timeout ${String(l.timeout).padStart(3)}s -> ${gaveUp ? "gives up" : "waits"}`);
  }
  const userSees = firstTimeout ? `${firstTimeout.status} after ${firstTimeout.timeout}s` : `200 after ${workSeconds}s`;
  const appKeepsWorking = firstTimeout && layers[layers.length - 1].timeout >= workSeconds;
  lines.push(`  user sees: ${userSees}${appKeepsWorking ? "; the app keeps working and logs 200 (wasted work)" : ""}`);
  return lines.join("\n");
}

const misaligned = [
  { name: "load balancer", timeout: 60, status: 504 },
  { name: "nginx", timeout: 300, status: 504 },
  { name: "framework", timeout: 999, status: 500 },
];
const aligned = [
  { name: "load balancer", timeout: 120, status: 504 },
  { name: "nginx", timeout: 110, status: 504 },
  { name: "framework", timeout: 100, status: 503 },
];
console.log("misaligned (65 s report):\n" + travel(misaligned, 65));
console.log("aligned, inner < outer (65 s report):\n" + travel(aligned, 65));
console.log("aligned, report too slow (130 s):\n" + travel(aligned, 130));

misaligned (65 s report):
  load balancer      timeout  60s -> gives up
  nginx              timeout 300s -> waits
  framework          timeout 999s -> waits
  user sees: 504 after 60s; the app keeps working and logs 200 (wasted work)
aligned, inner < outer (65 s report):
  load balancer      timeout 120s -> waits
  nginx              timeout 110s -> waits
  framework          timeout 100s -> waits
  user sees: 200 after 65s
aligned, report too slow (130 s):
  load balancer      timeout 120s -> gives up
  nginx              timeout 110s -> gives up
  framework          timeout 100s -> gives up
  user sees: 503 after 100s

Root cause

A request crosses a chain of components: client → CDN or load balancer → web server (nginx, Apache) → application server (PHP-FPM, Gunicorn/Uvicorn, Node.js, Tomcat) → framework → your code. Each has its own timeout. When an outer layer's timeout is shorter than an inner one, the outer layer returns an error to the user while the inner layers keep working, so the work is wasted and often repeated.

The fix

• Inner timeouts must be shorter than outer ones: framework < app server < web server < load balancer, with a few seconds of margin at each step. Then the innermost layer, which knows what went wrong, produces the error and stops the work.
• Give each request a deadline and pass the remaining budget to outbound calls (database, HTTP), so nested calls stop when the request can no longer succeed.
• Move genuinely long work (reports, exports) off the request path: accept it, return 202 Accepted with a job ID, process it in a queue, and let the client poll or receive a notification.
• Make expensive actions idempotent (an idempotency key per "Generate" click) so retries don't repeat the work.

Relevant settings include nginx's proxy_read_timeout, PHP-FPM's request_terminate_timeout, Gunicorn's --timeout, Node.js server.requestTimeout, and your load balancer's idle timeout.

Trade-offs

• Short timeouts protect capacity but fail slow-but-valid requests; move those to asynchronous jobs instead of raising every timeout.
• Some layers (CDNs, managed load balancers) have hard maximums you can't raise.

Detect it in production

• Compare status codes at each layer: 504s at the load balancer with 200s in app logs mean misaligned timeouts.
• Propagate a request ID (for example X-Request-ID or W3C traceparent) through every layer so one request can be followed end to end.

Key points

• Every request passes through several components, each with its own timeout.
• Make inner timeouts shorter than outer ones so the layer that knows the cause reports it and stops working.
• Move long work to background jobs and use request IDs to trace a request across layers.