Logging, Tracing and Retries

Scenario

Example app: a Java function processes payment notifications from EventBridge and calls a partner API through an HTTP client configured with 3 retries. Logging uses System.out.println and a mix of Log4j and Logback.

What you see in production

During a 20-minute partner outage, the partner reports receiving far more requests than usual and blocks the integration. Afterwards, nobody can find which notifications failed: the logs are multi-line stack traces split across many CloudWatch entries with no request IDs.

The naive code

public Void handleRequest(ScheduledEvent event, Context ctx) {
    System.out.println("processing " + event.getDetail());
    partner.notify(event.getDetail());     // HTTP client: 3 retries; EventBridge target and Lambda async retries on top
    return null;
}

Reproduce it

public class Main {
    public static void main(String[] args) {
        int eventbridgeAttempts = 3;     // target retry policy (attempts reaching the function)
        int lambdaAsyncAttempts = 3;     // 1 try + 2 Lambda async retries
        int httpClientAttempts = 3;      // client retries inside each invocation

        int perEvent = eventbridgeAttempts * lambdaAsyncAttempts * httpClientAttempts;
        System.out.println("partner calls per failed event, all layers retrying: " + perEvent);
        System.out.println("events in outage window: 2,000 -> partner calls: " + String.format("%,d", 2000 * perEvent));
        System.out.println("one retry layer with backoff (3 attempts): " + String.format("%,d", 2000 * 3));
    }
}

partner calls per failed event, all layers retrying: 27
events in outage window: 2,000 -> partner calls: 54,000
one retry layer with backoff (3 attempts): 6,000

Root cause

Every layer that retries multiplies attempts: the event source or async invocation, the function's own loop and the HTTP or SDK client. During an outage this turns a struggling dependency into a crushed one. Separately, System.out with multi-line stack traces produces unstructured, fragmented logs without correlation fields.

The fix

• Decide which layer retries: usually the event source (with a DLQ or failure destination) for asynchronous work, with SDK/HTTP clients limited to one quick retry and short timeouts. Use exponential backoff with jitter and a circuit breaker (for example Resilience4j) for flaky partners.
• Use Powertools for AWS Lambda (Java) Logging for structured JSON logs (with Log4j2 or Logback via SLF4J) including request ID, cold start, and custom keys such as the payment ID; Tracing for X-Ray subsegments; Metrics for business counters.
• Configure Lambda's JSON log format and log levels, and set retention on log groups.

public class PaymentHandler implements RequestHandler<ScheduledEvent, Void> {
    private static final Logger LOG = LoggerFactory.getLogger(PaymentHandler.class);

    @Logging(logEvent = false)
    @Tracing
    @Override
    public Void handleRequest(ScheduledEvent event, Context ctx) {
        String paymentId = (String) event.getDetail().get("payment_id");
        MDC.put("payment_id", paymentId);                 // appears as a field in every JSON log line
        try {
            partner.notify(event.getDetail());            // client: 1 retry, 2 s timeout, circuit breaker
            LOG.info("partner notified");
        } finally {
            MDC.remove("payment_id");
        }
        return null;                                      // failures propagate: EventBridge/Lambda retry, then DLQ
    }
}

Trade-offs

• Fewer retries mean more events in the DLQ during outages; that is a feature: replay after recovery instead of hammering the partner.
• Structured logging adds a dependency and some start-up cost; keep log volume in check.
• Circuit breakers need tuning; a breaker that opens too easily causes avoidable failures.

Detect it in production

• Calls to dependencies per incoming event; retry counts in client metrics.
• DLQ depth and failure destination messages.
• Log queries by business key (payment ID) across functions.

Key points

• Retries multiply across layers; choose one main retry layer and bound the others.
• Backoff with jitter, timeouts and circuit breakers for partners.
• Structured JSON logs with Powertools (Java) and correlation keys.

Further reading: Powertools for AWS Lambda (Java) · Lambda asynchronous invocation · Amazon Builders' Library: Timeouts, retries and backoff with jitter