Concurrency Inside a Java Function

Scenario

Example app: a Java function enriches each incoming order by calling four services. To speed it up, a developer creates an ExecutorService with 8 threads inside the handler and submits the four calls.

What you see in production

Responses are faster, but after a few hundred invocations in a long-lived environment the function slows down and eventually fails with OutOfMemoryError: unable to create native thread. Some enrichment results also belong to the wrong order.

The naive code

private Order current;                              // shared field written per request

public Order handleRequest(Order order, Context ctx) {
    current = order;
    ExecutorService pool = Executors.newFixedThreadPool(8);   // new pool every invocation, never shut down
    var futures = List.of(pool.submit(() -> pricing(current)), pool.submit(() -> stock(current)),
                          pool.submit(() -> loyalty(current)), pool.submit(() -> shipping(current)));
    return merge(order, futures);
}

Reproduce it

import java.util.concurrent.*;

public class Main {
    public static void main(String[] args) throws Exception {
        int invocations = 50;
        int liveBefore = Thread.activeCount();

        for (int i = 0; i < invocations; i++) {                  // naive: a pool per invocation, never shut down
            ExecutorService pool = Executors.newFixedThreadPool(4);
            for (int t = 0; t < 4; t++) pool.submit(() -> 42).get();
        }
        System.out.println("threads leaked by per-invocation pools: " + (Thread.activeCount() - liveBefore));

        try (ExecutorService shared = Executors.newVirtualThreadPerTaskExecutor()) {   // Java 21
            for (int i = 0; i < invocations; i++) {
                var a = shared.submit(() -> 1); var b = shared.submit(() -> 2);
                if (a.get() + b.get() != 3) throw new IllegalStateException();
            }
            System.out.println("virtual-thread executor handled " + invocations + " invocations with no pooled platform threads to leak");
        }
        System.exit(0);   // the leaked non-daemon threads would otherwise keep this demo JVM alive
    }
}

threads leaked by per-invocation pools: 200
virtual-thread executor handled 50 invocations with no pooled platform threads to leak

Root cause

The execution environment and its JVM live across invocations. A thread pool created per invocation and never shut down keeps its threads alive, so threads and memory accumulate until the process can create no more. Writing request data into instance fields shared by concurrent tasks (or later invocations) mixes data between requests.

The fix

• Create executors once (static or constructor) and reuse them, or use structured, per-request scopes that always close (try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { ... } on Java 21).
• On Java 21 runtimes, virtual threads make blocking I/O fan-out cheap without sizing a pool; keep downstream concurrency bounded (for example with a Semaphore).
• Pass request data to tasks as parameters; avoid mutable instance fields for request state.
• Wait for all tasks (with timeouts) before returning, because the environment freezes after the handler returns.
• Alternatively use the SDK's async clients (CompletableFuture) for concurrent AWS calls.

public class EnrichHandler implements RequestHandler<Order, Order> {
    @Override
    public Order handleRequest(Order order, Context ctx) {
        try (var scope = Executors.newVirtualThreadPerTaskExecutor()) {
            var price = scope.submit(() -> pricing(order));
            var stock = scope.submit(() -> stock(order));
            var loyalty = scope.submit(() -> loyalty(order));
            var shipping = scope.submit(() -> shipping(order));
            return merge(order, price.get(2, SECONDS), stock.get(2, SECONDS), loyalty.get(2, SECONDS), shipping.get(2, SECONDS));
        } catch (Exception e) {
            throw new EnrichmentException(order.id(), e);
        }
    }
}

Trade-offs

• Virtual threads remove pool sizing but not downstream limits; unbounded fan-out can still overload dependencies.
• Async SDK clients add callback/future complexity; virtual threads keep code sequential-looking.
• Some libraries pin virtual threads (synchronized blocks around I/O); check on your Java version.

Detect it in production

• Thread counts and memory growing across invocations (JFR, or Max Memory Used trending upwards per environment).
• unable to create native thread errors.
• Data mix-ups between requests, which indicate shared mutable state.

Key points

• The JVM outlives invocations: never create unbounded resources per invocation.
• Reuse executors or use closing scopes; virtual threads for I/O fan-out on Java 21.
• Pass request data as parameters; wait for all tasks before returning.

Further reading: JEP 444: Virtual Threads · Best practices for working with Lambda functions