Concurrent I/O: Threads and asyncio

Scenario

Example app: a dashboard API function gathers a customer's data from five sources: three DynamoDB tables, an internal HTTP service and S3. Each call takes 150 to 400 ms.

What you see in production

The endpoint takes about 1.2 seconds even though no single call takes more than 400 ms, and the function is billed for the whole time it spends waiting.

The naive code

def handler(event, context):
    profile = profiles.get_item(Key=key(event))
    orders = orders_table.query(**orders_query(event))
    tickets = tickets_table.query(**tickets_query(event))
    usage = http.get(f"{USAGE_URL}/{event['id']}", timeout=2).json()
    invoice = s3.get_object(Bucket=BUCKET, Key=invoice_key(event))
    return build_dashboard(profile, orders, tickets, usage, invoice)

Reproduce it

Simulated latencies for the five calls, run sequentially versus concurrently (a simulated clock keeps the output deterministic):

CALLS = {"profile": 150, "orders": 300, "tickets": 220, "usage": 400, "invoice": 180}   # ms

def sequential(calls):
    return sum(calls.values())

def concurrent(calls, max_workers):
    # each worker takes the next call when free; total = when the last worker finishes
    workers = [0] * max_workers
    for ms in sorted(calls.values(), reverse=True):
        i = workers.index(min(workers))
        workers[i] += ms
    return max(workers)

print("sequential:            ", sequential(CALLS), "ms")
print("concurrent, 5 workers: ", concurrent(CALLS, 5), "ms")
print("concurrent, 2 workers: ", concurrent(CALLS, 2), "ms")

sequential:             1250 ms
concurrent, 5 workers:  400 ms
concurrent, 2 workers:  670 ms

Root cause

Each call blocks the single Python thread while it waits for the network. The total is the sum of all latencies, even though the calls are independent. Lambda bills wall-clock duration, so waiting costs money too.

The fix

Run independent I/O concurrently. boto3 is synchronous and its clients are thread-safe, so a thread pool is the simplest option:

from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=8)      # module level, reused across invocations

def handler(event, context):
    futures = {
        "profile": executor.submit(profiles.get_item, Key=key(event)),
        "orders": executor.submit(orders_table.query, **orders_query(event)),
        "tickets": executor.submit(tickets_table.query, **tickets_query(event)),
        "usage": executor.submit(fetch_usage, event["id"]),
        "invoice": executor.submit(s3.get_object, Bucket=BUCKET, Key=invoice_key(event)),
    }
    results = {name: f.result(timeout=3) for name, f in futures.items()}
    return build_dashboard(**results)

For async-native libraries (httpx, aioboto3), asyncio.gather inside asyncio.run(...) in the handler works too. Always set timeouts on every call, and make sure all futures finish before returning (the environment freezes when the handler returns).

Trade-offs

• Note: boto3 resources are not thread-safe; share clients across threads, or create one resource per thread.
• More concurrency means more simultaneous connections to downstreams; bound max_workers.
• Lambda's CPU scales with memory; CPU-bound work does not speed up with threads (the GIL), only I/O does.

Detect it in production

• Traces (X-Ray or OpenTelemetry) showing calls laid end to end instead of overlapping.
• Duration close to the sum of downstream latencies.
• Timeouts that appear when one dependency slows down, because nothing else could proceed.

Key points

• Sequential independent calls cost the sum of their latencies.
• Use a thread pool for boto3 (clients are thread-safe), or asyncio with async libraries.
• Bound concurrency, set timeouts, and finish all work before returning.

Further reading: boto3: multithreading and multiprocessing · Python docs: concurrent.futures