Memory, Large Files and /tmp

Scenario

Example app: when a CSV file lands in S3, a function downloads it to /tmp, loads it with pandas.read_csv, transforms it and writes results to DynamoDB. Files range from 1 MB to 2 GB.

What you see in production

Large files fail with Runtime exited with error: signal: killed (out of memory). After a busy day, small files start failing with No space left on device, even though each file is small.

The naive code

import boto3, pandas as pd

s3 = boto3.client("s3")

def handler(event, context):
    bucket, key = location(event)
    s3.download_file(bucket, key, f"/tmp/{key.split('/')[-1]}")   # never deleted
    df = pd.read_csv(f"/tmp/{key.split('/')[-1]}")                  # whole file in memory
    write_rows(transform(df))

Reproduce it

MEMORY_MB, TMP_MB = 1024, 2048                    # ephemeral storage raised to 2 GB "to be safe"
files_mb = [40, 900, 120, 60, 200, 150, 300, 350]  # one environment handles these in turn

def naive(files):
    tmp_used, out = 0, []
    for f in files:
        if tmp_used + f > TMP_MB:
            out.append(f"{f} MB: No space left on device"); continue
        tmp_used += f                               # downloaded, never removed
        peak = f * 3                                # a DataFrame often needs several times the file size
        out.append(f"{f} MB: " + ("OOM killed" if peak > MEMORY_MB else "ok"))
    return out

def streaming(files, chunk_mb=8):
    return [f"{f} MB: ok (peak ~{chunk_mb * 3} MB, /tmp unused)" for f in files]

print("naive:    ", naive(files_mb))
print("streaming:", streaming(files_mb)[:2], "...")

naive:     ['40 MB: ok', '900 MB: OOM killed', '120 MB: ok', '60 MB: ok', '200 MB: ok', '150 MB: ok', '300 MB: ok', '350 MB: No space left on device']
streaming: ['40 MB: ok (peak ~24 MB, /tmp unused)', '900 MB: ok (peak ~24 MB, /tmp unused)'] ...

Root cause

Loading a whole file into a DataFrame needs memory proportional to (and usually several times) the file size, so large files exceed the function's memory. /tmp is ephemeral storage that persists across invocations in the same execution environment; files that are never deleted accumulate until the space (512 MB by default, configurable up to 10 GB) runs out.

The fix

• Stream instead of loading everything: read the S3 object body in chunks (get_object()["Body"].iter_lines()), process row by row with the csv module, and write in batches (batch_writer). With pandas, use chunksize= to process in pieces.
• If you must use /tmp, use a tempfile.TemporaryDirectory() so files are always removed, or configure more ephemeral storage deliberately.
• For very large or long jobs, split the file (S3 byte ranges, or Step Functions Distributed Map over the rows or over many files), or use a service built for it (AWS Glue, ECS tasks).

import csv, io, boto3
s3 = boto3.client("s3")
table = boto3.resource("dynamodb").Table("rows")

def handler(event, context):
    bucket, key = location(event)
    body = s3.get_object(Bucket=bucket, Key=key)["Body"]
    lines = (line.decode("utf-8") for line in body.iter_lines())
    with table.batch_writer() as batch:
        for row in csv.DictReader(lines):
            batch.put_item(Item=transform_row(row))

Trade-offs

• Streaming code is a little more complex than DataFrames and some operations (sorting, joins) need the whole data set.
• More memory also means more CPU, which can make in-memory processing viable for medium files; measure.
• Large /tmp sizes cost money; keep it as small as the workload needs.

Detect it in production

• Max Memory Used close to the configured memory in REPORT lines; signal: killed errors.
• No space left on device errors that appear only after an environment has handled many invocations.
• File size distribution versus failures.

Key points

• Whole-file loading needs memory proportional to file size; stream instead.
• /tmp persists between invocations in an environment; always clean up.
• Split very large jobs or use services designed for them.

Further reading: Configure ephemeral storage for Lambda functions · boto3: S3 get_object