Packaging Dependencies and Native Wheels

Scenario

Example app: an image and PDF processing function depends on Pillow, cryptography and pydantic. A developer builds the deployment ZIP on an Apple-silicon laptop with pip install -r requirements.txt -t package/ and uploads it to an x86_64 function.

What you see in production

The first invocation fails with Runtime.ImportModuleError: ... invalid ELF header (or cannot open shared object file). The same code passes every local test.

The naive code

pip install -r requirements.txt -t package/      # wheels built for the laptop's OS and CPU
cd package && zip -r ../function.zip . && cd ..
zip function.zip handler.py

Reproduce it

Wheels for packages with C extensions are tagged with the platform they run on. This check mimics pip's compatibility decision for the Lambda environment:

LAMBDA = {"python": "cp312", "platforms": {"manylinux2014_x86_64", "manylinux_2_17_x86_64", "manylinux_2_28_x86_64"}}

def compatible(wheel_name):
    name, version, python_tag, abi_tag, platform_tag = wheel_name[:-4].split("-")
    if platform_tag == "any":
        return True                                   # pure Python
    return python_tag == LAMBDA["python"] and platform_tag in LAMBDA["platforms"]

built_on_laptop = [
    "pydantic_core-2.27.2-cp312-cp312-macosx_11_0_arm64.whl",
    "pillow-11.1.0-cp312-cp312-macosx_11_0_arm64.whl",
    "requests-2.32.3-py3-none-any.whl",
]
built_for_lambda = [
    "pydantic_core-2.27.2-cp312-cp312-manylinux_2_17_x86_64.whl",
    "pillow-11.1.0-cp312-cp312-manylinux_2_28_x86_64.whl",
    "requests-2.32.3-py3-none-any.whl",
]
for label, wheels in [("laptop build", built_on_laptop), ("lambda build", built_for_lambda)]:
    print(label, [("OK " if compatible(w) else "BAD ") + w.split("-")[0] for w in wheels])

laptop build ['BAD pydantic_core', 'BAD pillow', 'OK requests']
lambda build ['OK pydantic_core', 'OK pillow', 'OK requests']

Root cause

Python packages with compiled extensions ship platform-specific wheels (macOS arm64, Windows, Linux x86_64 and aarch64). pip install picks wheels for the machine it runs on, so a package built on macOS contains binaries Lambda's Amazon Linux cannot load. The function's architecture (x86_64 or arm64) and Python version must also match.

The fix

Build dependencies for the target platform:

pip install -r requirements.txt --target package/ \
  --platform manylinux2014_x86_64 --implementation cp --python-version 3.12 --only-binary=:all:

Or build inside a Lambda-compatible container (sam build --use-container, or Docker with the AWS base image), which also handles packages that must be compiled from source. Tools such as AWS SAM, CDK's PythonFunction and the Serverless Framework's python-requirements plugin automate this. Keep the package small: exclude tests and caches, and put very large dependencies in a container image (up to 10 GB) or a versioned layer shared deliberately.

Trade-offs

• --only-binary fails if a dependency has no compatible wheel; building in a container is slower but always works.
• Switching to arm64 (Graviton) is cheaper per GB-second but requires aarch64 wheels for every native dependency.
• Container images remove size limits and give reproducible builds but need a registry and image scanning.

Detect it in production

• Runtime.ImportModuleError immediately after deploys: add a smoke-test invocation to the pipeline.
• Build in CI on Linux (or in a container), never from laptops.
• Track deployment package size and Init Duration per function.

Key points

• Native wheels are platform-specific; build for Lambda's OS, CPU architecture and Python version.
• Use pip --platform ... --only-binary=:all: or build in a Lambda-compatible container.
• Large dependencies: container images or deliberate layers.

Further reading: Working with .zip file archives for Python Lambda functions · pip install: --platform