Sign in Get started

Python Serverless on AWS Lambda · Capstone Case Studies

Capstone: A Nightly Import of Five Million Rows

Design a reliable nightly import of a large partner CSV into DynamoDB with Python on AWS.

Scenario

Example company: a retailer receives a 5-million-row product CSV (about 3 GB) from a supplier every night at 01:00 in S3. Prices and stock must be updated in DynamoDB by 06:00. The current Python function downloads the file to /tmp, loads it with pandas and times out on most nights.

Requirements

  • All valid rows are applied by 06:00; invalid rows are reported to the supplier with reasons.
  • Re-running the import (after a failure or a corrected file) must not create duplicates or apply stale data over newer data.
  • The team can see progress and which rows failed.

Constraints

  • Python on Lambda, S3, SQS, Step Functions and DynamoDB are available; no long-running servers.
  • The supplier sometimes sends the same file twice, and sometimes sends a corrected file at 03:00.
  • DynamoDB writes must not throttle the live website reading the same table.

Deliverables

  1. The processing architecture: how the file is split and processed within Lambda's limits.
  2. Validation and error reporting for bad rows.
  3. Idempotency and ordering: duplicate files, re-runs and corrected files.
  4. Protecting the live table: write rate, capacity and backpressure.
  5. Observability: what you log and alarm on, and how you prove the import finished.

Hints

  • Step Functions can iterate over the rows of a CSV in S3 directly.
  • A file version or timestamp on each item lets you refuse stale writes.
  • Concurrency limits apply to map runs and to queues.

Draft each deliverable below before revealing the model answer and rubric.

Further reading: Step Functions: Distributed Map · DynamoDB condition expressions

Deliverables

Describe the processing architecture.

Model answer

  • S3 ObjectCreated (filtered to the supplier prefix) starts a Step Functions execution named after the object key and version (a duplicate start for the same file is rejected).
  • A Distributed Map reads the CSV directly from S3 and batches rows (for example 500 rows per child) into Python Lambda invocations with a concurrency limit (for example 40).
  • Each child validates rows with Pydantic, writes valid rows with batch_writer, and returns counts plus invalid rows.
  • Results are written to S3 by the map run; a final step aggregates counts and produces a report for the supplier.

Rubric

Look for splitting into small units processed in parallel (Distributed Map or SQS fan-out), not one long function; streaming rather than loading 3 GB into memory; and a final aggregation step.

Explain validation and error reporting.

Model answer

  • A Pydantic model per row (types, ranges, required fields); invalid rows are collected with row number and reasons, not raised.
  • The map run's failure tolerance is set so a few bad rows do not fail the whole import, while systemic failure (most rows invalid) stops it and alerts.
  • The report (CSV of rejected rows with reasons) is stored in S3 and emailed or exposed to the supplier.

Rubric

Good answers separate row-level validation errors (expected, reported) from infrastructure failures (retried, alarmed) and set a tolerance threshold.

Handle idempotency, duplicates and corrected files.

Model answer

  • Execution name = file key + version ID, so the same file cannot start two executions.
  • Each item carries source_file_ts; writes use a condition attribute_not_exists(source_file_ts) OR source_file_ts <= :ts, so a re-run is harmless and a late, older file cannot overwrite newer data.
  • A corrected file at 03:00 has a newer timestamp, so it wins row by row; rows missing from the corrected file are handled explicitly (for example marked inactive in a final step) if the contract requires it.

Rubric

Look for conditional writes based on a source version or timestamp, and an explicit decision about rows absent from a newer file.

Protect the live table.

Model answer

  • Limit total write throughput: map concurrency x batch size x writes per second per child must stay within the capacity reserved for the import (or on-demand limits), leaving headroom for the website.
  • Handle throttling with SDK retries and backoff; if throttling persists, the map's concurrency is the lever to reduce.
  • Run the import in the low-traffic window and monitor read latency of the site during it.

Rubric

Strong answers compute a target write rate and connect it to a concrete concurrency setting, rather than relying on retries alone.

Observability and proof of completion.

Model answer

  • Structured logs per child batch (file, batch range, valid, invalid, duration) with Powertools Logger; metrics for rows applied, rows rejected, throttles.
  • The execution's final state plus a reconciliation count (rows in file = applied + rejected) proves completion.
  • Alarms: execution failed or not finished by 05:30, rejection rate above a threshold, DynamoDB throttling on the live table.

Rubric

Look for an explicit completeness check (counts reconcile) and a deadline alarm, not just error alarms.

Open in the app to mark lessons complete, take notes and track your progress.