Integrating LLM APIs · Capstone
Capstone: Document Extraction Pipeline
Extract structured data from 200k invoices per day reliably and cheaply.
Scenario
An accounts-payable product receives 200k supplier invoices per day as PDFs and emails. It must extract supplier, invoice number, dates, line items, totals and tax into its database for approval. Accuracy matters (finance data); most invoices can be processed within a few hours, but 5% are marked urgent.
Requirements
- Schema-valid extraction with confidence/flags
- Urgent invoices within 2 minutes
- Cost-efficient bulk processing
- Auditability
Constraints
- Shared provider rate limits
- Some PDFs are scans
Deliverables
- Design the pipeline (sync vs batch, schema, validation).
- Quality, audit and cost controls.
Draft each deliverable below before revealing the model answer and rubric.
Further reading: Structured outputs
Deliverables
Design the pipeline (sync vs batch, schema, validation).
Model answer
Ingest → store original in object storage → queue. Urgent: synchronous call from a worker pool with concurrency tuned to rate limits. Others: grouped into provider batch jobs (discounted) every 15 minutes, results mapped by custom_id. Structured outputs with a Pydantic/Zod schema (nullable fields for missing data). Post-validation: line items sum to subtotal, tax math, date sanity, supplier matched to vendor master; failures → one repair attempt with the validation errors, then human review queue.
Rubric
Look for the split between urgent sync and discounted batch, schema-constrained output and business-rule validation feeding a human review queue.
Quality, audit and cost controls.
Model answer
Gold set of 1,000 labeled invoices per supplier type; field-level accuracy dashboards; CI gate on prompt/model changes. Audit: store prompt version, model, raw output, validation results and reviewer edits per invoice. Cost: cached stable instructions, batch discount, compact page images/text, route simple layouts to a lower effort setting after evals confirm parity; per-customer cost attribution.
Rubric
Strong answers measure field-level accuracy and keep a full audit trail. Misses: no human review path; trusting totals without arithmetic checks.
Open in the app to mark lessons complete, take notes and track your progress.