Sign in Get started

Integrating LLM APIs · Capstone

Capstone: AI Support Assistant

Design a grounded, tool-using assistant for an e-commerce help center.

Scenario

An e-commerce company wants an assistant in its web and mobile apps that answers policy questions from 2,000 help articles, looks up the customer's orders, and can start returns. 50k conversations/day, peaks at 5× during sales. Legal requires no unapproved refunds, privacy requires EU data handling, and finance wants cost per resolved conversation below a target.

Requirements

  • Grounded answers with citations
  • Order lookup and returns via tools
  • Streaming UX on web and mobile
  • Measurable quality and cost

Constraints

  • Provider rate limits shared with other products
  • Help articles change daily
  • Agents take over when the AI can't help

Deliverables

  1. Design the architecture and request flow.
  2. Safety, privacy and reliability controls.
  3. Evaluation and cost plan.

Hints

  • Write the tool list first and decide which actions need approval.
  • Cache everything stable at the start of the prompt.
  • Define 'resolved' before optimizing cost.

Draft each deliverable below before revealing the model answer and rubric.

Further reading: Tool use overview

Deliverables

Design the architecture and request flow.

Model answer

Clients → backend chat API (auth, per-user limits) → orchestrator: versioned system prompt (cached prefix: policies + tool definitions), hybrid retrieval over help articles (reindexed on publish, chunked, reranked), tools get_orders, get_order, start_return executing with the customer's identity. SSE streaming to clients; cancel upstream on disconnect. Conversation store with full content blocks, prompt/model versions and usage. Handoff tool creates a ticket with a summary for human agents.

Rubric

Strong answers include retrieval freshness, user-scoped tools, streaming relay with cancellation and a human-handoff path. Misses: tools running as a service superuser; no citations.

Safety, privacy and reliability controls.

Model answer

start_return requires explicit user confirmation in the UI; refunds are never a tool. Prompt-injection containment: tools are read-only except returns, no secrets in context, sanitize rendered Markdown and restrict links to the help domain. Privacy: provider contract with EU processing/appropriate retention, redact payment data, 90-day transcript retention, deletion across DB + vector store. Reliability: SDK retries with retry-after, per-tenant token budgets, queue during peaks, circuit breaker → 'connect to an agent', kill-switch flag, evaluated fallback model.

Rubric

Look for approvals on side effects, output sanitization, data-retention decisions and graceful degradation.

Evaluation and cost plan.

Model answer

Eval set of 500 anonymized conversations with expected outcomes (answer correct + cited, correct tool calls, correct handoff); code graders for tool calls/citations, calibrated LLM judge for helpfulness/faithfulness; CI gate on prompt/model changes; production sampling and thumbs feedback. Cost: prompt caching for the stable prefix, retrieval top-k tuned, concise style, lower effort for simple intents via a classifier route, batch nightly summarization of transcripts; dashboards for cost per resolved conversation and per feature.

Rubric

Strong answers define quality metrics before cost optimizations and measure cost per resolution, not per request.

Open in the app to mark lessons complete, take notes and track your progress.