Cost Control: Prompt Caching, Routing & Budgets

Cost = input tokens × input price + output tokens × output price (+ cache writes/reads, tools). Levers, roughly in order: prompt caching (stable prefix — system prompt, tools, documents — is cached and later reads cost a fraction), trimming context (don't send what isn't needed), limiting output (concise instructions, sensible max_tokens), batch processing for non-urgent work, effort/model routing per task, and response caching for repeated identical questions. Track cost per feature and per tenant and alert on anomalies; set per-tenant budgets.

Cost levers

response = client.messages.create(
    model=MODEL, max_tokens=4096,
    system=[{"type": "text", "text": LONG_POLICY_HANDBOOK,       # stable, large prefix
             "cache_control": {"type": "ephemeral"}}],
    messages=[{"role": "user", "content": question}],          # volatile content after the breakpoint
)
u = response.usage
log.info("llm_usage", cache_read=u.cache_read_input_tokens, cache_write=u.cache_creation_input_tokens,
         uncached=u.input_tokens, output=u.output_tokens)

const response = await client.messages.create({
  model: MODEL, max_tokens: 4096,
  system: [{ type: "text", text: LONG_POLICY_HANDBOOK, cache_control: { type: "ephemeral" } }],
  messages: [{ role: "user", content: question }],
});
metrics.record("llm.cache_read_tokens", response.usage.cache_read_input_tokens ?? 0);

Common pitfalls

• Timestamps or request IDs in the system prompt (break the cache every call).
• No per-tenant cost attribution.
• Downgrading models without evals.

Key points

• Cache stable prefixes; put volatile content last
• Measure cost per feature/tenant
• Batch what isn't interactive

Further reading: Prompt caching