Integrating LLM APIs · LLM API Foundations
Tokens, Context Windows & Long Inputs
Counting tokens, budgeting context and handling long documents and conversations.
Count tokens with the provider's tokenizer or token-counting endpoint — tokenizers differ between providers and model generations. Budget the context: system prompt + tools + history + retrieved context + room for output. For long conversations, trim or summarize old turns (or use provider-side compaction); for long documents, retrieve relevant chunks rather than sending everything, or use large-context models when the whole document matters. Never silently truncate inputs the task depends on.
Long-context strategies
| Strategy | Good for | Risk |
|---|---|---|
| Send everything (large context) | Whole-document reasoning | Cost and latency per request |
| Sliding window of recent turns | Chat | Forgets early facts |
| Summarize / compact old turns | Long sessions | Lossy summaries |
| Retrieval (RAG) | Large knowledge bases | Missed chunks → wrong answers |
| Map-reduce over chunks | Summarizing huge corpora | Loses cross-chunk links |
count = client.messages.count_tokens(model=MODEL, system=SYSTEM, messages=messages)
if count.input_tokens > INPUT_BUDGET:
messages = compact_history(messages) # summarize older turns, keep recent verbatim
const { input_tokens } = await client.messages.countTokens({ model: MODEL, system: SYSTEM, messages });
if (input_tokens > INPUT_BUDGET) messages = await compactHistory(messages);
Common pitfalls
- Estimating tokens with another provider's tokenizer.
- Truncating the middle of a document silently.
- Filling the context with irrelevant retrieved text (hurts quality and cost).
Key points
- Count with the provider's tokenizer
- Budget context explicitly
- Retrieve or compact instead of truncating
Further reading: Token counting
Practice questions
Why shouldn't you reuse one provider's tokenizer to estimate another's tokens?
Answer
Tokenizers differ, sometimes significantly, even across model generations of the same provider, so estimates can be wrong for cost and limits.
What are two ways to keep a long chat within the context window?
Answer
Keep a sliding window of recent turns and summarize/compact older turns (or use provider-side compaction), optionally storing facts in retrievable memory.
Open in the app to mark lessons complete, take notes and track your progress.