Retrieval-Augmented Generation & Grounding — Integrating LLM APIs

RAG retrieves relevant content (from search indexes, vector stores or APIs) and puts it in the prompt so answers are grounded in your current data. Quality depends mostly on retrieval: chunking, hybrid search (keyword + vectors), reranking, metadata filters and access control at query time (only retrieve what the user may see). Ask the model to cite sources and to say when the answer isn't in the context. Evaluate retrieval (recall@k) separately from generation.

Retrieval options

Criterion	Keyword (BM25)	Vector (embeddings)	Hybrid + rerank
Exact terms, IDs, codes	Excellent	Weak	Excellent
Paraphrases / semantics	Weak	Strong	Strong
Infra	Search engine	Vector DB / pgvector	Both + reranker
Explainability	High	Lower	Medium
Typical quality	Good	Good	Best

chunks = await search.hybrid(query=question, k=8, filters={"tenant_id": user.tenant_id,
                                                             "acl": user.groups})   # ACL at query time
context = "\n".join(f'<doc id="{c.id}" title="{c.title}">{c.text}</doc>' for c in chunks)
response = client.messages.create(
    model=MODEL, max_tokens=4096,
    system=("Answer using only the documents provided. Cite document ids like [doc-12]. "
            "If the documents don't contain the answer, say you don't know."),
    messages=[{"role": "user", "content": f"<documents>{context}</documents>\n\nQuestion: {question}"}],
)

SELECT id, title, text
FROM chunks
WHERE tenant_id = $1
ORDER BY embedding <=> $2      -- cosine distance to the query embedding
LIMIT 20;                      -- then rerank and keep the best 8

Common pitfalls

Retrieving documents the user isn't allowed to see.
Huge chunks that dilute relevance.
No 'I don't know' path (hallucinated answers).

Key points

Retrieval quality dominates RAG quality
Enforce permissions in retrieval
Cite sources; allow 'not found'

Retrieval options

Common pitfalls

Key points

Practice questions