Retrieval Augmented Generation
Retrieval-Augmented Generation (RAG) is the architecture pattern that enhances LLM responses by first retrieving relevant documents from an external knowledge base and injecting them into the prompt context before generation — grounding the model's output in factual, up-to-date information rather than relying solely on parametric knowledge learned during pretraining, dramatically reducing hallucination and enabling domain-specific expertise.
RAG Pipeline
1. User Query: "What is the maximum operating temperature for the XYZ-3000 chip?"
2. Retrieval: Query embedded → similarity search against vector database → top-K relevant documents retrieved.
3. Augmentation: Retrieved documents prepended to the prompt as context.
4. Generation: LLM generates answer grounded in the retrieved context.
Components
| Component | Purpose | Examples |
|---|---|---|
| Embedding Model | Convert text to vectors | OpenAI ada-002, BGE, E5, GTE |
| Vector Database | Store and search embeddings | Pinecone, Weaviate, Qdrant, Chroma, pgvector |
| Chunking Strategy | Split documents into retrievable units | 256-1024 tokens, overlap 10-20% |
| Reranker | Re-score retrieved results for relevance | Cohere Rerank, BGE-reranker, cross-encoder |
| LLM | Generate final answer from context | GPT-4, Claude, LLaMA, Mistral |
Chunking Strategies
| Strategy | Chunk Size | Overlap | Best For |
|---|---|---|---|
| Fixed-size | 512 tokens | 50 tokens | General text |
| Sentence-based | 3-5 sentences | 1 sentence | Precise retrieval |
| Semantic | Variable | None | Coherent topics |
| Recursive character | 1000 chars | 200 chars | LangChain default |
| Parent-child | Small retrieve, large return | N/A | Best of both worlds |
Advanced RAG Techniques
- Hybrid Search: Combine vector similarity with keyword (BM25) search → better recall.
- HyDE: Generate hypothetical answer first → use it as retrieval query → better embedding match.
- Multi-Query: LLM generates multiple query variations → retrieve for each → union results.
- Reranking: Initial retrieval (fast, approximate) → reranker scores (slow, accurate) → top results.
- Agentic RAG: LLM decides when and what to retrieve iteratively — not just single retrieval.
Evaluation Metrics
| Metric | What It Measures |
|---|---|
| Faithfulness | Is the answer supported by retrieved context? |
| Answer Relevance | Does the answer address the question? |
| Context Relevance | Are retrieved documents relevant to the query? |
| Context Recall | Did retrieval find all necessary information? |
RAG vs. Fine-Tuning
- RAG: Dynamic knowledge, no retraining needed, traceable sources, handles knowledge updates.
- Fine-tuning: Bakes knowledge into weights, better for style/format changes, no retrieval latency.
- Best practice: Use both — fine-tune for domain style, RAG for factual knowledge.
RAG is the standard architecture for production LLM applications — by separating knowledge storage (database) from reasoning (LLM), it solves the core limitations of LLMs: outdated knowledge, hallucination, and lack of domain expertise, making it essential for enterprise AI deployments.