Retrieval Augmented Generation

Retrieval-Augmented Generation (RAG) is a framework that enhances LLM outputs by retrieving relevant documents from a knowledge base and including them in the prompt — combining parametric knowledge (model weights) with non-parametric knowledge (external documents).

RAG Architecture

1. Indexing: Chunk documents → embed each chunk → store in vector database.
2. Retrieval: Embed the user query → find top-k most similar chunks by vector similarity.
3. Augmentation: Inject retrieved chunks into the LLM prompt as context.
4. Generation: LLM generates an answer grounded in the retrieved context.

Why RAG?

• Reduces hallucination: LLM answers from retrieved facts rather than generating from memory.
• Up-to-date knowledge: Knowledge base can be updated without retraining the model.
• Attribution: Can cite sources — users can verify which documents were used.
• Cost: Cheaper than fine-tuning for knowledge-intensive tasks.

Key Components

• Chunking Strategy: Fixed size (512 tokens), sentence-based, or semantic chunking.
• Embedding Model: OpenAI text-embedding-3, E5, GTE, BGE for dense retrieval.
• Vector Database: Pinecone, Weaviate, Chroma, Qdrant, pgvector, FAISS.
• Reranking: Cross-encoder reranker (Cohere Rerank, BGE-reranker) improves retrieval quality.

Advanced RAG Techniques

• Hybrid Search: Combine dense (semantic) + sparse (BM25 keyword) retrieval.
• HyDE (Hypothetical Document Embeddings): Generate a hypothetical answer first, then retrieve.
• Self-RAG: Model decides when to retrieve and evaluates retrieved passages.
• Multi-hop RAG: Iterative retrieval for complex multi-step questions.

RAG vs. Fine-tuning: RAG is preferred for dynamic or large knowledge bases; fine-tuning is better for style, format, and capability changes.

RAG is the standard architecture for enterprise LLM applications — it bridges the gap between general-purpose LLMs and domain-specific knowledge requirements.