Retrieval Augmented Generation RAG

Retrieval-Augmented Generation (RAG) is the AI architecture that enhances large language model responses by retrieving relevant information from external knowledge sources at inference time — grounding LLM outputs in factual, up-to-date, and domain-specific documents rather than relying solely on parametric knowledge baked in during training, dramatically reducing hallucinations and enabling enterprise deployment without costly fine-tuning.

Why RAG Exists

LLMs have a knowledge cutoff date (training data stops at a point in time) and cannot access proprietary or real-time information. Fine-tuning is expensive, slow, and creates a new static snapshot. RAG solves both problems by retrieving relevant context dynamically at query time.

RAG Pipeline Architecture

Indexing Phase (Offline):
- Document Ingestion: Load documents from various sources (PDFs, databases, APIs, wikis).
- Chunking: Split documents into semantically coherent chunks (256-1024 tokens). Strategies: fixed-size with overlap, recursive character splitting, semantic chunking (split at topic boundaries using embeddings).
- Embedding: Encode each chunk into a dense vector using an embedding model (OpenAI text-embedding-3, BGE, GTE, E5).
- Vector Store: Index embeddings in a vector database (Pinecone, Weaviate, Qdrant, FAISS, Chroma) with metadata (source, date, section).

Query Phase (Online):
- Query Embedding: Encode the user query into the same embedding space.
- Retrieval: Approximate nearest neighbor search returns top-K relevant chunks (typically K=3-10).
- Context Assembly: Retrieved chunks are formatted into a prompt with the user query.
- Generation: LLM generates a response grounded in the retrieved context.

Advanced RAG Techniques

• Hybrid Search: Combine dense vector search with sparse BM25 keyword search using Reciprocal Rank Fusion. Captures both semantic similarity and exact keyword matches.
• Query Transformation: Rewrite the user query for better retrieval — HyDE (Hypothetical Document Embeddings) generates a hypothetical answer and uses it as the search query. Multi-query generates multiple reformulations and merges results.
• Re-Ranking: After initial retrieval, a cross-encoder re-ranks the top-K chunks by relevance. Cohere Rerank, BGE-reranker, and ColBERT provide significant precision improvement.
• Agentic RAG: The LLM decides when and what to retrieve through tool-calling — routing queries to different knowledge bases, performing multi-step retrieval, and synthesizing across sources.

Chunking Strategy Impact

Chunk size directly affects retrieval quality: too small (128 tokens) loses context continuity; too large (2048 tokens) dilutes relevance with irrelevant surrounding text. Optimal chunk size depends on document structure and query types — technical documentation benefits from larger chunks preserving procedure steps; FAQ-style content benefits from smaller, self-contained chunks.

Evaluation Metrics

• Retrieval Quality: Precision@K, Recall@K, NDCG — does the retriever find the right chunks?
• Generation Quality: Faithfulness (is the answer supported by retrieved context?), relevance (does the answer address the query?), completeness.
• RAGAS Framework: Automated evaluation using LLM-as-judge for faithfulness, answer relevance, and context relevance.

RAG is the pragmatic bridge between LLM capabilities and real-world knowledge requirements — enabling organizations to deploy AI assistants that answer questions accurately from their own documents, without the cost and data requirements of fine-tuning, while maintaining the conversational fluency of foundation models.