Dense Retrieval
Dense Retrieval and Embedding Models are the neural information retrieval systems that encode queries and documents into dense vector representations in a shared semantic space — enabling semantic search where relevance is measured by vector similarity rather than keyword overlap, finding conceptually related documents even with no shared vocabulary, powering applications from question answering systems to RAG pipelines and enterprise search.
Sparse vs Dense Retrieval
| Aspect | Sparse (BM25/TF-IDF) | Dense (Bi-Encoder) |
|---|---|---|
| Representation | Bag of words | Dense vector |
| Similarity | Term overlap | Dot product / cosine |
| Vocabulary mismatch | Fails (lexical gap) | Handles (semantic) |
| Speed | Very fast (inverted index) | Fast (ANN index) |
| Interpretability | High | Low |
| Out-of-domain | Robust | May degrade |
DPR (Dense Passage Retrieval)
- Karpukhin et al. (2020): Dual-encoder architecture for open-domain QA.
- Question encoder: BERT → 768-d vector for query.
- Passage encoder: Separate BERT → 768-d vector for document passage.
- Training: Contrastive loss — maximize similarity of (question, positive passage) pairs, minimize similarity to negatives.
- Retrieval: FAISS index over 21M Wikipedia passages → retrieve top-k by dot product.
- Key result: DPR significantly outperforms BM25 for natural language questions.
In-Batch Negatives Training
def contrastive_loss(q_embeds, p_embeds, temperature=0.07):
# q_embeds: [B, D] query embeddings
# p_embeds: [B, D] positive passage embeddings
# Other passages in batch serve as hard negatives
scores = torch.matmul(q_embeds, p_embeds.T) / temperature # [B, B]
labels = torch.arange(B) # diagonal is positive pair
return F.cross_entropy(scores, labels)Sentence Transformers (SBERT)
- Siamese BERT: Encode two sentences → mean-pool → compare with cosine similarity.
- Fine-tuned on NLI (entailment pairs as positives, contradiction as negatives).
- Enables efficient semantic textual similarity (STS) → used for clustering, semantic search.
- SBERT is 9,000× faster than cross-encoder for ranking 10,000 sentences.
Modern Embedding Models
| Model | Size | Notes |
|---|---|---|
| E5-large | 335M | Strong general embedding |
| BGE-M3 | 570M | Multilingual, multi-granularity |
| GTE-Qwen2 | 7B | LLM-based, very strong |
| text-embedding-3 (OpenAI) | Proprietary | 1536-d, MTEB SOTA |
| Voyage-3 (Anthropic) | Proprietary | Strong code + retrieval |
MTEB (Massive Text Embedding Benchmark)
- 56 tasks across 7 categories: Retrieval, classification, clustering, STS, reranking, etc.
- 112 languages → comprehensive multilingual evaluation.
- Standard leaderboard for comparing embedding models.
ANN (Approximate Nearest Neighbor) Search
- Exact k-NN over millions of vectors is too slow → approximate search.
- FAISS: Facebook AI similarity search → IVF (inverted file) + PQ (product quantization) → 100M vectors in < 10ms.
- HNSW: Hierarchical navigable small world graph → fast and accurate for moderate scales.
- ScaNN (Google): Optimized for TPU; state-of-the-art recall-latency trade-off.
Retrieval in RAG Pipelines
- Chunk documents → embed each chunk → store in vector database (Pinecone, Weaviate, Chroma).
- At query time: Embed query → retrieve top-k chunks by similarity → inject into LLM context.
- Hybrid retrieval: Combine dense score + BM25 score → better than either alone.
- Reranking: Cross-encoder rescores top-k retrieved passages → better precision at top positions.
Dense retrieval and embedding models are the semantic backbone of modern AI-powered search and knowledge retrieval — by learning that "cardiac arrest" and "heart attack" are semantically equivalent without sharing a single word, dense retrievers close the vocabulary gap that made keyword search frustrating for decades, enabling the retrieval-augmented generation pipelines that allow LLMs to access specialized knowledge bases, corporate documents, and up-to-date information far beyond what can fit in a context window.