Embedding Model Retrieval

Text Embedding Models for Retrieval are neural networks that map text passages of arbitrary length to fixed-dimensional dense vectors where semantic similarity is captured by vector proximity (cosine similarity or dot product) — enabling sub-second semantic search over millions of documents by replacing keyword matching with meaning-based matching, powering RAG systems, recommendation engines, and semantic search applications.

Why Dense Retrieval Outperforms Keyword Search

Traditional search (BM25, TF-IDF) matches exact terms — a query for "how to fix a flat tire" won't match a document about "repairing a punctured wheel." Dense retrieval encodes both query and document into vectors where semantically equivalent texts have high cosine similarity regardless of word choice, capturing synonymy, paraphrase, and conceptual similarity.

Architecture

• Bi-Encoder: Separate encoders for query and document (or shared encoder). Each text is independently encoded to a vector. Similarity = dot_product(q_vec, d_vec). Documents can be pre-encoded and indexed. At query time, only the query needs encoding. Standard for production systems.
• Cross-Encoder: Both query and document are concatenated and processed jointly through a single model. More accurate (full cross-attention between query and document tokens) but requires processing every query-document pair at search time — too slow for first-stage retrieval but excellent as a reranker.

Training

• Contrastive Learning: The embedding model is trained to maximize similarity between (query, positive_document) pairs and minimize similarity with negative documents. The InfoNCE loss pulls positive pairs together and pushes hard negatives apart.
• Hard Negative Mining: Random negatives are too easy. Effective training requires hard negatives — documents that are superficially similar to the query but not actually relevant. Mined from BM25 results or from the embedding model's own retrieval.
• Knowledge Distillation: Cross-encoder scores are distilled into bi-encoder training, using the cross-encoder's superior relevance judgments as soft labels.

Indexing and Search

• HNSW (Hierarchal Navigable Small World): The dominant approximate nearest neighbor (ANN) index. Builds a hierarchical proximity graph enabling ~90% recall at <1ms latency for 1M+ vectors. Libraries: FAISS, Milvus, Qdrant, Pinecone.
• IVF (Inverted File Index): Clusters vectors into Voronoi cells. At query time, searches only the nearest clusters. Trading recall for speed.
• Quantization (PQ, SQ): Compress vectors from 768×float32 (3KB) to 96 bytes via Product Quantization, enabling billion-scale indexes in memory.

Key Models

• E5 / BGE / GTE: Open-source embedding models trained on massive retrieval datasets. 768-1024 dimensional vectors. State-of-the-art on MTEB benchmarks.
• OpenAI text-embedding-3-large: Commercial embedding model with adjustable dimensionality (256-3072).

Text Embedding Models are the neural compression that maps the infinite space of human language into geometric points where meaning defines distance — enabling machines to find relevant information not by matching words but by understanding intent.