Sentence Transformer
Sentence Transformers are neural network models that produce fixed-length embeddings for sentences and paragraphs — enabling semantic similarity search, clustering, and retrieval by mapping semantically related texts to nearby points in embedding space.
The Core Problem
- BERT produces contextualized token embeddings — not a single sentence representation.
- Naive [CLS] token: Poor for semantic similarity (requires fine-tuning).
- Naive mean pooling: Better but still suboptimal.
- SBERT: Fine-tune with siamese/triplet networks → excellent sentence embeddings.
Sentence-BERT (SBERT) Architecture
- Siamese BERT: Two identical BERT models processing sentence pairs.
- Mean-pooled output → fixed-size sentence vector.
- Trained with: Natural Language Inference (NLI) data + triplet/cosine objectives.
- Cosine similarity of SBERT embeddings correlates strongly with human semantic judgment.
Training Objectives
- Cosine Similarity Loss: Minimize angle between positive pairs; maximize for negative pairs.
- Multiple Negative Ranking (MNR): In-batch negatives — scale efficiently.
- Triplet Loss: $|sim(a,p) - sim(a,n)| > \epsilon$ — anchor closer to positive than negative.
Bi-Encoder vs. Cross-Encoder
| Feature | Bi-Encoder | Cross-Encoder |
|---|---|---|
| Architecture | Two separate encoders | Joint encoding of pair |
| Inference | Pre-compute embeddings | Must process pair together |
| Speed | Fast (vector search) | Slow (no precomputation) |
| Accuracy | Good | Better |
| Use case | First-stage retrieval | Reranking |
RAG Retrieval Stack
- Bi-encoder: Retrieve top-100 from vector DB (milliseconds).
- Cross-encoder: Rerank top-100 → top-5 (100ms).
- Combine both for optimal quality/speed tradeoff.
Key Models
- all-MiniLM-L6-v2: 22M params, 384-dim, very fast — popular for production.
- BGE-large (Beijing Academy): Best MTEB score in open-source (mid-2024).
- E5-mistral-7b: LLM-based embeddings — top accuracy but expensive.
- OpenAI text-embedding-3-large: 3072-dim, top accuracy for SaaS.
Sentence transformers are the foundation of modern semantic search and RAG systems — their ability to compress arbitrary text into searchable vectors at millisecond speed is what makes LLM-powered knowledge bases and retrieval systems practical at scale.