Embedding

Embeddings are dense vector representations that capture semantic meaning of text, images, or other data — transforming words, sentences, or documents into fixed-dimensional numerical vectors where similar concepts are closer in the vector space, enabling semantic search, clustering, classification, and retrieval-augmented generation (RAG).

What Are Embeddings?

• Definition: Learned numerical representations of data in vector space.
• Property: Similar items have similar vectors (close by distance).
• Dimensions: Typically 384-3072 floating point values.
• Use: Foundation for semantic search, RAG, similarity comparison.

Why Embeddings Matter

• Semantic Understanding: Find similar meaning, not just matching keywords.
• Cross-Modal: Compare text to text, images, or even audio.
• Compression: Dense representation of meaning.
• Foundation: Enable vector databases, RAG, recommendations.
• Transfer Learning: Pre-computed representations reusable across tasks.

Embedding Levels

Word Embeddings (Legacy):
- Word2Vec, GloVe: One vector per word.
- Same vector regardless of context.
- "Bank" has same embedding in "river bank" and "savings bank."

Contextual Embeddings (Modern):
- BERT, transformer-based: Different vectors based on context.
- "Bank" differs in "river bank" vs. "savings bank."
- Captures nuance and polysemy.

Sentence/Document Embeddings:
- Entire text chunk → single vector.
- Sentence-BERT, E5, BGE models.
- Used for semantic search, RAG.

Popular Embedding Models

Model              | Dimensions | Use Case           | Provider
-------------------|------------|--------------------|-----------
text-embedding-3   | 256-3072   | General purpose    | OpenAI
E5-v2              | 1024       | Retrieval          | Microsoft
BGE-v2             | 1024       | Multilingual       | BAAI
Cohere Embed v3    | 1024       | Enterprise         | Cohere
all-MiniLM-L6      | 384        | Fast, lightweight  | SBERT
GTE                | 768-1024   | General purpose    | Alibaba

How Embeddings Work

Similarity Metrics

Cosine Similarity (most common):

cos(A, B) = (A · B) / (|A| × |B|)

Range: -1 to 1 (typically 0 to 1 for text)
Higher = more similar

Euclidean Distance (L2):

L2(A, B) = sqrt(Σ(ai - bi)²)

Lower = more similar
Works best with normalized vectors

Dot Product:

dot(A, B) = Σ(ai × bi)

Higher = more similar
Equivalent to cosine for normalized vectors

Embedding Applications

Semantic Search:

Query: "machine learning tutorials for beginners"
         ↓ embed
Query Vector: [...]
         ↓ similarity search
Similar docs: [doc_47, doc_123, doc_89, ...]

RAG (Retrieval-Augmented Generation):

1. User question → embed
2. Find similar knowledge chunks
3. Inject into LLM context
4. Generate grounded response

Clustering/Classification:

1. Embed all documents
2. Run clustering (K-means, HDBSCAN)
3. Discover topic groups automatically

Duplicate Detection:

1. Embed all items
2. Find pairs with similarity > threshold
3. Mark as likely duplicates

Embedding Best Practices

• Match Model to Use Case: Retrieval models for search, general for clustering.
• Consistent Processing: Same tokenization, truncation at query and index time.
• Batch Processing: GPU embedding is much faster in batches.
• Dimension Trade-off: Higher dims = more expressive, more memory/compute.
• Quantization: Store as int8 or binary for memory efficiency.

Embeddings are the bridge between human language and machine computation — by converting meaning into numbers, embeddings enable all the semantic AI applications that find "similar" rather than "exact" matches, making them foundational to modern AI systems.