Search

Information Retrieval and Search is the field of systems that find and rank relevant information from large collections in response to user queries — forming the backbone of search engines, enterprise knowledge bases, and retrieval-augmented generation (RAG) pipelines that ground AI systems in factual, up-to-date information.

What Is Search and Retrieval?

• Definition: Systems that, given a user query, efficiently find and rank the most relevant documents, passages, or records from a corpus potentially containing millions or billions of items.
• Query Types: Keyword queries ("TSMC 3nm process"), natural language questions ("What is the yield of N3 process?"), semantic queries (meaning-based), or structured queries (SQL, filters).
• Evaluation Metrics: Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG), Recall@K, Precision@K — measuring how well the relevant document ranks among retrieved results.
• Scale: Google indexes 100B+ web pages; enterprise search handles millions of internal documents; RAG systems retrieve from thousands to millions of passages.

Why Search and Retrieval Matters

• Knowledge Access: Enable users to find relevant information in seconds across vast document collections — from web search to scientific literature to enterprise wikis.
• RAG Foundation: Retrieval-augmented generation uses search to supply LLMs with relevant context — enabling AI systems to answer questions about current events and proprietary data without hallucination.
• E-Commerce: Product search and recommendation systems directly drive revenue — 1% improvement in search relevance can yield millions in revenue for large platforms.
• Legal & Compliance: Retrieve relevant case law, contracts, and regulatory documents for legal research and compliance verification.
• Customer Support: Find relevant help articles, past tickets, and product documentation to resolve customer issues quickly.

Keyword Search — The Classical Foundation

TF-IDF (Term Frequency–Inverse Document Frequency):
- Score = (how often term appears in document) × (how rare the term is across all documents).
- Rare terms in a matching document signal high relevance; common words ("the", "is") get near-zero weight.
- Fast, interpretable, no training required — but literal matching only; "car" and "automobile" are unrelated.

BM25 (Best Match 25):
- Probabilistic improvement over TF-IDF with term frequency saturation (diminishing returns for repeated terms) and document length normalization.
- Industry standard for keyword search — used in Elasticsearch, Lucene, and all major search engines as baseline.
- Parameters: k1 (term frequency saturation, typically 1.2–2.0), b (length normalization, typically 0.75).

Inverted Index:
- Data structure mapping each term to the list of documents containing it — enables O(log n) term lookup across billion-document corpora.
- Foundation of all keyword search systems.

Semantic Search — Neural Retrieval

Bi-Encoder (Dense Retrieval):
- Encode query and documents separately into dense vectors using BERT-based encoders.
- Retrieve by approximate nearest-neighbor search (FAISS, HNSW, ScaNN) in vector space.
- Captures semantic similarity — "vehicle" and "car" are near neighbors in embedding space.
- Training: contrastive learning on (query, relevant document, negative document) triplets.
- Models: DPR (Dense Passage Retrieval), E5, BGE, Cohere Embed, OpenAI text-embedding-3.

Cross-Encoder (Reranking):
- Jointly encode query + document through a single model — captures fine-grained interactions.
- Much more accurate than bi-encoders; 10–100x slower — used only for reranking top-K candidates.

ColBERT (Late Interaction):
- Compute token-level embeddings for query and document independently, then score via MaxSim (maximum similarity per query token).
- Balance between bi-encoder speed and cross-encoder accuracy.

RAG Search Pipeline

Step 1 — Indexing: Chunk documents into passages (128–512 tokens), embed with bi-encoder, store in vector database (Pinecone, Weaviate, pgvector, Chroma).

Step 2 — Retrieval: Given query, embed with same encoder, retrieve top-K passages via ANN search (typically K=20–100).

Step 3 — Reranking: Cross-encoder reranks top-K to top-5 — improving precision at the cost of latency.

Step 4 — Generation: LLM generates response conditioned on retrieved context + original query.

Retrieval System Comparison

Search and retrieval is the information access layer that determines whether AI systems answer from knowledge or hallucinate — as hybrid retrieval systems combining keyword precision with semantic understanding become standard, high-quality grounded AI applications will scale to every enterprise knowledge domain.