Reranking

Reranking for Better Retrieval

What is Reranking?
Reranking is a two-stage retrieval process: first retrieve many candidates quickly (using vector search), then rerank them for relevance using a more accurate model.

Why Rerank?
| Approach | Speed | Accuracy | Use |
|----------|-------|----------|-----|
| Bi-encoder (embedding) | Fast | Good | First retrieval |
| Cross-encoder (reranker) | Slow | Better | Rerank top-k |

Two-Stage Pipeline

Query
  |
  v
[Bi-encoder retrieval] (top 100)
  |
  v
[Cross-encoder reranking]
  |
  v
[Top 10 most relevant results]

Cross-Encoder vs Bi-Encoder

Bi-Encoder (Fast)
Encode query and documents separately:

query_embedding = embed(query)
doc_embeddings = [embed(doc) for doc in docs]
scores = cosine_similarity(query_embedding, doc_embeddings)

Cross-Encoder (Accurate)
Encode query and document together:

# Sees full context, can understand relationships
score = cross_encoder.predict([query, document])

Popular Rerankers
| Model | Type | Highlights |
|-------|------|------------|
| Cohere Rerank | API | Commercial, excellent quality |
| bge-reranker | Open | Various sizes, multilingual |
| cross-encoder/ms-marco | Open | Strong baseline |
| mixedbread-ai/mxbai-rerank | Open | State-of-the-art open |

Implementation

from sentence_transformers import CrossEncoder

# Load reranker
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

# First stage: vector retrieval
candidates = vector_store.query(query, top_k=100)

# Second stage: reranking
pairs = [[query, doc.text] for doc in candidates]
scores = reranker.predict(pairs)

# Sort by reranker scores
reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
top_results = reranked[:10]

When to Use Reranking
| Scenario | Recommendation |
|----------|----------------|
| High precision needed | Always rerank |
| Latency critical | Skip or use fast reranker |
| Large candidate pool | Essential |
| Domain-specific | Fine-tune reranker |

Performance Tips
- Retrieve more candidates than final need (100 or 50 for top 10)
- Consider reranker latency in architecture
- Batch reranking calls where possible
- Cache reranking for repeated queries