Retro (retrieval-Enhanced Transformer)

RETRO (Retrieval-Enhanced Transformer) is the language model architecture that deeply integrates retrieval augmentation into the transformer by splitting input into chunks, retrieving relevant passages from a trillion-token database for each chunk, and conditioning generation on both the input and retrieved content through dedicated cross-attention layers — demonstrating that a 7B parameter model with retrieval can match the performance of 25× larger dense models on knowledge-intensive tasks by offloading factual knowledge to an external database.

What Is RETRO?

• Definition: A transformer architecture with integrated retrieval — the input is split into fixed-size chunks (typically 64 tokens), each chunk triggers a nearest-neighbor search against a pre-built retrieval database, and retrieved passages are incorporated into generation via specialized chunked cross-attention (CCA) layers interleaved with standard self-attention.
• Chunked Cross-Attention (CCA): A novel attention mechanism where tokens in a chunk attend to the retrieved neighbors for that chunk — retrieved information is injected at specific points in the model rather than simply prepended to the context.
• Retrieval Database: A pre-computed index of trillions of tokens (e.g., MassiveText corpus) encoded into dense embeddings by a frozen BERT encoder — enabling fast approximate nearest-neighbor retrieval at each chunk.
• Architecture Integration: Retrieval is not a preprocessing step — it is woven into the model's forward pass, with CCA layers at every few transformer blocks enabling deep interaction between retrieved and generated content.

Why RETRO Matters

• 25× Parameter Efficiency: RETRO-7B matches the perplexity of GPT-3 175B on knowledge-heavy tasks — demonstrating that retrieval substitutes for parametric memorization of facts.
• Updatable Knowledge: The retrieval database can be updated without retraining the model — new facts, corrected information, and temporal knowledge can be inserted by updating the index.
• Reduced Hallucination: By conditioning on retrieved factual content, RETRO generates text grounded in actual documents rather than relying solely on compressed parametric knowledge.
• Cost-Effective Scaling: Scaling the retrieval database (adding more documents) is far cheaper than scaling model parameters — database storage costs pennies per GB while training compute costs millions per parameter doubling.
• Attribution: Retrieved passages provide implicit citations for generated content — enabling source tracking that pure parametric models cannot provide.

RETRO Architecture

Retrieval Pipeline:
- Split input into 64-token chunks: [c₁, c₂, ..., cₘ].
- For each chunk cᵢ, encode using frozen BERT → query embedding.
- Retrieve top-k nearest neighbors from the pre-built FAISS index.
- Each neighbor provides ~128 tokens of context surrounding the matched passage.

Chunked Cross-Attention (CCA):
- Every third transformer block contains a CCA layer after the self-attention layer.
- Tokens in chunk cᵢ cross-attend to the retrieved neighbors for cᵢ.
- Retrieved content does not attend to the input (asymmetric attention).
- CCA enables each generation chunk to be informed by relevant retrieved knowledge.

Training:
- Train with retrieval active — the model learns to use retrieved context from the start.
- Frozen retriever (BERT) — only the main model and CCA weights are updated.
- Loss is standard language modeling loss — retrieval improves predictions by providing relevant context.

RETRO Performance

RETRO is the architectural proof that knowledge storage and knowledge reasoning can be decoupled — demonstrating that relatively small language models become powerful knowledge engines when coupled with massive retrieval databases, establishing the blueprint for the retrieval-augmented generation paradigm that now pervades production LLM systems.