Fusion-in-Decoder (Fid)

Fusion-in-Decoder (FiD) is the retrieval-augmented generation architecture that processes multiple retrieved documents independently through the encoder and fuses information from all documents in the decoder through cross-attention — enabling scalable multi-document reasoning without the context-length limitations of concatenation-based approaches — the architectural pattern that became the standard backbone for retrieval-augmented question answering and knowledge-grounded generation systems.

What Is Fusion-in-Decoder?

• Definition: An encoder-decoder architecture (based on T5 or BART) where each retrieved passage is encoded independently with the query by the encoder, producing separate representations, and the decoder cross-attends to all encoder outputs simultaneously — performing information fusion across documents at the decoding stage.
• Independent Encoding: Each of k retrieved passages is concatenated with the query and encoded separately: hᵢ = Encoder(query ⊕ passageᵢ). This avoids the O(k²·n²) cost of concatenating all passages and running a single encoder.
• Decoder Fusion: The decoder cross-attends to the concatenated encoder outputs [h₁; h₂; ...; hₖ] — each decoder token can attend to any position in any retrieved passage, enabling information synthesis across documents.
• Scalability: Since encoding is independent and parallelizable, FiD scales to 50–100 retrieved passages without exceeding memory limits — far more context than concatenation allows.

Why FiD Matters

• Scales to Many Documents: Concatenating 50 passages of 200 tokens creates a 10,000-token input — exceeding most encoder limits. FiD encodes each passage independently (200 tokens each) and fuses in the decoder — handling any number of passages.
• State-of-the-Art QA: FiD achieved top results on Natural Questions, TriviaQA, and other open-domain QA benchmarks — demonstrating that multi-document fusion in the decoder is more effective than early fusion (concatenation) or late fusion (reranking).
• Information Aggregation: When the answer requires combining facts from multiple documents (multi-hop reasoning), FiD's decoder naturally learns to attend to different passages for different parts of the answer.
• Foundation for ATLAS and RAG: FiD became the generator component in ATLAS and influenced the design of many RAG systems — its encoder-decoder fusion pattern is the standard architectural choice for retrieval-augmented generation.
• Efficient Encoding: Independent passage encoding enables passage-level caching — when the corpus is fixed, encoder outputs can be pre-computed and reused across queries.

FiD Architecture

Encoding Phase (Parallelized):
- For each retrieved passage pᵢ (i = 1, ..., k):
- Concatenate: inputᵢ = "question: [query] context: [passageᵢ]"
- Encode: hᵢ = T5Encoder(inputᵢ) → [seq_lenᵢ × d_model]
- All k passages encoded independently — embarrassingly parallel.
- Total encoder memory: O(k × max_passage_len × d_model).

Fusion Phase (Decoder):
- Concatenate all encoder outputs: H = [h₁; h₂; ...; hₖ] → [k × seq_len × d_model].
- Decoder cross-attention attends to full H — each generated token can access any position in any passage.
- Decoder generates the answer auto-regressively.

FiD Behavior Analysis

Log-linear improvement: Performance scales logarithmically with number of passages — strong early gains with diminishing returns beyond 50 passages.

FiD vs. Alternative Fusion Strategies

Fusion-in-Decoder is the architectural workhorse of retrieval-augmented generation — solving the fundamental scalability problem of multi-document reasoning by separating independent passage understanding (encoder) from cross-document information synthesis (decoder), enabling systems to effectively aggregate knowledge from dozens of retrieved documents into coherent, informed answers.