Block-Recurrent Transformer

Block-Recurrent Transformer is the hybrid architecture that partitions input sequences into fixed-size blocks, applies full transformer self-attention within each block, and passes a learned recurrent state between blocks to propagate long-range context — combining the high-quality local attention of transformers with the unbounded-length capability of recurrent networks, enabling processing of arbitrarily long sequences with bounded O(block_size²) memory per step.

What Is a Block-Recurrent Transformer?

• Definition: A sequence model that divides input into non-overlapping blocks of B tokens, applies standard multi-head self-attention within each block, and transmits a fixed-size recurrent state vector from one block to the next — the recurrent state carries compressed information from all previous blocks.
• Within-Block: Full transformer attention — every token in the block attends to every other token in the same block. This provides the rich, parallel, high-quality representations that transformers excel at.
• Between-Block: Recurrent state update — a learned function (cross-attention to previous state, or gated RNN-style update) compresses the current block's output into a state vector passed to the next block.
• Bounded Memory: Memory usage is O(B²) per block plus O(d_state) for the recurrent state — independent of total sequence length, enabling arbitrarily long inputs.

Why Block-Recurrent Transformer Matters

• Infinite Context Length: Unlike standard transformers with fixed context windows, block-recurrent models process sequences of any length — the recurrent state theoretically carries information from the entire history.
• Bounded Compute Per Step: Each block requires O(B²) attention compute — regardless of how many blocks have been processed before. This makes both training and inference costs predictable and controllable.
• Best of Both Worlds: Full transformer attention within blocks captures rich local interactions; recurrence between blocks captures long-range dependencies — combining the strengths of both paradigm families.
• Streaming Capability: Can process input as a stream of blocks without storing the full sequence — suitable for real-time applications where input arrives continuously.
• Memory-Efficient Training: Gradient computation requires storing only O(number_of_blocks × d_state) recurrent states rather than the full O(sequence_length × d_model) activation cache.

Block-Recurrent Architecture

Forward Pass Per Block:
- Input: block of B tokens + recurrent state from previous block.
- Cross-attention: block tokens attend to previous recurrent state (context injection).
- Self-attention: standard multi-head attention within the B tokens.
- State update: compress block output into new recurrent state via attention pooling or gated combination.
- Output: processed B tokens + updated recurrent state.

Recurrent State Mechanisms:
- Cross-Attention State: Fixed number of state vectors; new block cross-attends to state for context, then state is updated via cross-attention from state to block output.
- Gated State Update: s_new = gate × s_old + (1 − gate) × compress(block_output) — similar to LSTM/GRU update.
- Memory-Augmented: State includes a small memory matrix that tokens can read from and write to — richer state representation.

Comparison With Other Long-Context Methods

Block-Recurrent Transformer is the architectural bridge between the transformer and recurrent paradigms — partitioning the challenging problem of long-range sequence modeling into a solved local problem (transformer attention within blocks) and a manageable global problem (recurrent state between blocks), achieving unbounded context with bounded resources.