H3 (hungry Hungry Hippos)

H3 (Hungry Hungry Hippos) is a hybrid deep learning architecture that combines State Space Model (SSM) layers with attention mechanisms to get the best of both worlds — the linear-time efficiency of SSMs for long sequences and the in-context learning ability of attention.

Architecture Design

• SSM Layers: The majority of layers use efficient SSM computation (building on S4) to process sequences in O(N) time, handling long-range dependencies without the quadratic cost of full attention.
• Attention Layers: A small number of standard attention layers are interspersed to provide the model with the ability to perform precise token-to-token comparisons — something SSMs struggle with on their own.
• Two SSM Projections: H3 uses two SSM-parameterized projections — one acting as a shift (moving information along the sequence) and another as a diagonal linear map — multiplied together before an output projection.

Why "Hungry Hungry Hippos"?

The name is a playful reference to the board game, reflecting how the model's SSM layers "gobble up" long sequences efficiently. The H3 paper (by Dan Fu, Tri Dao, et al.) showed that the architecture could match Transformer performance on language modeling while being significantly faster on long sequences.

Significance

• Bridge to Mamba: H3 was a critical stepping stone between S4 and Mamba. It demonstrated that SSMs needed attention-like capabilities, motivating the development of selective state spaces in Mamba.
• FlashAttention Connection: H3 was developed by the same research group behind FlashAttention, and insights from both projects cross-pollinated.
• Practical Impact: Showed that hybrid SSM-attention models could achieve state-of-the-art perplexity on language modeling benchmarks while being more efficient than pure Transformers on long sequences.