Glu

GLU (Gated Linear Unit) is a gating mechanism that splits the input into two halves — one serves as the "content" and the other as the "gate" — implemented as $\text{GLU}(x, y) = x \otimes \sigma(y)$ where $otimes$ is element-wise multiplication.

How Does GLU Work?

• Split: Given input of dimension $2d$, split into $x$ and $y$ of dimension $d$ each.
• Gate: $\text{GLU}(x, y) = x \otimes \sigma(y)$
• Variants: Bilinear ($x otimes y$), SwiGLU ($x \otimes \text{Swish}(y)$), GeGLU ($x \otimes \text{GELU}(y)$).
• Paper: Dauphin et al. (2017).

Why It Matters

• LLM Standard: SwiGLU/GeGLU variants are the default FFN activation in modern LLMs (LLaMA, PaLM, Gemma).
• Gradient Flow: The linear path through $x$ provides easy gradient flow (like a skip connection within the activation).
• Performance: GLU variants consistently outperform standard ReLU/GELU FFN blocks in transformers.

GLU is the half-and-half activation — splitting inputs into content and gate for multiplicative feature selection.