Self-Gating

Self-Gating is a mechanism where a neural network layer gates its own activations using a function of the same input — the input multiplied by a sigmoid (or similar gate) of itself, allowing the network to selectively amplify or suppress its features.

How Does Self-Gating Work?

• Formula: $y = x \cdot \sigma(Wx + b)$ where $\sigma$ is a gate function (sigmoid, tanh).
• Swish: The simplest self-gating: $x \cdot \sigma(x)$ (no learned gate parameters).
• SE-Net: Self-gating via channel attention: learn per-channel gates from global statistics.
• GLU: Gated Linear Unit splits input into two halves — one gates the other.

Why It Matters

• Expressiveness: Self-gating allows multiplicative interactions, which are more expressive than additive transformations.
• Feature Selection: The gate learns to suppress irrelevant features and amplify important ones.
• Foundation: Self-gating is the core principle behind Swish, GLU, SwiGLU, and SE-Net.

Self-Gating is the input controlling its own flow — a powerful mechanism where features decide their own importance.