Neural Architecture Highway

Highway Networks are deep feedforward networks that use gating mechanisms to regulate information flow across layers — extending skip connections with learnable gates that control how much information passes through the transformation versus the skip path.

How Do Highway Networks Work?

• Formula: $y = T(x) \cdot H(x) + C(x) \cdot x$ where $T$ is the transform gate and $C$ is the carry gate.
• Simplification: Typically $C = 1 - T$: $y = T(x) \cdot H(x) + (1 - T(x)) \cdot x$.
• Gate: $T(x) = \sigma(W_T x + b_T)$ (learned sigmoid gate).
• Paper: Srivastava et al. (2015).

Why It Matters

• Pre-ResNet: One of the first architectures to successfully train 50-100+ layer networks.
• Learned Skip: Unlike ResNet's fixed skip connections ($y = F(x) + x$), Highway Networks learn when to skip.
• LSTM Connection: Highway Networks are essentially feedforward LSTMs — same gating principle.

Highway Networks are LSTM gates for feedforward networks — the learned bypass mechanism that preceded and inspired ResNet's simpler identity shortcuts.