Lipschitz Constrained Networks

Lipschitz Constrained Networks are neural networks architecturally designed or trained to have a bounded Lipschitz constant — ensuring that the network's predictions cannot change faster than a specified rate, providing built-in robustness and stability guarantees.

Methods to Constrain Lipschitz Constant

• Spectral Normalization: Divide weight matrices by their spectral norm at each layer.
• Orthogonal Weights: Constrain weight matrices to be orthogonal ($W^TW = I$) — Lipschitz constant exactly 1.
• GroupSort Activations: Replace ReLU with GroupSort for tighter Lipschitz bounds.
• Gradient Penalty: Penalize the gradient norm during training to encourage small Lipschitz constant.

Why It Matters

• Guaranteed Robustness: A network with Lipschitz constant $L=1$ cannot be fooled by any perturbation that doesn't genuinely change the input class.
• Certified Radius: $L$ directly gives a certified robustness radius without expensive verification.
• Stability: Lipschitz-constrained networks are numerically more stable during training and inference.

Lipschitz Constrained Networks are sensitivity-bounded models — architecturally ensuring that outputs change smoothly and predictably with inputs.