Orthogonal Convolutions

Orthogonal Convolutions are convolutional layers with orthogonality constraints on the kernel matrices — ensuring that the convolutional transformation preserves the norm of feature maps, resulting in a layer-wise Lipschitz constant of exactly 1.

Implementing Orthogonal Convolutions

• Cayley Transform: Parameterize the convolution kernel using the Cayley transform of a skew-symmetric matrix.
• Björck Orthogonalization: Iteratively project weight matrices toward orthogonality during training.
• Block Convolution: Reshape the convolution into a matrix operation and enforce orthogonality on the matrix.
• Householder Parameterization: Compose Householder reflections to build orthogonal transformations.

Why It Matters

• Exact Lipschitz: Each orthogonal layer has Lipschitz constant exactly 1 — the full network's Lipschitz constant equals 1.
• No Signal Loss: Orthogonal layers preserve feature map norms — no vanishing or exploding signals.
• Certifiable: Networks with orthogonal convolutions have tight, easily computable robustness certificates.

Orthogonal Convolutions are norm-preserving feature extractors — convolutional layers that maintain exact Lipschitz-1 behavior for provably robust networks.