Binarized Neural Networks (Bnn)

Binarized Neural Networks (BNN) are a specific implementation framework for training and deploying binary neural networks — using the Straight-Through Estimator (STE) to handle the non-differentiable sign function during backpropagation.

What Is a BNN?

• Forward Pass: Binarize weights and activations using the sign function ($+1$ if $x geq 0$, else $-1$).
• Backward Pass: The sign function has zero gradient almost everywhere. The STE uses the gradient of a smooth approximation (hard tanh or identity) instead.
• Latent Weights: Full-precision "shadow" weights are maintained for gradient accumulation, then binarized for the forward pass.

Why It Matters

• Pioneering: Courbariaux et al. (2016) demonstrated the first practical BNN training procedure.
• Foundation: All subsequent binary/ternary network methods build on the STE trick introduced here.
• FPGA Deployment: BNNs are the go-to architecture for FPGA-based inference accelerators.

Binarized Neural Networks are the engineering blueprint for 1-bit AI — solving the fundamental training challenge of discrete-valued networks.