Integer-Only Inference

Integer-Only Inference is a deployment strategy where the entire neural network forward pass uses integer arithmetic exclusively — eliminating all floating-point operations to enable fast, power-efficient execution on edge devices and microcontrollers.

What Is Integer-Only Inference?

• Mechanism: All weights, activations, and intermediate computations use INT8 (or INT4).
• Quantization: Scale factors are pre-computed and fused. $y = GEMM_{int}(W_{int8}, x_{int8}) \cdot scale$.
• No Float: Even non-linearities (ReLU, Softmax) are approximated with integer lookup tables.
• Frameworks: TensorFlow Lite, ONNX Runtime, TVM.

Why It Matters

• Microcontrollers: ARM Cortex-M has no FPU. Integer-only is the *only* option.
• Speed: INT8 GEMM is 2-4x faster than FP32 on GPUs (Tensor Cores).
• Power: Integer ops consume significantly less energy than floating-point.

Integer-Only Inference is deployment-grade quantization — the final step to make AI models run on the smallest, cheapest silicon.