Model Compression for Mobile

Model compression for mobile encompasses techniques to reduce model size and computational requirements so that machine learning models can run efficiently on smartphones, tablets, IoT devices, and other resource-constrained platforms.

Why Compression is Necessary

• Memory: Mobile devices have 4–12GB RAM shared with the OS and other apps — a 7B parameter model in FP16 requires ~14GB.
• Storage: App store size limits and user expectations constrain model size to megabytes rather than gigabytes.
• Compute: Mobile CPUs, GPUs, and NPUs are far less powerful than data center hardware.
• Battery: Inference draws power — over-computation drains batteries and generates heat.
• Latency: Users expect instant responses — model must be fast enough for real-time interaction.

Compression Techniques

• Quantization: Reduce numerical precision from FP32 → FP16 → INT8 → INT4. Cuts model size by 2–8× with minimal quality loss. INT4 quantization is commonly used for on-device LLMs.
• Pruning: Remove redundant weights (near-zero values) or entire neurons/attention heads. Structured pruning removes entire channels for hardware-friendly speedups.
• Knowledge Distillation: Train a small "student" model to mimic a large "teacher" model. The student is compact but retains much of the teacher's capability.
• Architecture Optimization: Use efficient architectures designed for mobile — MobileNet, EfficientNet, SqueezeNet for vision; TinyLlama, Phi-3-mini for language.
• Weight Sharing: Multiple network connections share the same weight value, reducing unique parameters.
• Low-Rank Factorization: Decompose large weight matrices into products of smaller matrices, reducing parameters.

Mobile-Specific Optimizations

• Operator Fusion: Combine multiple operations (convolution + batch norm + activation) into a single optimized kernel.
• Hardware-Aware Optimization: Optimize for specific hardware features (Apple Neural Engine, Qualcomm Hexagon DSP, Google TPU in Pixel).
• Dynamic Shapes: Handle variable input sizes efficiently without padding waste.

Frameworks: TensorFlow Lite, Core ML, ONNX Runtime, NCNN, MNN, ExecuTorch.

Current State: On-device LLMs (3B–7B parameters with 4-bit quantization) now run on flagship smartphones, enabling local assistants, text generation, and code completion without cloud connectivity.

Model compression is the enabling technology for on-device AI — without it, modern neural networks are simply too large for mobile deployment.