Edge

Edge and On-Device Inference

Why Edge Inference?
Run models locally on devices for privacy, low latency, offline capability, and reduced cloud costs.

Edge Deployment Targets
| Target | Typical Power | Use Case |
|--------|---------------|----------|
| Mobile phones | 1-5W | Personal AI |
| Tablets | 2-10W | Field work |
| Raspberry Pi | 2-5W | IoT, prototypes |
| Jetson | 10-30W | Robotics, cameras |
| Edge servers | 100W+ | Local inference |

Model Optimization for Edge

Quantization

from optimum.intel import OVQuantizer

quantizer = OVQuantizer.from_pretrained("distilbert-base-uncased")
quantizer.quantize(save_directory="./quantized_model", calibration_dataset=dataset)

Pruning

import torch.nn.utils.prune as prune

# Prune 30% of weights
for module in model.modules():
    if isinstance(module, torch.nn.Linear):
        prune.l1_unstructured(module, name="weight", amount=0.3)

Distillation
Train smaller model to mimic larger:

# Teacher: large model
# Student: small model for edge
loss = kl_div(student_logits / T, teacher_logits / T) * T^2

Edge Frameworks
| Framework | Vendor | Target |
|-----------|--------|--------|
| TensorFlow Lite | Google | Mobile, embedded |
| ONNX Runtime | Microsoft | Cross-platform |
| OpenVINO | Intel | Intel hardware |
| TensorRT | NVIDIA | NVIDIA GPUs |
| CoreML | Apple | Apple devices |
| MLC LLM | Open source | Any device |

MLC LLM Example

# Compile model for device
mlc_llm compile ./model --target android

# Run on Android
mlc_chat ./compiled_model

Edge LLMs
| Model | Parameters | Target |
|-------|------------|--------|
| Gemma 2B | 2B | Mobile |
| Phi-2 | 2.7B | Edge servers |
| TinyLlama | 1.1B | Embedded |
| Qwen 0.5B | 0.5B | IoT |

Performance on Edge
| Device | Model | Tokens/sec |
|--------|-------|------------|
| iPhone 15 Pro | Llama 7B Q4 | 10-15 |
| M2 MacBook | Llama 13B Q4 | 20-30 |
| Jetson Orin | Llama 7B Q4 | 15-25 |

Best Practices
- Quantize to INT4 for smallest footprint
- Use device-specific frameworks
- Profile memory and power usage
- Consider progressive loading
- Test on actual target devices