Mlserver
MLServer is an open-source Python inference server by Seldon that serves ML models using the standardized V2 Inference Protocol — supporting multiple frameworks (Scikit-Learn, XGBoost, LightGBM, MLflow, Hugging Face Transformers) through a single unified API, providing adaptive batching that groups multiple requests into efficient tensor operations, and serving as the default inference runtime for both Seldon Core and KServe on Kubernetes, making it the production-grade serving solution for teams that need framework-agnostic model deployment.
What Is MLServer?
- Definition: A Python-based inference server (pip install mlserver) that implements the V2 Inference Protocol (standardized by KServe/NVIDIA Triton) — providing a common REST/gRPC API for any ML model regardless of framework.
- The Problem: Every ML framework has its own serving solution (TF Serving for TensorFlow, TorchServe for PyTorch). Teams with diverse model stacks need a unified serving layer. MLServer provides one server that handles any Python model.
- The V2 Protocol: A standardized inference API (originally called the "KServe Predict Protocol V2") that defines endpoints like /v2/models/{model}/infer — shared by MLServer, NVIDIA Triton, and TorchServe, enabling interchangeable backends.
Core Features
| Feature | Description | Benefit |
|---|---|---|
| Multi-Framework | Scikit-Learn, XGBoost, LightGBM, MLflow, HF, custom | One server for all your models |
| Adaptive Batching | Groups incoming requests into batches automatically | Higher GPU throughput |
| V2 Protocol | Standardized KServe/Triton-compatible API | Portable across serving platforms |
| Multi-Model Serving | Run multiple models in a single server instance | Resource efficiency |
| Custom Runtimes | Write a Python class to serve any custom model | Maximum flexibility |
| Parallel Inference | Multi-worker inference with configurable parallelism | Scale to high traffic |
Supported Runtimes
| Runtime | Framework | Install |
|---|---|---|
| mlserver-sklearn | Scikit-Learn | pip install mlserver-sklearn |
| mlserver-xgboost | XGBoost | pip install mlserver-xgboost |
| mlserver-lightgbm | LightGBM | pip install mlserver-lightgbm |
| mlserver-mlflow | MLflow models | pip install mlserver-mlflow |
| mlserver-huggingface | Transformers | pip install mlserver-huggingface |
| Custom | Any Python model | Implement MLModel class |
MLServer vs Alternatives
| Feature | MLServer | TF Serving | Triton | BentoML |
|---|---|---|---|---|
| Language | Python | C++ | C++ | Python |
| Protocol | V2 (KServe standard) | Custom TF protocol | V2 | Custom |
| Multi-Framework | Yes (via runtimes) | TensorFlow only | Yes (backends) | Yes |
| Kubernetes | Seldon Core / KServe native | Manual setup | KServe supported | BentoCloud |
| Best For | Python-first teams on K8s | TensorFlow shops | GPU-heavy, multi-framework | Rapid prototyping |
MLServer is the Python-native inference server for production model serving — providing a standardized V2 Protocol API, multi-framework support through pluggable runtimes, adaptive batching for throughput optimization, and native integration with Kubernetes orchestrators (Seldon Core, KServe) for teams that need a unified, scalable serving layer across diverse ML model stacks.