Triton

Triton Inference Server

What is Triton?
NVIDIA Triton Inference Server is a production-grade serving platform that supports multiple frameworks, dynamic batching, and GPU orchestration.

Key Features
| Feature | Description |
|---------|-------------|
| Multi-framework | PyTorch, TensorFlow, ONNX, TensorRT |
| Dynamic batching | Automatically batch requests |
| Model versioning | Serve multiple model versions |
| Ensemble models | Chain models together |
| GPU/CPU execution | Flexible resource allocation |
| Metrics | Prometheus metrics built-in |

Model Repository Structure

Model Configuration

# config.pbtxt
name: "llama"
platform: "onnxruntime_onnx"
max_batch_size: 16

input [
  {
    name: "input_ids"
    data_type: TYPE_INT64
    dims: [ -1 ]  # Variable length
  }
]

output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [ -1, 32000 ]
  }
]

dynamic_batching {
  preferred_batch_size: [ 4, 8, 16 ]
  max_queue_delay_microseconds: 50000
}

Running Triton

# Start server
docker run --gpus all -p 8000:8000 -p 8001:8001
  -v /path/to/models:/models
  nvcr.io/nvidia/tritonserver:24.01-py3
  tritonserver --model-repository=/models

Client Usage

import tritonclient.http as httpclient

client = httpclient.InferenceServerClient("localhost:8000")

# Create input
inputs = [httpclient.InferInput("input_ids", [1, 10], "INT64")]
inputs[0].set_data_from_numpy(input_array)

# Infer
outputs = [httpclient.InferRequestedOutput("logits")]
response = client.infer("llama", inputs, outputs=outputs)
result = response.as_numpy("logits")

Dynamic Batching
Triton automatically batches requests:

Benefits:
- Better GPU utilization
- Higher throughput
- Configurable latency trade-offs

Scaling
- Horizontal: Multiple Triton instances behind load balancer
- Multi-GPU: Multiple model instances across GPUs
- Kubernetes: Use Triton Inference Server Operator