Tensorrt

TensorRT Optimization

What is TensorRT?
NVIDIA TensorRT is an SDK for high-performance deep learning inference. It optimizes models for NVIDIA GPUs, providing significant speedups.

Optimizations Applied
| Optimization | Description |
|--------------|-------------|
| Layer fusion | Combine operations into single kernels |
| Precision calibration | INT8/FP16 quantization |
| Kernel auto-tuning | Select best kernel for hardware |
| Memory optimization | Efficient memory allocation |
| Dynamic tensor memory | Reuse memory during inference |

Conversion Pipeline

PyTorch → [Export] → ONNX → [TensorRT Build] → TRT Engine

Building TensorRT Engine

From ONNX

import tensorrt as trt

logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)

# Parse ONNX
with open("model.onnx", "rb") as f:
    parser.parse(f.read())

# Build config
config = builder.create_builder_config()
config.set_flag(trt.BuilderFlag.FP16)  # Enable FP16
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30)  # 1GB

# Build engine
engine = builder.build_serialized_network(network, config)

# Save engine
with open("model.trt", "wb") as f:
    f.write(engine)

Running TensorRT Engine

import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit

# Load engine
runtime = trt.Runtime(logger)
with open("model.trt", "rb") as f:
    engine = runtime.deserialize_cuda_engine(f.read())

context = engine.create_execution_context()

# Allocate buffers, run inference...

TensorRT-LLM
For LLMs, use NVIDIA's TensorRT-LLM:

# Build optimized LLM engine
python build.py
    --model_dir ./llama-hf
    --dtype bfloat16
    --output_dir ./llama-trt

Features:
- Optimized attention kernels
- Inflight batching
- PagedAttention support
- Multi-GPU support

Performance Comparison
| Framework | Throughput | Latency |
|-----------|------------|---------|
| PyTorch | Baseline | Baseline |
| ONNX Runtime | 1.5-2x | 0.7x |
| TensorRT | 2-4x | 0.3-0.5x |
| TensorRT-LLM | 3-5x | 0.2-0.4x |

When to Use TensorRT
| Scenario | Recommendation |
|----------|----------------|
| NVIDIA GPU production inference | Yes |
| Need lowest latency | Yes |
| Rapid prototyping | Overhead may not be worth it |
| Cross-platform deployment | Use ONNX instead |