Vllm

LLM Inference Engines: vLLM and TGI

vLLM

What is vLLM?
High-throughput LLM serving engine with PagedAttention for efficient KV cache management.

Key Features
| Feature | Description |
|---------|-------------|
| PagedAttention | Non-contiguous KV cache, like virtual memory |
| Continuous batching | Add/remove requests dynamically |
| High throughput | 24x higher than HuggingFace baseline |
| OpenAI-compatible API | Drop-in replacement |

Usage

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b-chat-hf")
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=256)

prompts = ["Hello, my name is", "The capital of France is"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)

API Server

# Start OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server
    --model meta-llama/Llama-2-7b-chat-hf
    --port 8000

# Use with OpenAI client
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
response = client.chat.completions.create(model="llama", messages=[...])

Text Generation Inference (TGI)

What is TGI?
Hugging Face's production-ready LLM inference server, powering their Inference Endpoints.

Key Features
- Flash Attention 2 by default
- Continuous batching
- Quantization support (GPTQ, AWQ, bitsandbytes)
- Tensor parallelism for multi-GPU
- Built-in streaming

Running TGI

docker run --gpus all -p 8080:80
    -v /data:/data
    ghcr.io/huggingface/text-generation-inference:latest
    --model-id meta-llama/Llama-2-7b-chat-hf
    --quantize bitsandbytes-nf4

Client Usage

from huggingface_hub import InferenceClient

client = InferenceClient("http://localhost:8080")
response = client.text_generation(
    "What is deep learning?",
    max_new_tokens=100,
    stream=True
)
for token in response:
    print(token, end="", flush=True)

Comparison
| Feature | vLLM | TGI |
|---------|------|-----|
| PagedAttention | ✅ Native | ✅ Supported |
| OpenAI API | ✅ Built-in | ❌ Different API |
| Quantization | Limited | ✅ Extensive |
| Multi-GPU | ✅ Tensor parallel | ✅ Tensor parallel |
| Speculative decoding | ✅ | ✅ |
| Ease of use | Very easy | Easy |

When to Use
- vLLM: Max throughput, OpenAI-compatible API
- TGI: Hugging Face ecosystem, many quantization options