Container Registry

Container Registries for ML

Why Container Registries?
Store and deploy ML model containers with versioning, security scanning, and access control.

Major Registries
| Registry | Provider | Features |
|----------|----------|----------|
| ECR | AWS | IAM integration, scanning |
| GCR/Artifact Registry | GCP | Multi-region, scanning |
| ACR | Azure | AAD integration |
| Docker Hub | Docker | Public images |
| Harbor | Self-hosted | Enterprise features |

ECR Setup

# Create repository
aws ecr create-repository --repository-name llm-inference

# Authenticate Docker
aws ecr get-login-password | docker login --username AWS --password-stdin
    123456789.dkr.ecr.us-east-1.amazonaws.com

# Build and push
docker build -t llm-inference .
docker tag llm-inference:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/llm-inference:v1
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/llm-inference:v1

Image Tagging Strategy

# Tag by version
llm-inference:1.0.0
llm-inference:1.0.1

# Tag by git commit
llm-inference:abc1234

# Tag by model version
llm-inference:gpt4-v2

# Tag by date
llm-inference:2024-01-15

ML-Specific Considerations
| Consideration | Solution |
|---------------|----------|
| Large images (10GB+) | Multi-stage builds, layer caching |
| Model weights | Separate from code, mount at runtime |
| GPU dependencies | Use NVIDIA base images |
| Security | Scan for vulnerabilities |

Dockerfile for ML

# Multi-stage build
FROM python:3.11-slim as builder
COPY requirements.txt .
RUN pip wheel --no-cache-dir --wheel-dir=/wheels -r requirements.txt

FROM nvidia/cuda:12.1-runtime-ubuntu22.04
COPY --from=builder /wheels /wheels
RUN pip install --no-cache /wheels/*

COPY app/ /app/
WORKDIR /app

# Dont include model weights in image
# Mount from S3 or volume at runtime
ENTRYPOINT ["python", "serve.py"]

Kubernetes ImagePullPolicy

spec:
  containers:
  - name: llm-server
    image: 123456.dkr.ecr.us-east-1.amazonaws.com/llm-inference:v1.2.0
    imagePullPolicy: IfNotPresent  # Cache locally

Best Practices
- Use immutable tags (version, not :latest)
- Enable vulnerability scanning
- Clean up old images (lifecycle policies)
- Use multi-stage builds for smaller images
- Store model weights separately from code