Cog

Cog is an open-source tool by Replicate that packages machine learning models into standard, production-ready Docker containers — solving the "works on my machine" problem by using a simple cog.yaml configuration file to automatically generate Dockerfiles with correct CUDA drivers, Python versions, system dependencies, and a standardized HTTP prediction API, turning any Python model into a deployable container without writing a single line of Docker configuration.

What Is Cog?

Definition: A command-line tool (pip install cog) that takes a Python prediction class and a YAML configuration file and produces a fully functional Docker container with an HTTP API at /predictions — handling all the CUDA, system library, and Python dependency complexity automatically.
The Problem: Data scientists train models in Jupyter notebooks with a chaotic mix of pip, conda, system packages, and specific CUDA versions. Getting this into a Docker container requires deep DevOps knowledge — writing Dockerfiles, managing CUDA driver compatibility, setting up HTTP endpoints, and handling GPU memory.
The Solution: Define dependencies in cog.yaml, write a predict() function, run `cog build` — done. Cog generates the Dockerfile, builds the container, and provides a standardized API.

How Cog Works

Step	What You Do	What Cog Does
1. Define dependencies	Write cog.yaml with Python version + packages	Generates multi-stage Dockerfile
2. Write predict function	Python class with setup() and predict() methods	Creates HTTP /predictions endpoint
3. Build	Run `cog build`	Builds Docker image with CUDA, dependencies
4. Test locally	Run `cog predict -i [email protected]`	Runs prediction in container
5. Deploy	Push to Replicate or any Docker host	Instant API hosting

cog.yaml Example

build:
  gpu: true
  python_version: "3.10"
  python_packages:
    - torch==2.1
    - transformers==4.36
  system_packages:
    - ffmpeg

predict: "predict.py:Predictor"

predict.py Example

from cog import BasePredictor, Input, Path

class Predictor(BasePredictor):
    def setup(self):
        """Load model into memory (runs once on startup)"""
        self.model = load_model("weights/model.pt")

    def predict(self, image: Path = Input(description="Input image")) -> Path:
        """Run inference on an input image"""
        output = self.model(image)
        return Path(output)

Cog vs Alternatives

Tool	Approach	Strengths	Limitations
Cog	YAML + predict class → Docker	Simplest path to container, Replicate integration	Replicate-specific ecosystem
BentoML	Python decorators → Bento → container	More flexible, multi-model support	More complex API
Docker (manual)	Write Dockerfile from scratch	Full control	Requires Docker expertise, CUDA pain
TorchServe / TF Serving	Framework-specific server	Optimized for specific framework	Framework lock-in
Triton	NVIDIA inference server	Best GPU performance	Complex configuration

Cog is the fastest path from ML model to production Docker container — eliminating the DevOps complexity of CUDA drivers, system dependencies, and HTTP API setup through a simple YAML configuration and Python prediction class, enabling data scientists to package any model into a standardized, deployable container without Docker expertise.

Cog

Take cog further