Cortex

Cortex: Serverless ML Infrastructure

Overview
Cortex is an open-source platform for deploying machine learning models as production-ready web APIs. It automates the infrastructure underlying model serving on AWS (EC2, EKS), abstracting away Kubernetes and Docker complexity.

Key Features

1. Unified Config
Deploy TensorFlow, PyTorch, Scikit-learn, or ONNX models using a simple `cortex.yaml` file.

2. Autoscaling
Automatically scales the number of replicas based on request traffic (Requests Per Second) or GPU utilization. Scales to zero to save costs.

3. Spot Instances
Built-in support for AWS Spot Instances, potentially saving 70-90% on compute costs, with auto-recovery if instances are reclaimed.

4. Rolling Updates
Updates APIs without downtime.

Configuration Example

# cortex.yaml
- name: sentiment-analyzer
  kind: RealtimeAPI
  predictor:
    type: python
    path: predictor.py
  compute:
    cpu: 1
    gpu: 1  # Uses GPU instance
    mem: 4G
  autoscaling:
    min_replicas: 1
    max_replicas: 10

Python Predictor

# predictor.py
class PythonPredictor:
    def __init__(self, config):
        self.model = load_model()

    def predict(self, payload):
        return self.model.inference(payload["text"])

Status
Note: Cortex was acquired by Databricks, and the open-source project is no longer actively maintained as of late 2021/2022.
Modern Alternatives include:
- BentoML: For packaging models.
- Ray Serve: For scalable serving.
- KServe: For Kubernetes native serving.
- AWS SageMaker: Managed alternative.