Clip (contrastive Language-Image Pre-Training)

CLIP (Contrastive Language-Image Pre-training) aligns text and image embeddings for zero-shot visual understanding. Approach: Train image encoder and text encoder jointly such that matching image-text pairs have similar embeddings, non-matching pairs have different embeddings. Contrastive learning across modalities. Training data: 400M image-text pairs from internet (WebImageText dataset). Scale is key. Architecture: Image encoder (ViT or ResNet), text encoder (Transformer), learned projection to shared embedding space, contrastive loss over batch. Zero-shot inference: Encode class names as text ("a photo of a {class}"), encode image, classify by highest similarity to text embeddings. Prompt engineering: "A photo of a {class}" works better than just class name. Prompt ensembling improves results. Capabilities: Zero-shot classification, image-text retrieval, supports many visual tasks without task-specific training. Limitations: Struggles with fine-grained categories, counting, spatial relationships. Impact: Foundation for many multimodal models, text-conditional image generation (DALL-E, Stable Diffusion use CLIP), revolutionized zero-shot visual recognition.