Llava
LLaVA (Large Language-and-Vision Assistant) is the pioneering open-source vision-language model that introduced visual instruction tuning — connecting a CLIP vision encoder to a LLaMA/Vicuna language model and training on GPT-4-generated visual conversation data to create a multimodal assistant that can describe images, answer visual questions, reason about visual content, and follow complex instructions involving both text and images.
What Is LLaVA?
- Definition: A multimodal model (from University of Wisconsin-Madison and Microsoft Research, 2023) that combines a pretrained CLIP ViT-L/14 vision encoder with a pretrained LLaMA/Vicuna language model through a trainable projection layer — fine-tuned on 158K visual instruction-following examples generated by GPT-4.
- Visual Instruction Tuning: The key innovation — using GPT-4 (text-only) to generate high-quality conversation, detailed description, and complex reasoning data about images (using image captions and bounding boxes as input to GPT-4), then training the multimodal model on this synthetic data.
- Architecture: CLIP ViT-L/14 encodes the image into patch embeddings → a linear projection (LLaVA 1.0) or MLP projection (LLaVA 1.5) maps visual tokens to the LLM's embedding space → visual tokens are concatenated with text tokens → the LLM generates the response.
- LLaVA 1.5: The improved version that replaced the linear projection with a 2-layer MLP, used higher resolution (336×336), and trained on 665K visual instruction examples — achieving state-of-the-art results on 11 benchmarks with a simple, reproducible architecture.
LLaVA Model Versions
| Version | Vision Encoder | LLM | Projection | Training Data | Key Improvement |
|---|---|---|---|---|---|
| LLaVA 1.0 | CLIP ViT-L/14 | Vicuna-13B | Linear | 158K | First visual instruction tuning |
| LLaVA 1.5 | CLIP ViT-L/14@336 | Vicuna-7B/13B | 2-layer MLP | 665K | Better projection, higher res |
| LLaVA 1.6 (NeXT) | CLIP ViT-L/14@672 | Mistral-7B/Vicuna-13B | MLP | 1M+ | Dynamic high resolution |
| LLaVA-OneVision | SigLIP | Qwen2-7B/72B | MLP | 3M+ | Video understanding |
Why LLaVA Matters
- Simplicity: LLaVA's architecture is remarkably simple — a vision encoder, a projection layer, and an LLM. No complex cross-attention modules, no additional encoders. This simplicity made it reproducible and extensible.
- Data-Centric Innovation: The breakthrough was the training data, not the architecture — using GPT-4 to generate visual instruction data showed that synthetic data quality matters more than architectural complexity.
- Open-Source Standard: LLaVA became the reference architecture for open-source VLMs — most subsequent models (InternVL, Cambrian, LLaVA-NeXT) follow the same encoder-projector-LLM pattern.
- Community Impact: Fully open-source (code, data, weights) — spawned hundreds of derivative models, fine-tunes, and research papers building on the LLaVA architecture.
LLaVA is the open-source vision-language model that established visual instruction tuning as the standard approach for building multimodal AI assistants — demonstrating that connecting a CLIP vision encoder to an LLM through a simple projection layer, trained on GPT-4-generated visual conversation data, produces powerful multimodal capabilities that rival proprietary systems.