Llava (large Language and Vision Assistant)

LLaVA (Large Language and Vision Assistant) is an open-source multimodal model — that combines a vision encoder (CLIP ViT-L) with an LLM (Vicuna/LLaMA) to creating a "visual chatbot" with capabilities similar to GPT-4 Vision.

What Is LLaVA?

• Definition: End-to-end trained large multimodal model.
• Architecture: Simple projection layer connects CLIP (frozen) to LLaMA (fine-tuned).
• Data Innovation: Used GPT-4 (text-only) to generate multimodal instruction-following data from image captions and bounding boxes.
• Philosophy: Simple architecture + High-quality instruction data = SOTA performance.

Why LLaVA Matters

• Simplicity: Unlike the complex Q-Former of BLIP-2, LLaVA just uses a linear projection (MLP).
• Open Source: The code, data, and weights are fully open, driving the open VLM community.
• Science QA: Achieved state-of-the-art on reasoning benchmarks.

Training Stages

1. Feature Alignment: Pre-training to align image features to word embeddings.
2. Visual Instruction Tuning: Fine-tuning on the GPT-4 generated instruction data (conversations, reasoning).

LLaVA is the "Hello World" of modern VLMs — its simple, effective recipe became the standard basline for nearly all subsequent open-source multimodal research.