Kosmos

KOSMOS is a multimodal large language model (MLLM) developed by Microsoft — trained from scratch on web-scale multimodal corpora to perceive general modalities, follow instructions, and perform in-context learning (zero-shot and few-shot).

What Is KOSMOS?

• Definition: A "Language Is Not All You Need" foundation model.
• Architecture: Transformer decoder (Magneto) that accepts text, audio, and image embeddings as standard tokens.
• Training: Monolithic training on text (The Pile), image-text pairs (LAION), and interleaved data (Common Crawl).

Why KOSMOS Matters

• raven's Matrices: Demoed the ability to solve IQ tests (pattern completion) zero-shot.
• OCR-Free: Reads text in images naturally without a separate OCR engine.
• Audio: KOSMOS-1 handled vision; KOSMOS-2 and variants added grounding and speech.
• Grounding: Can output bounding box coordinates as text tokens to localize objects.

KOSMOS is a true generalist model — treating images, sounds, and text as a single unified language for the transformer to process.