Multimodal

A multimodal model is a single network that accepts more than one kind of input — text, images, audio, video — and projects all of them into one shared embedding space, so a question asked in words can be answered about a picture. The word names an interface, not an architecture. Almost every production multimodal system running today is a language model with encoders bolted to its front, and nearly all the interesting engineering lives in the seam between them.

The encoder is where each modality actually gets solved, and it is almost always pretrained and then frozen. A vision encoder is a Vision Transformer trained contrastively against text: CLIP ViT-L/14 at 304M parameters across 24 transformer layers, SigLIP-SO400M at 400M, InternViT-6B at 6B, and at the extreme the 22B ViT inside PaLI-X. It cuts the image into fixed 14 x 14 pixel patches and emits one embedding per patch, which makes resolution and token count the same number wearing different clothes: a 336 x 336 input is 24 x 24 patches and 576 vectors, a 384 x 384 input is 729. Zero-shot ImageNet top-1 separates the families cleanly, from SigLIP-SO400M at 83.2% through DINOv2-g at 86.5% to InternViT-6B at 88.2%.

The projector is the cheapest component in the system and the one that decides whether it works. It carries well under 1% of total parameters and is frequently the only thing trained during alignment. A single linear layer over 600,000 image-text pairs was enough to give a frozen CLIP encoder and a frozen language model genuine visual reasoning. The current default is a two-layer MLP computing W2 . GELU(W1 . Zv). BLIP-2 instead uses a Q-Former, a small transformer whose 32 learnable query vectors cross-attend to the patch features and compress any number of patches down to a fixed 32 outputs. Flamingo and Idefics use a perceiver resampler feeding gated cross-attention layers interleaved between the frozen backbone's own blocks, which preserves text-only performance precisely because the backbone never changes.

Images are expensive in a way that text is not, and the bill arrives in the context window. A single 1024 x 1024 image at high detail costs roughly 1,500 prompt tokens, about 2,000 English words spent on one photograph, and providers differ by more than a factor of five on identical input. Qwen2.5-VL runs from 144 tokens at 336 x 336 up to 10,549 for a single 3840 x 2160 frame. On one identical 1024 x 1024 image, GPT-4o charges about 765 tokens at high detail, Claude Sonnet 4.6 roughly 1,400, and Llama 3.2 Vision about 1,601 at 1120 x 1120, while Gemini 2.5 Pro bills roughly 258 per 768 x 768 tile. The resolution strategy matters more than the encoder: for a 1 megapixel image, fixed 336 px costs 576 tokens at baseline compute, fixed 384 px costs 729 at 1.3x, AnyRes with pixel shuffle at r=3 costs 640 at 1.1x, and native resolution costs 5,329 at 9.3x.

Audio and video are the same problem with a time axis attached, and the time axis is what kills you. An audio encoder turns a waveform into a mel spectrogram and then roughly 50 frames per second of embeddings, which stays tractable because speech is low-bandwidth. Video is not: a frame is an image, images cost between 258 and 10,549 tokens each, and so the sampling rate is the entire design. Production systems sample 0.5 to 2 frames per second and pool aggressively afterward. At 1 frame per second with pooling an hour of footage costs about 60,000 tokens, very nearly half a 128k window spent before the question is asked. The economics survive only because of that pooling: a 10 minute clip analysed this way costs around 0.03 USD of input at 3 USD per million tokens.

Serving a multimodal model moves the bottleneck from decode to prefill, which breaks most of the intuitions built on text-only deployment. The encoder itself is nearly free, a forward pass running in roughly 3 ms on an H100, but it hands the backbone a prompt thousands of tokens long before the user's question is even appended. Carrying 1,600 image tokens through a 90B model costs about 70 MB of KV cache per request during prefill alone, so concurrency collapses long before a text-only workload would notice. A vision query runs five to ten times the cost of a text query end to end, which is why routing text-only requests away is worth roughly 17% of a mixed workload's spend. Voice stacks have the tightest budget of all: 50 to 100 ms for voice activity detection, 100 to 300 ms for streaming speech recognition, 400 ms to the model's first token and 50 to 400 ms to first audio out, totalling 600 to 1,200 ms before the system sounds slow.

Native multimodal training changes what the model can do, not merely what it can read. Bolting an encoder onto a finished language model produces a system that describes images; training the modalities jointly produces one that reasons across them. The mechanical obstacle is position encoding, because a patch grid has height and width while text has only sequence order. M-RoPE decomposes the rotary embedding into separate temporal, height and width components, which is what lets Qwen2-VL accept arbitrary resolutions without retraining. The loss moved too: SigLIP replaced CLIP's softmax with a sigmoid scoring each image-text pair independently, removing the enormous-batch requirement contrastive training had always carried. The payoff shows up on documents rather than captions, with current systems near 92% on OCRBench and 88% on ChartQA while VQAv2 has sat around 83% long enough to stop being informative.

The failure modes are modality-specific and none of them look like a text hallucination. An encoder that never saw 14 px glyphs at training resolution will confidently misread a downscaled table, because the information was destroyed before the language model was involved. A Q-Former compressing to 32 tokens discards spatial detail a linear projector would have kept, so counting and layout degrade while captioning does not. Frame pooling averages away the one frame that mattered. Each of these is diagnosable by checking the token count first: if the image arrived as 144 tokens, no amount of prompting recovers what the encoder discarded.

Read multimodal through a token budget lens rather than a perception lens: the model does not see an image, it receives a number of vectors fixed entirely by resolution and sampling rate, and every quality decision available to you — how far to downscale, how many frames per second, which connector, whether to pool — is a decision about how many of those vectors reach the backbone. Get the budget right and a 400M encoder answers document questions correctly; get it wrong and no backbone at any size recovers what was discarded at the patch grid.