Interleaved Image-Text Generation

Interleaved Image-Text Generation is the process of generating coherent sequences containing both text and images — enabling models to write illustrated articles, create instructional manuals with diagrams, or tell visual stories that flow naturally between modalities.

What Is Interleaved Generation?

• Definition: Output stream contains sequence of $[T_1, T_2, I_1, T_3, I_2, ...]$.
• Contrast: Most models are "Text-to-Image" (generating one image) or "Image-to-Text" (captioning). Interleaved models do both continuously.
• Models: CM3, MM-Interleaved, GPT-4V (in principle), Gemini.

Why It Matters

• Rich Communication: Humans naturally mix speech, gesture, and showing objects; AI should too.
• Storytelling: Can generate a children's book with consistent characters and plot.
• Documentation: Automatically generating "How-To" guides with screenshots inserted at the right steps.

Technical Challenges

• Modality Gap: Aligning the vector space of text tokens and image pixels/tokens.
• Coherence: Ensuring the image $I_2$ is consistent with the text $T_1$ and previous image $I_1$.
• Tokenization: Requires efficient visual tokenizers (like VQ-VAE) to treat images as "words" in the vocabulary.

Interleaved Image-Text Generation is the future of automated content creation — moving beyond static media to dynamic, multi-modal narratives.