Text to Video
Text-to-Video Generation is the AI capability that synthesizes coherent video sequences from natural language descriptions — extending diffusion and transformer models from static image generation to temporal sequences, requiring the model to understand scene composition, object persistence, physical dynamics, camera motion, and temporal coherence across dozens to hundreds of frames, representing one of the most challenging frontiers in generative AI.
Core Technical Challenges
| Challenge | Why It's Hard | Current Approach |
|---|---|---|
| Temporal coherence | Objects must persist across frames | 3D-aware + temporal attention |
| Physical dynamics | Objects should obey (approximate) physics | Large-scale video pretraining |
| Computational cost | Video = 30× more data than image per second | Latent space diffusion |
| Training data | Need diverse, high-quality video datasets | Web scraping + filtering |
| Evaluation | No good automated metrics for video quality | Human evaluation + FVD |
Architecture Approaches
Approach 1: Spacetime DiT (Sora-style)
[Text] → [T5/CLIP encoder] → conditioning
[Noise latent: T×H×W×C] → [3D DiT with spacetime attention] → [Video]
Approach 2: Cascaded generation
[Text] → [Generate keyframes] → [Interpolate intermediate frames] → [Super-resolve]
Approach 3: Autoregressive
[Text] → [Generate frame 1] → [Generate frame 2 conditioned on frame 1] → ...Major Systems
| System | Developer | Architecture | Key Innovation |
|---|---|---|---|
| Sora | OpenAI (2024) | Spacetime DiT | Variable resolution/duration, world simulation |
| Kling | Kuaishou (2024) | DiT + 3D VAE | Long coherent video (2+ min) |
| Gen-3 Alpha | Runway (2024) | Transformer diffusion | Fine-grained control |
| Stable Video | Stability AI | Temporal U-Net | Open-source, image-to-video |
| Veo 2 | Google DeepMind | Cascaded diffusion | High fidelity, 4K output |
| HunyuanVideo | Tencent (2024) | DiT | Open-source, long video |
Latent Video Diffusion
- Raw video: 720p × 30fps × 5sec = 1920×1080×150×3 ≈ 900M pixels → impossible to process directly.
- Solution: Encode video into latent space using 3D VAE.
- Compression: 8×8 spatial + 4× temporal compression → latent is 240×135×38×4.
- Diffusion operates in latent space → denoise → decode to pixel space.
Temporal Attention
- Spatial attention: Each frame attends to all patches within that frame.
- Temporal attention: Each spatial location attends across all frames at that position.
- Full spacetime attention: Every patch attends to every other patch across space and time → O(T²×N²) → only tractable in latent space.
Training
- Datasets: WebVid-10M, InternVid, HD-VILA-100M, proprietary web-scraped video.
- Compute: Training frontier video models requires 1000s of GPUs for weeks.
- Progressive training: Start with low-res short videos → fine-tune on high-res long videos.
- Caption generation: Use VLMs to generate detailed descriptions for training videos.
Current Limitations
- Physics violations: Objects pass through each other, impossible transformations.
- Identity drift: Characters change appearance over long sequences.
- Hand/finger artifacts: Fine details still challenging.
- Cost: Generating a single minute of video can take minutes to hours on top hardware.
Text-to-video generation is the frontier that will transform media production, education, and entertainment — while current systems produce impressive short clips with occasional physics violations, the rapid improvement trajectory suggests that within a few years, AI-generated video will be indistinguishable from real footage for many applications, fundamentally changing how visual content is created and consumed.