Nanogpt

nanoGPT is a minimal, readable implementation of GPT-2/GPT-3 training and inference created by Andrej Karpathy in two files of clean PyTorch code — designed to be the simplest possible codebase that can reproduce GPT-2 (124M parameters) training on a single GPU, enabling thousands of engineers to understand transformer language models by stepping through the training loop line-by-line in a debugger rather than navigating Hugging Face's deep abstraction layers.

What Is nanoGPT?

• Definition: A ~300-line PyTorch implementation of the GPT architecture (decoder-only transformer with causal attention) that includes both training (`train.py`) and inference (`sample.py`) — the entire GPT-2 architecture, training loop, and text generation in two readable files.
• Creator: Andrej Karpathy — created nanoGPT as part of his mission to make deep learning fundamentally understandable, following micrograd (autograd in 100 lines) with a complete language model in 300 lines.
• Reproducible GPT-2: nanoGPT can reproduce OpenAI's GPT-2 (124M) training results on OpenWebText — training on a single A100 GPU in ~4 days, achieving comparable perplexity to the original model.
• Simplicity Over Generality: Unlike Hugging Face Transformers (which handles 100+ architectures with deep abstraction trees), nanoGPT implements exactly one architecture (GPT) with zero abstraction — every line of code maps directly to a concept in the "Attention Is All You Need" paper.

What nanoGPT Teaches

• Transformer Architecture: The complete GPT block — multi-head causal self-attention, layer normalization, feed-forward network with GELU activation, residual connections — all visible in a single `Block` class.
• Training Loop: Data loading, forward pass, loss computation (cross-entropy), backward pass, gradient clipping, optimizer step (AdamW), learning rate scheduling — the complete training recipe in one function.
• Text Generation: Autoregressive sampling with temperature, top-k — the `generate()` method shows exactly how language models produce text token by token.
• Scaling: The same code trains a 124M GPT-2 or a larger model by changing config parameters — demonstrating that model scaling is just changing dimensions, not changing architecture.

nanoGPT vs Alternatives

nanoGPT is the repository that taught a generation of engineers how transformer language models actually work — by implementing GPT-2 training and inference in 300 lines of transparent PyTorch code, Karpathy created the definitive educational resource that makes the architecture behind ChatGPT, Claude, and every modern LLM fundamentally understandable.