Micrograd

micrograd is a tiny autograd engine created by Andrej Karpathy that implements backpropagation and a dynamic computation graph in under 100 lines of Python — demonstrating that the core mechanism behind PyTorch, TensorFlow, and all modern deep learning frameworks (automatic differentiation via reverse-mode accumulation on a directed acyclic graph) can be understood by reading a single file, making it the most influential educational resource for demystifying how neural networks actually learn.

What Is micrograd?

• Definition: A minimal automatic differentiation engine that implements scalar-valued backpropagation — each `Value` object tracks its data, gradient, the operation that created it, and its parent nodes, forming a computation graph that `backward()` traverses in reverse topological order to compute gradients via the chain rule.
• Creator: Andrej Karpathy — former Director of AI at Tesla, founding member of OpenAI, and Stanford CS231n instructor. micrograd accompanies his legendary "Neural Networks: Zero to Hero" YouTube lecture series.
• Educational Purpose: micrograd exists to teach, not to compete — it proves that PyTorch is "not magic" by showing that the entire autograd mechanism (the engine that computes gradients for training neural networks) fits in 100 lines of readable Python.
• Scalar Operations: Unlike PyTorch (which operates on tensors/matrices), micrograd operates on individual scalar values — making every gradient computation explicit and traceable at the single-number level.

Core Implementation

The entire engine is built around a `Value` class:
- data: The scalar value (a single float).
- grad: The gradient of the loss with respect to this value (accumulated during backward pass).
- _backward: A closure that computes the local gradient contribution.
- _prev: Set of parent Value nodes in the computation graph.
- backward(): Topological sort of the graph, then call `_backward()` on each node in reverse order — this is backpropagation.

Supported Operations: Addition, multiplication, power, ReLU, negation, subtraction, division — enough to build multi-layer perceptrons and train them with gradient descent.

Why micrograd Matters

• Demystifies Deep Learning: Reading micrograd's 100 lines teaches you that neural network training is just: (1) build a math expression graph, (2) compute the output (forward pass), (3) walk the graph backward computing derivatives (backward pass), (4) nudge each parameter in the direction that reduces the loss.
• "Software 2.0" Foundation: Karpathy uses micrograd to teach that neural networks are mathematical expressions optimized via gradient descent — the foundation of his "Software 2.0" thesis that neural networks are a new programming paradigm.
• Gateway to PyTorch: After understanding micrograd, PyTorch's `autograd` module becomes transparent — it's the same algorithm operating on tensors instead of scalars, with GPU acceleration and thousands of optimized operations.
• Millions of Learners: The accompanying YouTube video has millions of views — micrograd has taught more people how backpropagation works than any textbook.

micrograd is the 100-line Python program that demystified deep learning for millions of developers — proving that the autograd engine at the heart of every modern ML framework is simply reverse-mode differentiation on a computation graph, making neural network training conceptually accessible to anyone who can read basic Python.