Autograd

Autograd (Automatic Differentiation) is the engine behind modern deep learning frameworks, enabling automatic computation of gradients for arbitrary computation graphs, which is essential for backpropagation. Principle: chain rule of calculus applied recursively. Forward pass: execute operations and build dynamic (PyTorch) or static (TensorFlow 1.x) computation graph; store intermediate tensors needed for backward. Backward pass: traverse graph in reverse; compute gradients of loss with respect to inputs using stored values and operation derivatives. Dynamic vs Static: Dynamic (Define-by-Run) builds graph during execution (flexible, easier debug); Static (Define-and-Run) builds graph first (optimization potential). Vector-Jacobian Product (VJP): core operation; efficient for scalar loss. Memory cost: must store activations from forward pass; creates memory trade-off (checkpointing exchanges compute for memory). Higher-order gradients: autograd can differentiate the derivative itself (Hessian). Custom functions: users can define custom autograd.Function by specifying forward and backward logic. Autograd democratized deep learning by removing the need to manually derive backpropagation formulas.