Activation Maximization

Activation Maximization is the optimization-based approach to generating inputs that maximally activate a target neuron or output class in a neural network — using gradient ascent in input space to find (or synthesize) the input pattern that a neuron responds most strongly to.

Activation Maximization Process

• Target: Choose a neuron, channel, layer, or output class to maximize.
• Initialize: Start with noise, a fixed image, or a learned prior (generator network).
• Gradient Ascent: Compute $\nabla_x a_{target}(x)$ and update the input: $x leftarrow x + eta \nabla_x a_{target}$.
• Regularization: Apply image priors (total variation, frequency penalization, learned priors) to produce natural-looking results.

Why It Matters

• Neuron Identity: Reveals the "ideal stimulus" for each neuron — what it has learned to represent.
• Class Visualization: Generate the "ideal" input for each output class — the network's prototype of each category.
• GAN Priors: Using a GAN generator as the parameterization produces photorealistic activation maximization.

Activation Maximization is finding the neuron's favorite input — the optimization-based core technique behind feature visualization and neural network understanding.