Influence Functions

Influence Functions are a technique from robust statistics applied to ML that measures how each training example affects a model's prediction — quantifying the change in a test prediction if a specific training point were upweighted or removed, enabling data attribution and debugging.

How Influence Functions Work

• Question: How would the model's prediction on test point $z_{test}$ change if training point $z_i$ were removed?
• Approximation: $\mathcal{I}(z_i, z_{test}) = -\nabla_\theta L(z_{test})^T H_{\theta}^{-1} \nabla_\theta L(z_i)$ where $H$ is the Hessian.
• Hessian Inverse: Computed approximately using conjugate gradients or stochastic estimation.
• Attribution: Rank training points by their influence on the test prediction.

Why It Matters

• Data Debugging: Identify mislabeled, corrupted, or anomalous training examples that hurt predictions.
• Data Valuation: Quantify the value or harm of each training data point.
• Model Debugging: Understand why a model makes a specific prediction by tracing it to influential training data.

Influence Functions are tracing predictions to training data — measuring which training examples are most responsible for a model's behavior.