Model Extraction Attack

Model extraction attack (also called model stealing) is a security attack where an adversary aims to recreate a proprietary ML model by systematically querying it and using the input-output pairs to train a substitute model that closely mimics the original. This threatens the intellectual property and competitive advantage of model owners.

How Model Extraction Works

• Step 1 — Query Selection: The attacker crafts a set of inputs to query the target model. These can be random, from a relevant domain, or strategically chosen using active learning techniques.
• Step 2 — Response Collection: The attacker collects the model's outputs — which may include predicted labels, probability distributions, confidence scores, or generated text.
• Step 3 — Surrogate Training: Using the collected (input, output) pairs as training data, the attacker trains a substitute model that approximates the target's behavior.
• Step 4 — Refinement: The attacker iteratively queries the target to improve the surrogate, focusing on regions where the two models disagree.

What Gets Extracted

• Decision Boundaries: The surrogate learns to make similar predictions on similar inputs.
• Architectural Insights: Query patterns and response analysis can reveal information about model architecture, training data distribution, and feature importance.
• Downstream Attacks: A good surrogate enables transfer attacks — adversarial examples crafted against the surrogate often fool the original model too.

Defenses

• Rate Limiting: Restrict the number of queries a user can make.
• Output Perturbation: Add noise to confidence scores or round probabilities to reduce information leakage.
• Watermarking: Embed detectable patterns in the model's behavior that survive extraction, enabling ownership verification.
• Query Detection: Monitor for suspicious query patterns indicative of extraction attempts.
• API Design: Return only top-k labels instead of full probability distributions.

Why It Matters

Model extraction threatens the business model of ML-as-a-Service providers. A stolen model can be deployed without paying API fees, used to find vulnerabilities, or reverse-engineered to infer training data characteristics.