Pooling by Multihead Attention

PMA (Pooling by Multihead Attention) is an attention-based aggregation mechanism that pools a set of features into a fixed number of output vectors — using learnable "seed" vectors as queries that attend to all set elements, replacing simple mean/max pooling with learned aggregation.

How Does PMA Work?

• Seed Vectors: $S \in \mathbb{R}^{k \times d}$ — $k$ learnable query vectors.
• Attention: $\text{PMA}_k(X) = \text{MAB}(S, X)$ — seeds attend to all input elements via multi-head attention.
• Output: $k$ output vectors, each a learned weighted combination of all input elements.
• $k = 1$: Produces a single set-level representation (like a learned global pooling).

Why It Matters

• Learned Pooling: More expressive than mean/max pooling — different seed vectors can capture different aspects of the set.
• Multiple Outputs: Can produce $k > 1$ outputs for tasks requiring multiple set-level predictions.
• Flexible: Differentiable and end-to-end trainable as part of any set-processing pipeline.

PMA is learned pooling via attention — using trainable query vectors to extract $k$ informative summaries from a variable-size set.