Set Transformer

Set Transformer is a transformer architecture designed for set-structured inputs (unordered collections) — using attention-based mechanisms to process variable-size sets while maintaining permutation invariance, the key symmetry property of set functions.

How Does Set Transformer Work?

• SAB (Set Attention Block): Standard multi-head self-attention applied to set elements.
• ISAB (Induced Set Attention Block): Uses $m$ inducing points to reduce $O(N^2)$ to $O(N \cdot m)$ complexity.
• PMA (Pooling by Multihead Attention): Aggregates set elements into $k$ output vectors using learned seed vectors.
• Paper: Lee et al. (2019).

Why It Matters

• Permutation Invariance: The output is the same regardless of the order of input elements — essential for set functions.
• Efficient: ISAB enables processing large sets (thousands of elements) efficiently.
• Applications: Point cloud processing, amortized inference, few-shot learning, set prediction.

Set Transformer is attention for unordered collections — processing variable-size sets with permutation invariance and efficient inducing-point attention.