Xlnet

XLNet uses permutation language modeling to capture bidirectional context while maintaining autoregressive pre-training benefits. Problem addressed: BERT uses artificial MASK tokens not present at fine-tuning (pre-train/fine-tune discrepancy). Autoregressive models miss bidirectional context. Solution: Train on all permutations of token orderings. Each token sees different random subsets of other tokens as context. Permutation LM: For sequence [1,2,3,4], might use order [3,1,4,2], so position 2 sees positions 3,1,4 as context. Two-stream attention: Target-aware representations that know position but not content of token being predicted. Segment recurrence: Carry hidden states across segments for longer context, inspired by Transformer-XL. Results: Outperformed BERT on 20 benchmarks when released. Strong performance across tasks. Complexity: More complex than BERT, harder to implement and train. Current status: Influential but largely superseded by simpler approaches that scale better. Showed creative alternatives to MLM were possible.