Bidirectional Attention

Bidirectional attention allows each token to attend to all other tokens in the sequence, capturing full context. How it works: No masking of attention (except padding), every position can see every other position. Full context available at each position. Used in: BERT, RoBERTa, encoder-only models, encoder portion of encoder-decoder models. Advantage: Richer representations since both left and right context inform each token. Better for understanding tasks. Limitation: Cannot be used for generation directly since it requires seeing tokens that dont exist yet. MLM training: Masked Language Modeling works because model sees context around masked token. Would be trivial with causal masking. Applications: Text classification, NER, question answering (extractive), sentence embeddings, semantic similarity. Comparison to causal: Bidirectional is more powerful for understanding but unsuitable for generation. Hybrid approaches: Encoder uses bidirectional, decoder uses causal (T5, BART). XLNet uses permutation-based bidirectional context.