Global Context Block

Global Context (GC) Block is a simplified and efficient version of the Non-Local block — observing that Non-Local attention maps are nearly identical for different query positions, and replacing the per-query computation with a single global context vector shared across all positions.

How Does the GC Block Work?

• Global Context: $c = \sum_j \frac{\exp(W_k x_j)}{\sum_m \exp(W_k x_m)} \cdot x_j$ (attention-weighted global average).
• Transform: $c' = \text{LayerNorm}(W_2 \cdot \text{ReLU}(W_1 \cdot c))$ (bottleneck transform like SE).
• Broadcast: Add $c'$ to every spatial position: $y_i = x_i + c'$.
• Paper: Cao et al. (2019).

Why It Matters

• Efficiency: One global context vector vs. N×N attention matrix -> dramatically cheaper than Non-Local.
• Same Quality: Achieves similar or better results than Non-Local blocks at a fraction of the cost.
• Insight: Revealed that query-independent attention is sufficient — you don't need per-pixel attention.

GC Block is Non-Local attention simplified — the insight that one shared global context works as well as expensive per-position attention.