Semantic Segmentation Deep Learning

Semantic Segmentation is the dense prediction task that assigns a class label to every pixel in an image — requiring the network to simultaneously understand global scene context (what objects are present) and precise local boundaries (exactly where each object begins and ends), producing a pixel-wise classification map matching the input resolution.

Foundational Architectures:
- Fully Convolutional Network (FCN): replaced final FC layers with 1×1 convolutions and added transposed convolutions for upsampling — first end-to-end trainable segmentation network; skip connections from earlier layers recover spatial detail lost during downsampling
- U-Net: symmetric encoder-decoder with dense skip connections at every resolution level — encoder captures context through downsampling, decoder reconstructs spatial detail through upsampling; skip connections concatenate encoder features to decoder for precise boundary recovery; dominant in medical image segmentation
- DeepLab v3+: atrous (dilated) convolutions with Atrous Spatial Pyramid Pooling (ASPP) — dilated convolutions expand receptive field without reducing spatial resolution; ASPP applies parallel dilated convolutions at multiple rates to capture multi-scale context
- PSPNet: Pyramid Pooling Module aggregates features at multiple scales — global average pooling, 6×6, 3×3, and 1×1 pooling branches capture sub-region patterns at different granularities; outperforms FCN on complex scenes

Modern Approaches:
- Transformer-Based (SegFormer, Mask2Former): self-attention captures global context from the first layer — hierarchical Transformer encoders produce multi-scale features; MLP decoders aggregate features from all scales for segmentation prediction
- Panoptic Segmentation: unified framework combining semantic segmentation (stuff: sky, road) with instance segmentation (things: cars, people) — Panoptic FPN, MaskFormer, and Mask2Former produce both class-labeled regions and individual instance masks
- Real-Time Segmentation: BiSeNet, DDRNet use dual-branch architectures — one branch for spatial detail (high resolution, shallow), one for semantic context (low resolution, deep); achieves >30 FPS at reasonable accuracy for autonomous driving
- Semi-Supervised Methods: leverage unlabeled images with pseudo-labels generated by teacher model — FixMatch-based approaches achieve near-supervised accuracy using only 1-10% labeled data

Training and Evaluation:
- Loss Functions: cross-entropy per pixel is baseline — Dice loss handles class imbalance by optimizing the overlap ratio directly; Lovász loss is a smooth surrogate for IoU optimization; combination losses (CE + Dice) common in practice
- Mean IoU (mIoU): primary evaluation metric — IoU per class = intersection/union of predicted and ground-truth pixels; averaged across all classes; IoU penalizes both false positives and false negatives equally
- Data Augmentation: random scaling (0.5-2.0×), random crop, horizontal flip, color jitter — multi-scale training helps the model generalize across object sizes
- Class Imbalance: outdoor scenes dominated by sky, road, vegetation — class-weighted loss, oversampling rare classes, or Online Hard Example Mining (OHEM) focuses training on under-represented or difficult pixels

Semantic segmentation is the visual understanding task that enables pixel-level scene parsing — powering autonomous driving perception, medical image analysis, satellite imagery interpretation, and augmented reality by providing the dense spatial understanding that detection and classification alone cannot achieve.