Feature Pyramid Network
Feature Pyramid Network (FPN) is the multi-scale feature extraction architecture that builds a top-down pathway with lateral connections to create feature maps at multiple resolutions — combining the high-resolution, low-semantic features from early layers with the low-resolution, high-semantic features from deep layers, enabling strong performance on scale-variant tasks like object detection and instance segmentation where objects of vastly different sizes must be detected simultaneously.
The Scale Problem
- Small objects: Need high-resolution feature maps (early layers) → but these lack semantic meaning.
- Large objects: Need semantically rich feature maps (deep layers) → but these are low resolution.
- Single-scale detection: Either misses small objects or lacks context for large objects.
- FPN: Creates a pyramid of features where EVERY level has strong semantics AND appropriate resolution.
FPN Architecture
1. Bottom-Up Pathway: Standard backbone (ResNet) produces feature maps at decreasing resolutions.
- C2: 1/4 resolution, C3: 1/8, C4: 1/16, C5: 1/32.
2. Top-Down Pathway: Upsample deep features (2x nearest neighbor) and add via lateral connections.
- P5 = 1×1 conv(C5)
- P4 = Upsample(P5) + 1×1 conv(C4)
- P3 = Upsample(P4) + 1×1 conv(C3)
- P2 = Upsample(P3) + 1×1 conv(C2)
3. Output: Apply 3×3 conv to each merged level → {P2, P3, P4, P5} — all with 256 channels.
Lateral Connections
- 1×1 convolution: Reduces channel dimension of bottom-up feature to match top-down (256).
- Element-wise addition: Merges semantic info (top-down) with spatial info (bottom-up).
- 3×3 convolution: Smooths artifacts from upsampling + addition.
FPN in Object Detection
| Detector | How FPN Is Used |
|---|---|
| Faster R-CNN + FPN | RPN proposals assigned to pyramid levels based on object size |
| RetinaNet | Dense anchors on each FPN level → focal loss |
| Mask R-CNN | FPN features for both detection and mask prediction |
| FCOS | Anchor-free detection with FPN level assignment |
| DETR | Encoder operates on multi-scale FPN features |
Level Assignment for Detection
$k = \lfloor k_0 + \log_2(\sqrt{wh}/224) \rfloor$
- k₀ = 4 (default). Object of size 224×224 → assigned to P4.
- Larger objects → higher pyramid level (P5, P6).
- Smaller objects → lower pyramid level (P2, P3).
FPN Variants
| Variant | Modification | Improvement |
|---|---|---|
| PANet (2018) | Add bottom-up path after FPN | Better localization |
| BiFPN (EfficientDet) | Bidirectional with learned weights | Better feature fusion |
| NAS-FPN | Architecture search for FPN topology | Task-optimized structure |
| PAFPN (YOLO) | PANet-style FPN in YOLO detectors | Balanced features |
Feature Pyramid Networks are the standard multi-scale architecture in computer vision — their elegant combination of top-down and bottom-up information flow creates semantically rich features at all resolutions, directly enabling the detection of objects ranging from tiny faces to large vehicles within the same image.