Stochastic Depth in ViT

Stochastic Depth is the layer-wise dropout that randomly skips transformer blocks during training so very deep Vision Transformers do not overfit or suffer exploding gradients — each block is bypassed with probability p, turning a 100-layer network into a mixture of shallower networks while still evaluating every block at inference time.

What Is Stochastic Depth?

• Definition: A regularization technique where entire residual blocks are dropped (replaced with identity mappings) independently per sample during training.
• Key Feature 1: Drop probability often increases linearly from the shallowest to the deepest block, ensuring deeper layers are more likely to be skipped.
• Key Feature 2: The outputs of surviving blocks are scaled by 1/(1-p) so that the expected activations remain stable.
• Key Feature 3: It effectively enforces an ensemble of networks with different depths, which improves generalization.
• Key Feature 4: Works with ViT because each transformer block naturally lends itself to identity skip connections.

Why Stochastic Depth Matters

• Trainability: Deep ViTs benefit from skip patterns that reduce gradient path length during early training.
• Robustness: The randomness prevents reliance on any single block, increasing resilience to ablations.
• Efficiency: With dropout masks, the average layer count per sample decreases, slightly reducing compute.
• Ensembling Effect: The model behaves like an ensemble of networks with varying depths, improving accuracy.
• Confidence Calibration: Predictions are less overconfident because the path depth varies.

Drop Schedules

Linear Schedule:
- p increases linearly from zero at the first layer to a target near 0.2-0.3 at the final layer.
- Encourages early layers to remain stable while deeper layers oscillate.

Uniform Schedule:
- All layers share the same drop probability for simplicity.
- Useful to test sensitivity.

Head-Wise:
- Apply stochastic depth separately per attention head group for more granular randomization.

How It Works / Technical Details

Step 1: Sample Bernoulli masks for each training sample and each block. Multiply the block output by mask / (1 - p), and add it to the identity path.

Step 2: At inference, masks are disabled so all blocks execute, giving the full depth while benefiting from the regularized representations learned during training.

Comparison / Alternatives

Tools & Platforms

• timm: Allows `drop_path_rate` to control stochastic depth in ViT blocks.
• Deep Learning Frameworks: PyTorch’s `DropPath` modules are widely available.
• Hydra Configs: Vary drop path rates to find best generalization/accuracy trade-offs.
• Profiling: Track actual block usage to verify the expected depth distribution.

Stochastic depth is the depth regularizer that keeps Vision Transformers stable and generalizable even when they grow to 100+ layers — it trains as a committee of subnetworks yet retains the full model at inference.