Mixture of Depths

Mixture-of-Depths (MoD) is a transformer efficiency technique built on a simple observation: a standard transformer spends exactly the same amount of computation on every token, whether that token is a throwaway "the" or a pivotal technical term that the whole prediction hinges on. MoD breaks that uniformity by letting the model *choose*, at every layer, which tokens are worth the full cost of that layer's attention and feed-forward computation and which can simply skip it and ride the residual connection through unchanged. It is conditional computation along the *depth* axis of the network — hence the name — and it is the depth-wise cousin of Mixture-of-Experts, which does the same trick along the *width* axis.

A dense transformer wastes compute by treating every token identically. Every position flows through every block, paying the identical FLOP cost for self-attention and the MLP, regardless of how much processing that position actually needs. But language is not uniform: some tokens are trivially predictable from local context and some require deep, many-layer reasoning. Spending a fixed, maximal budget on all of them means the easy tokens are massively over-served while the compute that could have gone to hard tokens is spread thin. MoD is the attempt to reallocate that fixed budget toward the tokens that need it.

MoD puts a router before each block that admits only the top-k tokens; the rest take the residual shortcut. At every MoD layer a small learned router scores each token, and only the highest-scoring fraction — a fixed *capacity*, say 12.5% or 50% of the sequence — is passed through the block's attention and MLP. The non-selected tokens bypass the block entirely via the identity residual, arriving at the next layer unchanged. The crucial engineering choice is that the capacity is *static*: exactly k tokens are processed per block, known ahead of time, so the compute graph has a fixed shape and batches efficiently on a GPU or TPU. This is what separates MoD from classic *early-exit* / adaptive-depth schemes, where each token dynamically decides when to stop — flexible in theory but a nightmare to batch because different sequences finish at different layers.

MoD, Mixture-of-Experts, and early exit are three different answers to "where do we save compute?" Mixture-of-Experts routes each token to a few of many parallel expert MLPs — it saves compute along *width*, activating only a sparse slice of a very large parameter count while keeping full depth. MoD routes along *depth*, keeping the same parameters but letting most tokens skip most layers, cutting FLOPs per token. Early exit varies depth *dynamically* per token, which maximizes flexibility but sacrifices the static, hardware-friendly shape MoD preserves. Because MoE and MoD save on orthogonal axes, they compose: a "MoDE" block can route across experts *and* across depth at once, stacking both savings.

The unhelpful way to think about Mixture-of-Depths is as yet another niche efficiency hack layered onto the transformer. The useful way is to see it as answering a question the dense architecture never asks: not *how* to process a token but *whether* this token, at this layer, is worth processing at all. By giving every block a router with a fixed capacity, MoD reallocates a constant compute budget toward the tokens that need deep processing and lets the easy ones coast on the residual — capturing much of the benefit of dynamic early-exit while keeping the static, batch-friendly compute shape that hardware demands. Set beside Mixture-of-Experts, which sparsifies the network's *width*, MoD sparsifies its *depth*, and the two combine cleanly. Read Mixture-of-Depths through a spend-compute-only-where-it's-needed lens rather than an every-token-deserves-every-layer lens, and the router, the fixed capacity, and the residual shortcut stop looking like tricks and become the natural machinery for buying back the compute a uniform transformer throws away.