Token Dispatch

Token dispatch is the routing-stage data movement that groups and sends token representations to their assigned experts - it transforms router decisions into contiguous expert-ready batches for efficient sparse computation.

What Is Token dispatch?

• Definition: Permutation and transfer process that maps tokens from source order to destination-expert order.
• Core Steps: Build routing indices, pack token buffers by expert, and transmit shards to expert-owner ranks.
• Memory Objective: Create contiguous blocks so expert GEMM kernels run with high efficiency.
• Execution Layer: Implemented with fused permutation kernels, communication collectives, and metadata tables.

Why Token dispatch Matters

• Step-Time Share: Dispatch can consume large runtime when token counts or expert groups are large.
• Bandwidth Use: Packing quality affects payload efficiency and network overhead.
• Compute Readiness: Poor dispatch layouts degrade expert kernel throughput.
• Scalability: Dispatch bottlenecks limit gains from adding more experts or devices.
• Stability: Deterministic dispatch logic is required for correct token-to-output mapping.

How It Is Used in Practice

• Kernel Optimization: Use high-throughput pack and permutation kernels to reduce staging overhead.
• Metadata Design: Maintain compact index structures for fast combine reversal.
• End-to-End Profiling: Separate dispatch latency from expert compute to target the right bottleneck.

Token dispatch is a foundational MoE runtime primitive - efficient token packing and transfer directly determine sparse execution performance.