Token Combine
Token combine is the post-expert reconstruction stage that returns routed outputs to original token order and merges multi-expert contributions - it closes the MoE routing loop and restores sequence-consistent activations.
What Is Token combine?
- Definition: Inverse mapping process that gathers expert outputs and places them back at source token indices.
- Combination Logic: Applies router weights to blend outputs when top-k routing uses multiple experts.
- Data Dependency: Relies on dispatch metadata to ensure exact correspondence between tokens and expert results.
- Runtime Position: Executed after expert compute and before downstream transformer operations.
Why Token combine Matters
- Correctness: Any index or weighting error corrupts token representations and model quality.
- Latency Contribution: Combine can become a hidden bottleneck in large expert-parallel groups.
- Memory Traffic: Inefficient scatter and gather patterns increase HBM and network overhead.
- Numerical Integrity: Weighted merge precision influences stability in mixed-precision training.
- Pipeline Balance: Fast combine is required so expert compute gains are not canceled downstream.
How It Is Used in Practice
- Inverse Indexing: Store compact permutation maps during dispatch for exact reconstruction.
- Fused Operations: Merge gather and weighting steps to reduce extra memory passes.
- Validation Suite: Test token-order restoration and top-k weighting parity against reference implementation.
Token combine is a critical correctness and performance stage in MoE execution - robust reconstruction logic ensures sparse routing produces usable and efficient transformer activations.