Shared Expert in MoE

Shared expert in MoE is the always-active expert path that processes every token alongside routed sparse experts - it provides a stable general-purpose representation channel while specialist experts handle token-specific patterns.

What Is Shared expert in MoE?

• Definition: A designated expert that receives all tokens regardless of router top-k assignment.
• Architectural Role: Acts as a dense backbone inside an MoE block to preserve common language and reasoning signals.
• Routing Interaction: Shared output is combined with routed expert outputs before residual integration.
• Deployment Pattern: Common in large MoE systems where pure sparse routing can be unstable early in training.

Why Shared expert in MoE Matters

• Training Stability: Guarantees each token has a reliable processing path even when routing is noisy.
• Knowledge Retention: Captures broad capabilities such as syntax and generic semantics that should not depend on expert selection.
• Drop Mitigation: Reduces quality loss when capacity limits or imbalance cause routed-token pressure.
• Convergence Support: Helps routers specialize gradually without catastrophic dependence on early routing decisions.
• Production Robustness: Improves consistency under variable loads and imperfect expert utilization.

How It Is Used in Practice

• Block Design: Add one shared FFN expert per MoE layer and merge it with sparse expert outputs.
• Weight Tuning: Calibrate mixing coefficients so shared and routed paths contribute appropriately.
• Monitoring: Track whether shared path over-dominates, which can hide needed expert specialization.

Shared expert in MoE is a practical stability anchor for sparse transformer architectures - it balances specialist routing with dependable general-purpose computation.