Expert parallelism (EP) suffers from load imbalance, especially on fine-grained MoE models (many “small” experts, activated sparsely) and large EP degrees (e.g., EP64 on rack-scale nodes like NVL72). Such imbalance causes compute stragglers, token all-to-all bottlenecks, and OOMs on overloaded ranks. Auxiliary losses and routing bias balance the load only in a statistical sense in training, and at the granularity of a single microbatch the hot experts keep moving — which is where up to 2× of the achievable throughput quietly goes.

What UltraEP does. Rather than predicting from a stale history window as EPLB does, UltraEP reads the exact post-gating load and rebalances on the spot, inside every layer of every microbatch: hot experts are replicated onto idle ranks and tokens are rerouted to the new replicas. Our planner solves the post-reroute load and placement schemes on the GPU, and dedicated relay-based communication kernels move expert state within the scale-up domain. Together they keep the critical-path cost under 300 µs, simply 1–2% of step time.

What UltraEP buys. Across GLM4.5-106B, Qwen3-235B and DeepSeek-V3 at EP64, UltraEP reaches 94.3% of force-balanced ideal throughput on average — a 1.49× gain over state-of-the-art training and inference frameworks — and compresses inter-rank imbalance from 1.30–4.01 down to 1.01–1.04. It has carried the production training of a 288B-parameter MoE model, holding at or above 92% of ideal performance over the whole run.

Built to be adopted. UltraEP ships as a standalone Python/CUDA runtime, decoupled from token all-to-all libraries such as DeepEP, so integrating it into Megatron-LM or SGLang takes a few hundred lines. It preserves the mathematical equivalence of MoE computation, stays compatible with CUDA graphs, DP/PP/VPP, activation checkpointing and FP8, and does no dynamic memory allocation at runtime.

The idea is becoming standard. MoonEP, the expert-parallelism library behind Moonshot AI’s Kimi K3, follows the same route: redundant experts planned online from the current router output, layer by layer. Real-time, system-side balancing is turning into a basic capability of expert parallelism rather than an optimization. The natural next frontier is RL post-training, where expert load is as dynamic as inference prefill and algorithm-side regulation is absent — the setting ReLibra targets on top of UltraEP.