NVIDIA unveils optimizations to its Transformer Engine that substantially speed up Mixture-of-Experts training in JAX, delivering about a 10x improvement in end-to-end throughput for the 671-billion-parameter DeepSeek-V3 MoE model.

The approach uses dropless MoE processing to ensure every token is processed despite load imbalances, preserving model quality by avoiding data loss while employing block-sparse operations and dynamically shaped kernels to handle uneven distributions.

The enhancements tackle MoE bottlenecks like token routing and inter-GPU communication, introducing specialized kernels such as grouped GEMM and expert-parallel operations to efficiently manage ragged tensors.

Broader significance: this work aligns with industry efforts to scale and optimize MoE training, complementing parallel research from Microsoft and the Allen Institute for AI on balancing performance and efficiency in large AI models.

The MoE training stack demonstrates strong scalability, maintaining about 97% efficiency on clusters of up to 1,024 GPUs, aided by XLA multistreaming collectives and a Latency Hiding Scheduler to minimize large-scale inefficiencies.

NVIDIA leverages NCCL EP for MoE-optimized communication, fusing token dispatch and accumulation into a single kernel to keep GPUs highly utilized and reduce idle time during data transfers.

Context and impact: MoE architectures enable trillions-of-parameter models without proportional cost increases, making them attractive for large-scale language and multimodal models, with NVIDIA positioning these optimizations as a key accelerator.

Beyond performance gains, NVIDIA highlights practical accessibility via the NVIDIA NGC MaxText container to reproduce results with DeepSeek-V3, and hints at future enhancements like NVFP4 quantization and further kernel fusions.