nemo-mbridge-perf-moe-comm-overlap

This NVIDIA skill is a focused tuning guide for Mixture-of-Experts (MoE) expert-parallel communication overlap in Megatron Bridge. It addresses a specific performance problem in large-scale distributed training: when expert parallelism is used and token dispatch/combine communication becomes a visible slice of step time, overlapping that communication with compute can recover throughput. The guide helps engineers decide when the optimization is worthwhile and how to enable it correctly without destabilizing an already-working run.

It documents the exact configuration flags (overlap_moe_expert_parallel_comm, optional delay_wgrad_compute, and disabling moe_shared_expert_overlap), the prerequisites (expert_model_parallel_size > 1, more than one expert, an alltoall or flex token dispatcher, BF16/FP16 precision, and VPP when pipeline parallelism is active), and the subtlety that setting a flex dispatcher backend alone does not activate flex dispatch. It explains interactions with recompute and CUDA graphs, lists code anchors in the Megatron Bridge source and tests, enumerates common pitfalls, and includes a measured but explicitly directional short-run benchmark on H100 hardware showing a throughput improvement when EP communication was exposed. A verification section describes how the overlap validation raises on unmet prerequisites so a clean startup confirms the feature is active.

It targets performance engineers and researchers tuning MoE training throughput or tracing a regression to a comm-overlap config change. The content is analytical and configuration-oriented, with a couple of read-only performance-harness commands, and its published card reports a passing evaluation with a 100% security dimension. It is a knowledge and configuration reference, not a system-modifying tool.