nemo-mbridge-perf-moe-long-context

This skill is an NVIDIA performance-tuning reference for training Mixture-of-Experts (MoE) models at long sequence lengths with Megatron Bridge. It solves the problem of fitting and efficiently running long-context MoE training, where attention memory and activation residency dominate and naive configurations cause out-of-memory errors or degraded throughput. It explains what changes past the 4K-class regime and which levers help: context parallelism, selective recompute, lower precision, CPU offload for optimizer state, and dispatcher and pipeline layouts that preserve the shrinking data-parallel budget.

The skill distills practical patterns from real long-context experiments on DSV3 (H100) and Qwen3 and Qwen3-Next (GB200), noting for each how throughput and memory behave as context grows and how selective recompute, CP sizing, pipeline layout, and grouped-GEMM improvements interact. It offers CP sizing rules of thumb (starting from a 4K shard target of roughly seq_len over 4096 rounded to a power-of-two layout, keeping DP alive, preferring selective recompute of modules like up_proj, norm, moe, and mlp, and avoiding SDPA-heavy recompute). It gives representative config families for 128K and 256K contexts including TP, CP, EP, PP, VPP, precision, dispatcher, and recompute choices, plus CUDA graph guidance and common pitfalls. Its bundled benchmark reports a 100% security score in NVSkills-Eval.

It targets ML performance engineers and researchers training large MoE models at long context. Typical use cases include planning parallelism for 128K-token training, choosing recompute strategy, and investigating a commit that caused long-context MoE OOM or throughput regressions. It is guidance and configuration knowledge rather than executable tooling.