nemo-mbridge-perf-moe-vlm-training

MoE VLM Training is an NVIDIA instruction skill offering practical guidance for training Mixture-of-Experts vision-language models (VLMs) in Megatron Bridge. It solves the problem of choosing between FSDP and 3D-parallel training strategies for multimodal MoE models and of avoiding common pitfalls, drawing on rounded lessons from Qwen3-VL, Qwen3-Next, and other multimodal experiments. It is also framed for investigating commits that caused MoE VLM training failures or OOM.

The skill compares FSDP (simplest path to a working multimodal run, best for first bring-up and memory-first tuning) against 3D parallel (higher ceiling after tuning, best for stable models with a clean pipeline layout), and recommends a workflow of getting a first reliable run with FSDP, stabilizing the real-data path, and moving to 3D parallel only when the throughput headroom is worth it. It summarizes representative findings (FSDP on GB200-class systems reaching healthy utilization; mock image-free data being an untrustworthy performance proxy versus real multimodal input), a decision guide for each strategy, key tuning knobs (freezing the vision stack, sweeping micro-batch size, preferring selective recompute once the model fits, matching CUDA-graph scope, using expert tensor parallelism only when needed), representative config families for GB200 paths, a compatibility table, and pitfalls such as the vision encoder unexpectedly dominating.

The target users are ML researchers and performance engineers training or debugging multimodal MoE models on NVIDIA GB200/B200-class systems. This is conceptual and configuration guidance — a knowledge reference with no scripts, no destructive operations, and no network or credential handling.