nemo-automodel-distributed-training

NeMo AutoModel Distributed Training is an NVIDIA-authored guidance skill for selecting and configuring distributed training strategies in NeMo AutoModel, which uses PyTorch-native distributed training orchestrated through a single MeshContext object holding device meshes, strategy configs, and axis names. It solves the practical difficulty of choosing among FSDP2, Megatron FSDP, and DDP and correctly setting parallelism sizes, so engineers can answer configuration questions and debug multi-GPU or multi-node failures without spelunking through the codebase.

The skill provides a strategy-selection decision tree and compact answer patterns. It recommends FSDP2 as the general-purpose default supporting tensor, pipeline, context, expert, and hybrid-sharded parallelism; positions Megatron FSDP as a Megatron-style option that does not support pipeline parallelism, expert parallelism, or sequence parallel; and describes DDP as simple data parallelism only. It documents the YAML distributed section fields (tp_size, pp_size, cp_size, ep_size, pipeline and moe sub-configs), key sizing constraints such as dp_size being inferred from world_size divided by the product of tp, pp, and cp sizes, and rules like ep_size needing to divide dp_size times cp_size. It maps configuration to internal parsing functions and manager classes, and gives worked recommendations for large multi-node models, mixture-of-experts expert parallelism, and Megatron FSDP limitations. A bundled benchmark report documents evaluation across security, correctness, discoverability, effectiveness, and efficiency dimensions.

The intended users are ML engineers and researchers adding or modifying distributed strategies, debugging distributed failures, or tuning sharding and parallelism in NeMo AutoModel. It is Apache-2.0 licensed reference guidance, focused on producing correct YAML and constraints rather than executing training itself.