nemo-mbridge-multi-node-slurm

Multi-Node Slurm is an NVIDIA instruction skill for converting single-node training scripts into multi-node Slurm sbatch jobs for Megatron Bridge (NeMo), and for debugging the common failures that appear at multi-node scale. It solves the practical pain of getting distributed training launches right: how to structure sbatch headers, spawn processes, mount containers, and diagnose NCCL and launch errors.

The skill lays out two launch approaches with a clear recommendation. The preferred srun-native shape uses `--ntasks-per-node=8` and a direct `srun ... uv run python <script>` launch, relying on Megatron Bridge to auto-derive RANK, WORLD_SIZE, LOCAL_RANK, MASTER_ADDR, and MASTER_PORT from SLURM environment variables during initialize.py. The legacy `uv run torch.distributed` approach keeps `--ntasks-per-node=1` and lets torchrun spawn per-node processes, with explicit warnings about EADDRINUSE port collisions if misconfigured. It covers Enroot container setup, a two-phase srun pattern (warm the uv cache once, then launch on all ranks), shared-filesystem requirements for repo/data/logs/HF_HOME/UV_CACHE_DIR/NEMO_HOME, interactive salloc+srun allocation, and a first-log NCCL-timeout triage checklist (grep for real errors, find the first failed rank, check ncclUniqueId/timeout). It also gives strong security-conscious guidance: keep credentials like HF_TOKEN, GH_TOKEN, and WANDB_API_KEY out of sbatch templates and logs, provide them through the scheduler environment or a restricted secrets file, and never hardcode token values.

Target users are ML engineers and researchers scaling Megatron Bridge training to multiple nodes on Slurm clusters, or investigating a commit that caused a multi-node failure. It is documentation-and-template guidance; any commands shown are standard cluster training operations.