How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.
The mcore-run-on-slurm skill is a reference guide for launching distributed Megatron-LM training jobs on a SLURM cluster. It solves the problem of correctly wiring up multi-node distributed training, an area where subtle mistakes in environment variables, launcher choice, or hardware-specific tuning flags cause jobs to fail, deadlock, or assert. It leads with answer-first constants so an agent can respond to text-only setup questions before reproducing a full script.
The guide provides a minimal, annotated sbatch skeleton with one srun task per node launching workers through uv run python -m torch.distributed.run (not bare torchrun), deriving MASTER_ADDR from scontrol, and computing NNODES, GPUS_PER_NODE, and WORLD_SIZE. It codifies the nuanced CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes: set to 1 on pre-Blackwell Hopper/Ampere with TP>1 or CP>1 in non-FSDP mode, unnecessary on Blackwell, must not be 1 under Torch-FSDP2 or Megatron-FSDP, and 32 when overlap_moe_expert_parallel_comm is enabled. It also covers multi-node filesystem rules (shared worktree visible to all nodes), container conventions, monitoring, and per-rank failure diagnosis.
Target users are ML engineers and researchers running Megatron-LM at scale on SLURM GPU clusters who need to write or debug sbatch scripts, configure torch.distributed, and diagnose job failures. Prerequisites include a SLURM login with GPU-partition submission rights, Megatron-LM checked out on a shared filesystem, and uv installed with the training/dev extras synced.
When submitting a SLURM job, writing or debugging an sbatch script, configuring multi-node distributed training, setting MASTER_ADDR/MASTER_PORT/WORLD_SIZE, or diagnosing a SLURM job failure for Megatron-LM.
It depends on hardware and parallelism mode. Pre-Blackwell Hopper/Ampere with TP>1 or CP>1 and non-FSDP requires 1 (the code path asserts otherwise); Blackwell does not need it; Torch-FSDP2 or Megatron-FSDP must not use 1; and overlap_moe_expert_parallel_comm uses 32. Do not export it unconditionally.
A SLURM cluster login with submission rights to a GPU partition, Megatron-LM checked out on a filesystem visible to all nodes (NFS, Lustre, or similar), and uv installed with uv sync run once for the training and dev (or lts) extras so the .venv is materialized for every node.
The skill prescribes one srun task per node launching workers via uv run python -m torch.distributed.run as a single worker group across all nodes, rather than starting independent single-node jobs, so ranks coordinate correctly.
The guide focuses on setup and diagnosis; the associated test tooling documents retry behavior, but for run-on-SLURM the emphasis is correct configuration, writing checkpoints and logs to shared storage, and per-rank failure diagnosis.
Quick Setup:
.claude/skills/Repository
nvidia/skills