Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data. Covers correlation testing, available recipes, and multi-GPU examples.
This skill is an NVIDIA reference for running and comparing Megatron-LM (MLM) and Megatron Bridge training. It solves the practical problem of translating between the two training stacks and diagnosing why loss curves diverge, for example after a code change. It provides a first-answer checklist naming the canonical Bridge recipe, entry points for both MLM and Bridge, the shared torch.distributed launch wrapper, and required environment setup such as adding Megatron-LM to PYTHONPATH and cleaning nemo_experiments before a fresh Bridge run.
The core of the skill is loss-correlation testing: it gives paired MLM and Bridge command-line invocations for a small GPT configuration on a single GPU, using mock data and BF16, so that matched-parameter runs should produce nearly identical LM losses to within BF16 rounding. It extends this to multi-GPU examples with tensor parallelism and sequence parallelism, and it lists available recipes such as the minimal vanilla GPT pretrain config and larger Llama configurations. A companion doc maps MLM CLI arguments to Bridge configuration keys and documents gotchas and a translation script. The skill notes that files under the vendored Megatron-LM directory should not be modified. Its bundled benchmark and skill card report NVSkills-Eval results including a 100% security score, no required credentials, and guidance to avoid secrets in prompts and logs.
It targets ML engineers and researchers running large-model pretraining who need to reproduce, compare, or debug training between MLM and Megatron Bridge. Typical use cases include verifying loss correlation, translating MLM arguments to Bridge config, and investigating post-commit divergence.
Running Megatron-LM and Megatron Bridge training with mock or real data, comparing their loss curves, translating MLM CLI arguments to Bridge config, and diagnosing why loss curves diverged after a commit.
Run the paired single-GPU correlation commands with matched parameters using the vanilla_gpt_pretrain_config recipe and mock data; the LM losses should agree at each iteration to within BF16 rounding.
No. The skill card reports no required API keys or external credentials, and advises not including secrets in prompts, logs, or output and using least-privilege credentials.
MLM runs need Megatron-LM on the PYTHONPATH, Bridge runs should clear the nemo_experiments directory first for a fresh run, and vendored files under the Megatron-LM directory should not be modified.
Yes, it includes 2-GPU examples with tensor-model-parallel-size 2 and sequence parallelism for both MLM and Bridge.
Quick Setup:
.claude/skills/Repository
nvidia/skills