Operational guide for choosing and combining parallelism strategies in Megatron Bridge, including sizing rules, hardware topology mapping, and combined parallelism configuration.
Parallelism Strategy Selection is an NVIDIA operational guide for choosing and combining parallelism strategies (tensor, data, pipeline, context, and expert parallelism) in Megatron Bridge. It solves the problem of deciding how to shard a model across GPUs: which degrees to use for a given model size, hardware topology, and sequence length, and how to trace an OOM or throughput regression back to a parallelism configuration change.
The skill provides decision tables by model size for both dense and MoE models (with concrete starting points for named models like DeepSeek-V2/V3, GLM-4.5, Qwen3, Kimi-K2), guidance by hardware topology (keep tensor parallelism within a single NVLink domain, use pipeline or data parallelism for cross-node scaling), and by sequence length (adding sequence and context parallelism as sequences grow). It gives copy-ready enablement snippets for 3D and 4D parallelism and MoE EP+PP configurations, and — notably — corrects a widespread sizing error: the minimum GPU count is PP times max(TP*CP, EP*ETP), not the full product PP*TP*CP*EP*ETP, because the attention and expert meshes overlap within a pipeline stage rather than multiply. It also covers implicit data-parallel sizing, scaling above the minimum, a memory-estimation worked example for a 70B model, and code anchors into the docs and model provider.
The target users are ML engineers and researchers configuring or debugging large-scale Megatron Bridge / NeMo training runs and sizing Slurm allocations. This is reference documentation and configuration guidance only — there are no scripts, destructive operations, credential handling, or network activity involved.
How to choose and combine tensor, data, pipeline, context, and expert parallelism degrees in Megatron Bridge based on model size, hardware topology, and sequence length, and how to trace OOM or regressions to a parallelism config change.
Because only a fraction of parameters are active per token, tensor parallelism is sized by active (not total) parameters and can often stay at 1 or 2, while expert parallelism becomes the primary scaling dimension and pipeline parallelism handles cross-node depth.
min_gpus = PP * max(TP*CP, EP*ETP). The common PP*TP*CP*EP*ETP full product is wrong because the attention and expert meshes share the same GPUs within a pipeline stage and overlap rather than multiply.
Within a single NVLink domain. Use pipeline or data parallelism for cross-node scaling, since tensor parallelism across nodes is almost always a performance loss.
Below 2K use standard TP+PP+DP; 2K-8K add sequence parallelism; 8K-32K add context parallelism of 2; and 32K+ use context parallelism of 4-8, considering a2a+p2p for large context parallel.
Quick Setup:
.claude/skills/Repository
nvidia/skills