nemo-mbridge-perf-tp-dp-comm-overlap

nemo-mbridge-perf-tp-dp-comm-overlap is an NVIDIA operational guide for enabling tensor-parallel (TP), data-parallel (DP), and pipeline-parallel (PP) communication overlap in Megatron-Bridge. It solves a common distributed-training performance problem: overlapping communication with computation to reduce idle time, and tracing throughput regressions back to a comm-overlap config change. It provides a minimal Bridge override that wires up TP comm overlap with sequence parallelism, distributed optimizer, and grad-reduce/param-gather overlap, plus an optional user-buffers TP preset and the correct placement of precision knobs under mixed precision.

The skill includes code anchors into Bridge source for the overlap gating logic, PP overlap selection (auto-selecting overlap_p2p_comm only when PP>1 and VPP>1), DP overlap defaults, and launch-time environment tuning such as CUDA_DEVICE_MAX_CONNECTIONS and LayerNorm SM margin. A pitfalls section warns about non-obvious behavior: TP overlap silently disabling itself without sequence parallel or Transformer Engine, bucket_size being a parameter-count rather than byte-size knob, and precision/env settings belonging to mixed precision or launch plugins rather than CommOverlapConfig. Verification is done through checked-in unit tests with explicit pass counts (26 passed).

It targets performance and infrastructure engineers tuning large-model training throughput on Megatron-Bridge. The output is configuration instructions plus pytest verification commands — no destructive operations, credential handling, or network exfiltration. NVSkills-Eval reported a 100% Security score.