Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.
This NVIDIA skill documents the resiliency features available in Megatron Bridge for large-scale distributed training, and how to enable them. It solves a costly operational problem: long multi-node training jobs are vulnerable to hardware faults, hung ranks, stragglers, and preemption, and recovering from these manually wastes cluster time. The skill gathers the fault-tolerance, straggler-detection, preemption, re-run, and in-process-restart mechanisms into a single enablement reference with concrete configuration and parameters.
It shows two ways to enable Slurm fault tolerance (a recommended NeMo Run plugin and a direct FaultToleranceConfig launched via ft_launcher), with parameter tables for restart limits, heartbeat timeouts, and section-based timeout monitoring saved to a state file. It covers NVRx straggler detection with relative and individual GPU performance thresholds, preemption handling via plugin or signal-handler config, an experimental re-run state machine that can validate results and check for NaN or spiky loss with documented exit codes, and experimental in-process restart with node/rank granularity, soft/hard timeouts, heartbeat and barrier timeouts, and required environment variables. It notes constraints such as the NCCL watchdog timeout needing to exceed the hard timeout.
It targets ML infrastructure and training engineers running Megatron Bridge on clusters (particularly Slurm) who need to keep long jobs alive through faults and detect degraded hardware. The content is configuration documentation with Python and shell snippets that set training options, including a fault-simulation switch intended only for testing resiliency; there are no destructive or exfiltration behaviors. Its published card reports a passing evaluation.
Fault tolerance, NVRx straggler detection, preemption handling, an experimental re-run state machine, and experimental in-process restart, all for Megatron Bridge training.
The documented fault-tolerance enablement is for Slurm, offered via a NeMo Run plugin or a direct config launched with ft_launcher (not torchrun); non-Slurm use requires setting GROUP_RANK.
It is a testing switch that injects a simulated fault (rank_hung, rank_killed, or random) so you can validate that fault-tolerance handling works; it defaults to False.
NVRxStragglerDetectionConfig compares ranks against each other and tracks per-rank degradation over time against configurable relative and individual performance thresholds, optionally stopping training if detected.
It is experimental, uses soft/hard timeouts and heartbeats, requires certain environment variables, and the PyTorch NCCL watchdog timeout must exceed the configured hard_timeout.
Quick Setup:
.claude/skills/Repository
nvidia/skills