nemo-mbridge-perf-cpu-offloading

CPU Offloading is an NVIDIA instruction skill covering how to validate and use CPU offloading in Megatron Bridge to relieve GPU memory pressure during training. It addresses two independent mechanisms: layer-level activation offloading (moving per-transformer-layer activations, and optionally weights, from GPU to CPU) and fractional optimizer-state offloading (moving Adam momentum and variance to CPU via HybridDeviceOptimizer). It also helps investigate commits that changed offloading config and caused OOM or crashes.

The skill provides a quick-decision matrix mapping situations to recommendations — for example, optimizer offloading for large MoE models needing pipeline parallelism greater than 1 (since activation offloading requires PP=1), activation offloading for small/medium models where activation memory dominates, and a tunable optimizer_offload_fraction for a memory-speed tradeoff. It gives exact Python and CLI enablement for both mechanisms, a full config-parameter reference with defaults, and detailed compatibility constraints (activation offloading requires PP=1, recompute_granularity=None, and no CUDA graphs; optimizer offloading requires the distributed optimizer but has no PP/recompute/CUDA-graph restrictions). It includes a minimal runnable command for a Qwen3-30B-A3B pretrain config and a practical note that activation offloading is blocked for large MoE models because the PP=1 constraint makes weights plus optimizer states exceed H100 80 GB.

The intended users are ML performance engineers tuning Megatron Bridge / NeMo training to fit memory budgets and those debugging offloading-related OOM or crashes. This is commercial/non-commercial-ready reference documentation with configuration examples and read-only training commands, with no destructive or exfiltration behavior.