nemo-mbridge-perf-activation-recompute

Validate and use selective and full activation recompute in Megatron Bridge to reduce GPU memory usage at the cost of extra compute.

2,741stars319forksUpdated 7/31/2026

Data Science & ML #performance-tuning #gpu #machine-learning #nvidia #memory-optimization

Security Assessment

Safe(95/100)

Security Score95/100

About nemo-mbridge-perf-activation-recompute

Activation Recompute is an NVIDIA instruction skill covering how to validate and use selective and full activation recomputation in Megatron Bridge to reduce GPU memory usage at the cost of extra compute. It solves a common training-scale problem: when activation memory causes out-of-memory errors, which recompute strategy to apply and how to configure it correctly, including how to trace a regression back to a commit that changed recompute settings.

The skill explains the two granularities — selective (submodule-level, via a recompute_modules list such as core_attn or mlp) and full (layer-level, via recompute_num_layers plus recompute_method) — with a decision flow that first rules out allocator fragmentation, then starts with selective core_attn recompute, then adds modules by cost. It provides a per-module cost/savings table, exact Python and CLI enablement snippets, and detailed compatibility constraints (for example, full-layer recompute being incompatible with TE-scoped CUDA graphs, distribute_saved_activations conflicting with sequence_parallel, and cpu_offloading being incompatible with pipeline parallelism greater than 1). It also includes measured benchmark results on Llama3 70B SFT across 32 H100s showing throughput and peak-memory tradeoffs, and code anchors pointing to the relevant Megatron-LM and recipe source files.

The intended users are ML performance engineers and researchers tuning Megatron Bridge / NeMo training jobs to fit memory budgets, and those debugging OOM or throughput regressions. The content is reference documentation with configuration examples and read-only performance-harness commands; it contains no destructive or network-exfiltration behavior.

FAQ

What does activation recompute do?

It trades GPU compute for memory by discarding intermediate activations during the forward pass and recomputing them during backward, reducing peak activation memory at a throughput cost.

Selective or full recompute — which should I start with?

First rule out allocator fragmentation (expandable_segments), then start with selective recompute using recompute_modules=[core_attn]; use full-layer recompute only when selective does not fit and set recompute_method and recompute_num_layers.

What are the key compatibility constraints?

Full-layer recompute is incompatible with TE-scoped CUDA graphs, distribute_saved_activations cannot combine with sequence_parallel, and CPU offloading is incompatible with pipeline parallelism greater than 1.

Which modules give the best memory savings?

Per the benchmarks, mlp recompute saves the most memory (~3 GB on Llama3 70B) but costs ~16% throughput, layernorm is nearly free but saves negligible memory, and core_attn gives moderate savings at low cost.