nemo-mbridge-perf-sequence-packing

nemo-mbridge-perf-sequence-packing is an NVIDIA operational guide for validating and using packed sequences and long-context training in Megatron-Bridge. It solves the problem of correctly enabling sequence packing while respecting the framework's many interacting constraints, and it draws a clear distinction between offline packed SFT for LLM finetuning and in-batch packing for VLM finetuning. It provides ready-to-use config snippets for each path, including PackedSequenceSpecs setup, micro-batch-size requirements, and a long-context baseline.

A central focus is context-parallel (CP) and sequence-parallel (SP) correctness. The skill shows how to set pad_seq_to_mult for CP, notes that ConfigContainer computes the CP/SP LCM automatically only for in-batch packing (so offline packing must be aligned explicitly), and flags that pad_cu_seqlens with CUDA graphs additionally requires a metadata JSON file or dataset initialization will hit an assertion. It backs this with code anchors into Bridge source for the packed SFT config surface and the validation logic that enforces mutual exclusivity of offline and in-batch packing, the presence of offline_packing_specs, per-token-loss requirements under CP, and micro-batch-size limits.

It is aimed at ML engineers enabling sequence packing or long-context SFT, or investigating a commit that broke packing behavior. Output is configuration instructions plus source references — no destructive operations, credentials, or network access. NVSkills-Eval reported a 100% Security score and a strong uplift over the no-skill baseline.