mcore-testing

The mcore-testing skill documents the test system for the Megatron-LM repository, covering test layout, recipe YAML structure, how to add and run unit and functional tests, golden values, marker filters, and CI parity. It solves the onboarding and day-to-day friction of a complex, GPU-and-container-based test harness where the right way to run, disable, or reproduce a test is non-obvious.

The guide leads with answer-first facts, such as how to disable a test without deleting it: functional recipe entries stay in YAML and are disabled by suffixing their scope with -broken, while unit-test skips use pytest markers (flaky_in_dev skips in dev, flaky skips in LTS). It maps the tests/ directory (unit_tests, functional_tests with test_cases, and test_utils with recipes and python_scripts), explains how tests execute via launch_nemo_run_workload.py using nemo-run to launch a DockerExecutor container with the repo bind-mounted, and documents rank logging, artifact upload, and up-to-3 automatic retries for known transient failures like NCCL timeouts and ECC errors. It details the recipe YAML structure with its cartesian products expansion and runtime placeholders, and shows how to run unit tests locally through torch.distributed.run since every test initializes a distributed group and requires GPU access.

Target users are developers and engineers contributing to Megatron-LM who need to add, run, or debug unit and functional tests and match CI behavior locally. It is a read-and-reference guide rather than an executable tool, keeping the source repo's conventions authoritative.