launch-nemo-rl

Playbook for launching, monitoring, stopping, and debugging NeMo-RL recipes on a Kubernetes cluster via the nrl-k8s CLI. Covers ephemeral vs long-lived RayCluster modes, iterating on runs, and debugging hung or failed training jobs.

2,728stars318forksUpdated 7/30/2026

DevOps & CI/CD #nemo-rl #devops #gpu #reinforcement-learning #training #kubernetes

Security Assessment

Low Risk(84/100)

Detected risks:

Shared Cluster State Changes(The skill launches, submits, and tears down RayClusters and training jobs on a shared, expensive GPU cluster, and its own guidance warns that the cost of a wrong action is high., Long-lived mode reuses or recreates clusters and a --recreate flag replaces an existing cluster, so an incorrect invocation could disrupt other users' work on the shared resource.)

Security Score84/100

About launch-nemo-rl

launch-nemo-rl is a playbook for launching, monitoring, stopping, and debugging NVIDIA NeMo-RL reinforcement-learning training recipes on a Kubernetes cluster via the nrl-k8s CLI (located at infra/nrl_k8s/). It solves the operational problem of running RLHF-style training jobs on shared GPU clusters: choosing between ephemeral (one-shot, auto-teardown) and long-lived RayCluster modes, iterating on runs, and debugging hung or failed jobs. It emphasizes verifying current state (kubectl, git log, recipe and infra files) before acting, because the cluster is shared and the cost of a wrong action is high.

The CLI centers on a single nrl-k8s run command with two lifecycle modes plus observability and stage control subcommands: check (validate a recipe/infra pair and optionally render manifests), status (per-role RayCluster and pod state), cluster up/down/list/dashboard (manage RayClusters independently), and job list/logs/stop plus a logs tailer. Every launch pairs a recipe file (NeMo-RL config: model, GRPO/SFT knobs, cluster shape) with a separate infra file (namespace, image, service account, RayCluster spec, submitter, launch mode). Long-lived mode exposes independent dimensions via a --mode macro (interactive versus batch) with submitter and code-source overrides.

Target users are ML engineers and researchers running NeMo-RL training on Kubernetes. The skill both launches and tears down clusters and jobs on a shared, expensive resource, so its guidance repeatedly stresses verifying state first and using reuse-with-drift-warning semantics rather than blind recreation. Its pre-publication benchmark returned an overall PASS verdict, evaluated with claude-code and codex agents, with security scored around 80 percent.

FAQ

What is the difference between the two run modes?

Ephemeral (default nrl-k8s run) applies a RayJob, runs it, and tears the cluster down automatically, best for one-shot runs. Long-lived (--raycluster) reuses a matching live cluster (or applies one if absent) and keeps it up afterward, best for iterative dev loops.

What files does a launch require?

Two: a recipe YAML (NeMo-RL config with model and GRPO/SFT knobs and cluster shape) and a separate infra YAML (Kubernetes/Ray shape) passed with --infra, following the naming convention recipe.profile[.prod].infra.yaml.

How do I get logs or check why a run is stuck?

Use nrl-k8s status for per-role RayCluster and pod state, nrl-k8s job list/logs/stop for jobs already submitted, and nrl-k8s logs to tail a role's pod or daemon logs without a submission id.

What should I do before launching?

Verify current state with kubectl, git log, and the recipe and infra files. The playbook stresses this because the cluster is shared and a wrong action is costly; drift on a reused cluster triggers a warning and reuse rather than silent recreation (use --recreate to replace).

Can it tear down clusters?

Yes. Ephemeral runs auto-tear-down, and nrl-k8s cluster down manages RayClusters directly, so it performs state-changing operations on a shared cluster and should be used with the state-verification cautions the playbook describes.