dynamo-troubleshoot

Dynamo Troubleshoot turns a failed or unhealthy NVIDIA Dynamo Kubernetes deployment into a clear problem class, the strongest diagnostic signal, and a concrete next action. It is the day-2 companion to the recipe-runner and router-starter bring-up skills, focused on evidence-first triage: start with read-only data, avoid secrets, and fix one layer at a time.

The workflow collects a read-only debug bundle via a bundled collect_dynamo_debug_bundle.py script (pods, events, jobs, PVCs, and DynamoGraphDeployment custom-resource status), classifies the failure into one primary bucket (cluster/platform, namespace/secret, model cache/PVC/download, image pull, GPU scheduling, operator reconciliation, frontend/router, worker/backend, endpoint/API, or benchmark job), then debugs top-down from namespace and storage down through PVCs, custom-resource status, pod logs, the frontend service, and finally the API endpoints. Remediation is deliberately conservative: it prefers the smallest reversible change and returns commands or patches rather than executing them. The skill is emphatic about safety, instructing that it must not collect Kubernetes secrets and must not print Hugging Face tokens, and it advertises itself as read-only, never mutating the cluster.

Target users are ML infrastructure engineers and operators diagnosing Dynamo deployment failures with read-only cluster access (pods, events, jobs, PVCs, and CR status, explicitly not secrets). Its pre-publication benchmark returned an overall FAIL verdict with medium findings largely about undeclared tool permissions and the self-declared read-only claim, but the documented behavior itself is diagnostic and non-destructive, collecting evidence and proposing fixes for a human to apply.