Back to Skills

tao-finetune-huggingface-model

Fine-tune any HuggingFace CV / VLM / LLM model on local NVIDIA GPUs inside an NGC PyTorch container. Use when the user wants to fine-tune a HuggingFace model (full or LoRA), train a vision / VLM / LLM model end-to-end, generate a reproducible HF training pipeline, smoke-test a HuggingFace model locally before scale-up, push a fine-tuned model to the HF Hub with a model card, or emit a self-contained rerun skill for an existing HuggingFace finetune. Supports image classification, object detection

2,984stars345forksUpdated 8/17/2026

Security Assessment

Safe(92/100)
Security Score92/100

About tao-finetune-huggingface-model

This NVIDIA skill orchestrates local fine-tuning of HuggingFace models on NVIDIA GPUs inside an NGC PyTorch container. It solves the problem of turning a HuggingFace model ID plus a dataset into a reproducible, deployable training pipeline without hand-assembling containers, scripts, and configs. It supports a broad task range — image classification, object detection, semantic/instance/panoptic segmentation, depth estimation, image-text-to-text VLM (SFT/LoRA), and LLM SFT/DPO/GRPO — with both full and LoRA fine-tuning.

The workflow is a fixed six-step process: inspect and qualify the model, pick hardware and the NGC image, research (live-fetching the model card, HF repo examples, task docs, and papers, with curated `references/*.md` as a fallback), generate scripts and smoke-test, train plus evaluate plus run inference, then push to the HF Hub with a model card and emit a self-contained rerun skill. It ships worked examples (ConvNeXt/CIFAR-10, DETR/CPPE-5, SegFormer/FoodSeg103, SmolVLM/VQAv2), extensive references for datasets, hardware, error playbooks, reporting, and testing, and an NVSkills-Eval BENCHMARK.md documenting a PASS verdict with 100% security scores. All settings live in `config.yaml` rather than hardcoded Python.

It targets ML engineers and researchers who want to fine-tune or smoke-test a HuggingFace model locally before scaling up, and to produce a reproducible, publishable pipeline. Credential handling is careful and defensive: `HF_TOKEN` is read only when a model/dataset is gated or when pushing to the Hub, and its value is never read — only its presence is checked; `WANDB_API_KEY`/`WANDB_PROJECT` apply only when Weights & Biases logging is enabled. It requires Docker with the NVIDIA Container Toolkit, an NVIDIA GPU (driver ≥545, ≥24 GB VRAM for models up to ~3B), and roughly 40 GB of free disk.

FAQ

What tasks and fine-tuning methods does it support?

Image classification, object detection, semantic/instance/panoptic segmentation, depth estimation, image-text-to-text VLM (SFT/LoRA), and LLM SFT/DPO/GRPO, with both full and LoRA fine-tuning.

What hardware and software are required?

Docker plus the NVIDIA Container Toolkit, an NVIDIA GPU (driver ≥545, at least 24 GB VRAM for models up to about 3B parameters), and roughly 40 GB of free disk. Training runs inside an NGC PyTorch container.

How are credentials handled?

`HF_TOKEN` is only needed when the model/dataset is gated or when pushing to the Hub, and its value is never read — the skill only checks presence via `[ -n "$HF_TOKEN" ]`. WANDB_API_KEY/WANDB_PROJECT are used only when WandB logging is enabled.

Can I reproduce or rerun a finetune later?

Yes. The final workflow step emits a self-contained rerun skill for an existing finetune, and all parameters live in `output_dir/config.yaml` rather than being hardcoded in Python, so runs are reproducible.

How do I supply data?

Provide exactly one of: a HuggingFace `dataset_id`, a `local_dataset_path` (with optional format among imagefolder, coco, voc, jsonl, arrow, parquet, csv), or omit it to let the agent recommend popular datasets.

All Files

73 files
BENCHMARK.md4.1 KB
View
examples/convnext-tiny-cifar10/config.yaml2.1 KB
View
examples/convnext-tiny-cifar10/infer.py2.5 KB
View
examples/convnext-tiny-cifar10/.gitignore0.9 KB
View
evals/evals.json0.8 KB
View
examples/convnext-tiny-cifar10/Dockerfile0.8 KB
View
examples/detr-resnet50-cppe5/infer.py3.4 KB
View
examples/detr-resnet50-cppe5/run_eval.py3.6 KB
View
examples/segformer-b0-foodseg103/config.yaml2.1 KB
View
examples/segformer-b0-foodseg103/requirements.txt0.7 KB
View
examples/smolvlm-256m-vqav2/infer.py2.8 KB
View
examples/smolvlm-256m-vqav2/prepare_data.py1.1 KB
View
examples/smolvlm-256m-vqav2/run_eval.py2.9 KB
View
references/dataset-patterns.md10.5 KB
View
references/dataset-recommendations.md11.0 KB
View
references/detailed-workflow.md0.7 KB
View
references/model-discovery.md10.5 KB
View
references/workflow-generate-train.md7.3 KB
View
examples/convnext-tiny-cifar10/reports/baseline_results.json0.4 KB
View
examples/convnext-tiny-cifar10/reports/eval_results.json0.4 KB
View
examples/convnext-tiny-cifar10/train.py5.5 KB
View
examples/detr-resnet50-cppe5/prepare_data.py1.3 KB
View
examples/detr-resnet50-cppe5/requirements.txt0.9 KB
View
examples/segformer-b0-foodseg103/.gitignore0.9 KB
View
examples/segformer-b0-foodseg103/run_eval.py3.1 KB
View
examples/segformer-b0-foodseg103/train.py6.6 KB
View
examples/smolvlm-256m-vqav2/.gitignore0.9 KB
View
examples/smolvlm-256m-vqav2/merge_lora.py1.1 KB
View
examples/smolvlm-256m-vqav2/reports/baseline_results.json1.6 KB
View
examples/smolvlm-256m-vqav2/train.py4.9 KB
View
references/hardware-container.md8.4 KB
View
references/research-priorities.md6.4 KB
View
references/vlm-scripts.md27.7 KB
View
skill-card.md4.1 KB
View
examples/convnext-tiny-cifar10/prepare_data.py1.4 KB
View
examples/detr-resnet50-cppe5/.gitignore0.9 KB
View
examples/detr-resnet50-cppe5/config.yaml2.0 KB
View
examples/detr-resnet50-cppe5/train.py9.0 KB
View
examples/segformer-b0-foodseg103/prepare_data.py1.1 KB
View
examples/smolvlm-256m-vqav2/config.yaml2.2 KB
View
examples/smolvlm-256m-vqav2/Dockerfile0.8 KB
View
examples/smolvlm-256m-vqav2/reports/eval_results.json1.2 KB
View
examples/smolvlm-256m-vqav2/requirements.txt0.8 KB
View
references/compat-workarounds.md11.8 KB
View
references/deliverables.md18.0 KB
View
references/reporting.md17.8 KB
View
references/tao-rerun-convnext-cifar10.md4.4 KB
View
references/tao-rerun-segformer-foodseg103.md3.7 KB
View
references/tao-rerun-smolvlm-vqav2.md4.4 KB
View
skill.oms.sig21.5 KB
View
examples/convnext-tiny-cifar10/requirements.txt0.9 KB
View
examples/convnext-tiny-cifar10/run_eval.py3.0 KB
View
examples/detr-resnet50-cppe5/Dockerfile0.8 KB
View
examples/detr-resnet50-cppe5/reports/baseline_results.json0.5 KB
View
examples/detr-resnet50-cppe5/reports/eval_results.json0.5 KB
View
examples/README.md6.0 KB
View
examples/segformer-b0-foodseg103/Dockerfile0.8 KB
View
examples/segformer-b0-foodseg103/infer.py3.3 KB
View
examples/segformer-b0-foodseg103/reports/baseline_results.json0.2 KB
View
examples/segformer-b0-foodseg103/reports/eval_results.json0.2 KB
View
references/core-rules.md4.7 KB
View
references/cv-scripts.md33.2 KB
View
references/dataset-sources.md17.0 KB
View
references/docker-runs.md8.9 KB
View
references/error-playbook.md3.7 KB
View
references/hub-push.md5.4 KB
View
references/pipeline-skill-template.md8.9 KB
View
references/progress-tracking.md9.5 KB
View
references/tao-rerun-detr-cppe5.md3.8 KB
View
references/testing.md20.8 KB
View
SKILL.md17.4 KB
View
references/workflow-intake-preflight.md22.2 KB
View
references/workflow-push-rerun.md3.0 KB
View

Install tao-finetune-huggingface-model

Download and extract the skill files to your .claude/skills/ directory.

Quick Setup:

  1. Copy the skill folder to .claude/skills/
  2. Claude will automatically detect and use the skill

Repository

nvidia/skills