Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deploymen
Detected risks:
Agent Platform Eval Flywheel measures and improves the quality of GenAI models and agents on Google Cloud using the 'Eval Quality Flywheel' methodology and the Agent Platform GenAI Evaluation SDK (google.genai / agentplatform). It applies when evaluating an agent or model, building an eval dataset, choosing or writing metrics, analyzing failures, or comparing results before and after a fix. For fine-tuning and general deployment it defers to sibling skills.
Setup requires installing google-cloud-aiplatform[evaluation]>=1.154.0 and google-genai>=1.0.0, and setting GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION (newer Gemini models often need location set to global). The skill enforces safety tiers before running anything: Tier R read-only helper scripts (inspect, compare, validate, parse traces, render reports) run without confirmation, while Tier M operations that invoke LLMs or remote services and incur compute costs - run_inference, evaluate, generate_user_scenarios, generate_loss_clusters - require interactive confirmation.
The Flywheel is five stages, run in order on the first pass and then looped from stage two to five until quality targets are met. Prepare Data produces an EvaluationDataset from an EvalCase list, a pandas DataFrame, or, with no data at all, server-side synthesized user scenarios. Run Inference populates responses and traces (skipped when traces already exist). Grade always runs client.evals.evaluate() with metrics chosen by intent - agent metrics like multi_turn_task_success, multi_turn_trajectory_quality, and multi_turn_tool_use_quality, or single-turn metrics like general_quality and final_response_match. Later stages analyze rubric verdicts and cluster failures. The skill explicitly warns against shortcuts such as lowering thresholds to pass, skipping flaky cases, or stopping after one iteration, since real improvement typically takes five to ten or more loops.
The Agent Platform GenAI Evaluation SDK (google.genai / agentplatform), centered on client.evals.evaluate() for grading.
Install google-cloud-aiplatform[evaluation]>=1.154.0 and google-genai>=1.0.0, and set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION; newer Gemini models often need the location set to global.
Tier R read-only helper scripts run without confirmation, while Tier M operations that invoke LLMs and incur compute costs - like run_inference, evaluate, and generate_user_scenarios - require interactive confirmation.
Cold-start by synthesizing scenarios server-side with client.evals.generate_user_scenarios and a UserScenarioGenerationConfig, then play them out during Run Inference.
Start with multi-turn agent metrics such as multi_turn_task_success, multi_turn_trajectory_quality, and multi_turn_tool_use_quality; single-turn model evaluation uses metrics like general_quality and final_response_match.
Quick Setup:
.claude/skills/Repository
google/skills