PluginEval quality methodology — dimensions, rubrics, statistical methods, and scoring formulas. Use this skill when understanding how plugin quality is measured, when interpreting a low score on a specific dimension, when deciding how to improve a skill's triggering accuracy or orchestration fitness, when calibrating scoring thresholds for your marketplace, or when explaining quality badges to external partners like Neon.
The evaluation-methodology skill provides a structured framework for assessing the quality of plugins and AI skills within the PluginEval ecosystem. It addresses the challenge of objectively measuring skill performance across multiple dimensions, ensuring consistent scoring and actionable insights for improvement. By defining precise evaluation layers and scoring mechanisms, it helps developers, marketplace operators, and quality teams understand why a skill performs well or poorly and guides targeted enhancements.
This skill features a three-layered evaluation process: Static Analysis, LLM Judge, and Monte Carlo Simulation. Static Analysis examines the SKILL.md file for structural completeness, token efficiency, and ecosystem coherence, producing rapid, deterministic results. The LLM Judge leverages anchored rubrics to score triggering accuracy, orchestration fitness, output quality, and scope calibration using natural language evaluation. Monte Carlo Simulation performs statistical analysis by simulating multiple skill invocations to measure activation rates, output consistency, failure rates, and token efficiency. The skill also incorporates anti-pattern penalties, composite scoring formulas, Elo ranking, and threshold-based quality badges to provide a comprehensive assessment of skill performance.
Targeted for developers, quality assurance teams, and marketplace managers, this skill is useful for understanding plugin performance, diagnosing weaknesses, calibrating thresholds, and communicating quality metrics externally. Typical use cases include interpreting low scores on specific dimensions, optimizing skill triggering and orchestration, setting marketplace standards, and explaining quality badges to partners or stakeholders. It empowers users to systematically improve skill effectiveness and maintain high-quality plugin ecosystems.
The skill can be used to assess plugins or skills by running the three evaluation layers: Static Analysis, LLM Judge, and Monte Carlo Simulation. Each layer produces scores for specific dimensions, which are combined to generate a composite quality score.
The skill primarily evaluates SKILL.md files and related references. While it is language-agnostic in terms of the content being evaluated, it assumes the presence of PluginEval-compatible skill definitions and references.
Static Analysis cannot assess output quality or triggering accuracy. LLM Judge scoring is non-deterministic and may vary slightly between runs. Monte Carlo Simulation requires multiple simulated invocations and can be time-consuming. Anti-pattern detection may not cover all possible skill design issues.
Yes, when multiple LLM judges are used, their scores are averaged and inter-judge agreement is reported using Cohen's kappa to assess reliability.
Quality badges are based on composite scores derived from all three evaluation layers, with specific thresholds for each badge. Anti-pattern penalties and dimension-specific weights influence the final score.
Quick Setup:
.claude/skills/Repository
wshobson/agents