This skill should be used when the user asks to implement LLM-as-judge, compare model outputs, create evaluation rubrics, mitigate evaluation bias, or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
The advanced-evaluation skill is designed to facilitate the evaluation of large language model (LLM) outputs by using LLMs themselves as judges. This skill addresses the challenges of ensuring consistent and reliable evaluation methods in scenarios where model outputs need to be compared, scored, or analyzed for quality. It synthesizes insights from academic research and industry practices to provide a robust framework for developing evaluation systems that mitigate bias and enhance reliability. By offering a structured approach to evaluation, this skill helps users navigate the complexities associated with assessing LLM performance effectively.
Key features of the advanced-evaluation skill include the ability to implement various evaluation approaches such as direct scoring and pairwise comparison, each tailored to different evaluation contexts. This skill also emphasizes the importance of understanding and mitigating biases that can affect evaluation outcomes, such as position bias and length bias. Additionally, it provides a metric selection framework to guide users in choosing the most appropriate metrics based on the type of evaluation task they are undertaking, ensuring that the evaluation process is both comprehensive and contextually relevant.
Target users for the advanced-evaluation skill include data scientists, AI researchers, and developers working with LLMs who need to implement evaluation pipelines or compare model outputs. It is particularly useful for those involved in quality assessment, A/B testing, or the development of evaluation rubrics. By leveraging this skill, users can enhance their evaluation strategies and ensure that their assessments are fair, consistent, and aligned with established quality standards.
Activate this skill when building automated evaluation pipelines, comparing model responses, or creating evaluation rubrics.
This skill supports direct scoring and pairwise comparison for evaluating LLM outputs.
Yes, this skill addresses biases such as position bias, length bias, and self-enhancement bias.
Primary metrics include Recall, Precision, F1 for binary classification, and Spearman's ρ, Kendall's τ for ordinal scales.
The documentation does not specify compatibility with particular programming languages or frameworks.
Quick Setup:
.claude/skills/