content-experimentation-best-practices
Content experimentation and A/B testing guidance covering experiment design, hypotheses, metrics, sample size, statistical foundations, CMS-managed variants, and common analysis pitfalls. Use this skill when planning experiments, setting up variants, choosing success metrics, interpreting statistical results, or building experimentation workflows in a CMS or frontend stack.
Security Assessment
Detected risks:
About content-experimentation-best-practices
This skill provides comprehensive guidance for planning, executing, and analyzing content experiments including A/B tests and multivariate tests. It addresses the common challenge of making data-driven decisions about content changes rather than relying on opinions or assumptions. The skill covers the complete experimentation lifecycle from initial hypothesis formation through statistical analysis of results, helping teams build an experimentation culture that improves conversion rates, engagement, and user experience.
The skill includes four core reference areas: experiment design covering hypothesis frameworks, metrics selection, and sample size calculations; statistical foundations explaining p-values, confidence intervals, power analysis, and Bayesian methods; CMS integration patterns for managing variants at the field level or through external platforms; and a catalog of 17 common pitfalls spanning statistical mistakes, design flaws, execution issues, and interpretation errors. It emphasizes both the theoretical foundations of statistical testing and the practical implementation details needed for production experimentation systems.
This skill is designed for product managers planning experiments, engineers building experimentation infrastructure, data analysts interpreting test results, and content teams running CMS-integrated experiments. It's particularly valuable when setting up new A/B testing systems, debugging why experiments aren't reaching significance, integrating experimentation capabilities into content management workflows, or training teams to avoid common statistical and methodological mistakes that invalidate test results.
FAQ
When should I use A/B testing versus multivariate testing?
Use A/B testing when comparing two distinct variants or when traffic is limited. Multivariate testing is appropriate when you need to test multiple variables simultaneously to find optimal combinations, but requires significantly more traffic to reach statistical significance.
What statistical concepts do I need to understand to run valid experiments?
The skill covers essential statistical foundations including p-values, confidence intervals, statistical significance, power analysis, and Bayesian methods. The statistical-foundations reference provides detailed guidance on these concepts and when to apply them.
How do I integrate experimentation into my CMS?
The CMS integration reference covers patterns for managing variants within your content management system, implementing field-level variants, and connecting to external experimentation platforms. It addresses both technical implementation and content workflow considerations.
What are the most common mistakes that invalidate experiment results?
The skill catalogs 17 common pitfalls across four categories: statistical errors (like peeking at results early), design flaws (like testing too many variants), execution issues (like inconsistent implementation), and interpretation mistakes (like confusing statistical significance with practical significance).
How do I determine the right sample size for my experiment?
Sample size calculation is covered in the experiment design reference, which includes guidance on power analysis, expected effect sizes, and traffic requirements. The statistical foundations reference provides the mathematical basis for these calculations.
Install content-experimentation-best-practices
Quick Setup:
- Copy the skill folder to
.claude/skills/ - Claude will automatically detect and use the skill
Repository
sanity-io/agent-toolkit