launchdarkly-experiment-setup

Guides the end-to-end setup and operation of experiments in LaunchDarkly, where feature-flag variations are measured against key metrics. It covers designing the experiment, creating it with the right treatments and metrics, starting data collection, evolving the design across iterations, and stopping with a declared winner. It depends on the remotely hosted LaunchDarkly MCP server and its tools such as create-experiment, start-experiment-iteration, get-experiment, and stop-experiment-iteration, plus optional helpers for listing and updating experiments and managing metrics.

An experiment is built from treatments (the flag variations being compared, each with an allocationPercent that should sum to 100), metrics (one designated primary), a flag config (flagKey, ruleId, flagConfigVersion), and iterations (data-collection windows that move through not_started, running, and stopped). Optional holdouts exclude a group of users for baseline measurement. The documented lifecycle is: prepare metrics, identify the targeting rule, create the experiment with its first iteration, start data collection, monitor via get-experiment, evolve the design mid-flight if needed, then stop and ship the winner.

Use it when you want statistically grounded feature rollouts rather than guesswork. Core principles it enforces include metrics-first (ensure referenced metrics exist before creating the experiment), a clear hypothesis per iteration, exactly one baseline treatment, sufficient sample size, and testing one change at a time. Because most structural fields lock while an iteration is running, changing treatments, metrics, or methodology mid-experiment is done through save-and-start-experiment-iteration, which stops the current iteration, creates a new draft with the changes, and starts it in one call. Stopping requires declaring a winningTreatmentId (and winningReason); if nothing outperformed, the baseline is chosen as winner. Supported methodologies include bayesian (default), frequentist, and export_only, with optional Snowflake or Databricks data sources.