arize-annotation

arize-annotation creates and manages annotation configs and annotation queues on the Arize observability platform, and applies human annotations to project spans programmatically. Annotation configs define the label schema — categorical (pick from a list), continuous (numeric range), or freeform (free text) — while annotation queues drive human review workflows across spans, dataset examples, experiment records, and queue items.

The skill works through the ax CLI (ax annotation-configs, ax annotation-queues) and the Python SDK's ArizeClient.spans.update_annotations for bulk labeling. It is designed to be run directly without upfront environment checks, and troubleshoots failures reactively — pointing to profile setup for auth errors and space listing when the target space is ambiguous. Notably, it is security-conscious: it instructs the agent never to read .env files or search the filesystem for credentials, relying on ax profiles for Arize keys and ax ai-integrations for LLM provider keys instead.

It targets ML and LLM teams doing evaluation and human-feedback labeling in Arize who need to define label schemas, stand up review queues, and attach annotations to traced spans via CLI and SDK.