sentry-setup-ai-monitoring

A feature-setup skill that configures Sentry AI Agent Monitoring in a project so it can track LLM calls, agent executions, tool usage, and token consumption. It activates when asked to monitor LLM calls, track AI agents or conversations, or instrument SDKs such as OpenAI, Anthropic, Vercel AI, LangChain, Google GenAI, or Pydantic AI, and it works by detecting installed AI SDKs and configuring the appropriate integrations. The skill notes that the SDK versions, API names, and code samples it contains are examples and should be verified against docs.sentry.io before implementing, since APIs and minimum versions may change.

AI monitoring requires tracing to be enabled (a non-zero traces sample rate), and for multi-turn chats the skill recommends setting a conversation ID since Sentry uses gen_ai.conversation.id to group related AI spans into Conversations. A prominent data-capture warning explains that recording prompts and outputs captures user content that is likely PII; before enabling send-default-PII or per-integration prompt/output capture, the agent must confirm the privacy policy permits it, that captured data complies with regulations like GDPR and CCPA, and that retention settings are appropriate, and it must ask the user for explicit confirmation. It also advises against using a 100% sample rate in production, suggesting a lower rate or a tracesSampler function. A detection-first step greps package.json or requirements/pyproject files to find installed AI SDKs, followed by a sampling check; if the sample rate is below 1.0 with no sampler, it offers to set up a tracesSampler that keeps AI traces at 100% because agent runs are sampled as complete span trees.

Supported SDK tables cover JavaScript integrations (openAIIntegration, anthropicAIIntegration, vercelAIIntegration, langChainIntegration, langGraphIntegration, googleGenAIIntegration) with minimum Sentry SDK versions, and Python integrations that auto-enable when the package is installed, with litellm noted as requiring explicit registration. Configuration examples show Node.js auto-enabled setup, manual client wrapping for browser and Next.js via instrumentOpenAiClient, LangChain/LangGraph, and Vercel AI with experimental_telemetry per call, plus Python init with stream_gen_ai_spans. For unsupported SDKs, manual instrumentation follows the Sentry Conventions for gen_ai.* attributes, with span types for individual LLM calls, agent invocation, tool execution, and agent handoffs.