alerting-irm

Configures Grafana Alerting, Incident Response Management (IRM), and SLOs end to end. Coverage spans Grafana-managed and data-source-managed alert rules, contact points (Slack, PagerDuty, email, webhook), hierarchical notification policies with label matchers, silences, mute timings, on-call schedules and escalation chains, incident-management integrations, and SLOs with multi-window burn-rate alerts. It applies whether the user explicitly names Alerting or IRM or simply asks to be paged on errors, route alerts to a team, set up on-call, declare incidents, or define an SLO. A documented end-to-end workflow provisions a new alert: create contact points, where notifications go, via the provisioning contact-points API and verify them, add notification policies, which alerts go where, using label matchers, write the alert rule as Grafana-managed, Prometheus/Mimir ruler, or Loki LogQL, then verify routing before going live by force-firing a test alert and inspecting Alertmanager to confirm the expected receiver appears. Routing to IRM and on-call uses a Grafana Alerting webhook integration whose URL becomes a webhook contact point, with a policy matcher directing the right severity to it. Defining an SLO auto-generates recording rules, dashboards, and burn-rate alerts, and the guidance recommends multi-window burn-rate alerts over single-window to avoid firing on noise. Contact points and notification policies are shown as YAML provisioning files, including a hierarchical routing tree that sends critical alerts to PagerDuty, team-scoped alerts to Slack, and everything else to email, with nested routes and regex matchers. Silences are created through the Alertmanager API to suppress matching notifications without stopping evaluation. A reference table documents alert-rule states of Normal, Pending, Firing, NoData, Error, and Recovering, and a standard provisioning directory layout is provided. When provisioning through the API, adding the X-Disable-Provenance: true header keeps the resulting resources editable in the UI. Deeper material lives in referenced files for full alert-rule YAML, generated SLO rules, and IRM integration and troubleshooting.