tempo

Grafana Tempo guidance for standing up a cost-efficient distributed-tracing backend, one that needs only object storage, and for querying it with TraceQL. Tempo accepts traces over OTLP, Jaeger, Zipkin, OpenCensus, and Kafka, and stores Parquet blocks in S3, GCS, or Azure through a distributor to live-store to block-builder to object-storage write path. It covers ingestion, the metrics-generator that produces RED span metrics and service graphs, Helm-based deployment, multi-tenancy, and correlation links from traces to logs, metrics, and profiles. Several end-to-end workflows are documented. A local quick start clones the official Docker Compose example, verifies readiness at the /ready endpoint, pushes a synthetic OTLP span with curl, confirms ingestion via the tempo_distributor_spans_received_total metric and a trace fetch, then queries it in Grafana Explore. A second path forwards application traces through Grafana Alloy, configuring an OTLP receiver and an OTLP exporter pointed at Tempo, and verifies delivery through Alloy's exporter metrics. TraceQL examples show filtering slow requests with a duration predicate, using structural operators such as >> to find a server span with a downstream error, and computing error rate per service with rate() by resource.service.name. Queries can also be issued through the /api/search endpoint with URL-encoded parameters. For production, a Helm install of the tempo-distributed chart (version 1.61.3) wires up S3 backend storage, and readiness is checked across the distributor, ingester, querier, query-frontend, and compactor pods. Multi-tenancy is enabled with multitenancy_enabled and an X-Scope-OrgID header on every request. Troubleshooting guidance addresses a 503 from /ready, 429s on push, missing traces in Explore, and slow TraceQL. Deeper references cover the full TraceQL cheat sheet plus architecture, ports, tuning, and datasource links. Use it when deploying Tempo, writing TraceQL for slow or errored requests, sizing components, or wiring trace-to-log correlation.