observability-logs-search

A guided workflow for searching and filtering Observability logs with ES|QL during incident investigation. It is built for moments when you are chasing log spikes, errors, or anomalies, gauging volume and trends, or drilling into a specific service or container while an incident unfolds. It is explicitly scoped to logs - metrics and traces are handed off to their dedicated tools - and it insists on ES|QL via POST /_query rather than Query DSL.

The central idea is a funnel that mirrors Kibana Discover. Start broad with just a scope filter such as a service name and a time range, then iteratively add NOT exclusion clauses for the dominant noise patterns, always keeping the full accumulated filter, until fewer than twenty distinct log patterns remain - the point at which the surviving set is small enough to read as the actual errors or root cause. Stopping early, the skill warns, means reporting noise instead of failures. It standardizes parameter names (start, end, kqlFilter, limit, groupBy), tells you to filter on ECS field names, and stresses context minimization: KEEP only a handful of fields per sample and cap sample sizes so a step stays around a thousand tokens rather than thousands.

To drive each round it recommends a single five-way FORK query returning a trend histogram, a total count, sample logs, common message patterns, and rare message patterns in one request, and explains how to map and read those fork results to decide the next exclusion. The KQL guidance is notably candid about pitfalls - phrase queries and wildcards are preferred, while filtering on log.level or blindly searching for the word error is called out as often flawed because level metadata is unreliable and error matches harmless text. It also describes pivoting to an isolated entity or viewing logs in context when the funnel alone does not reveal the cause.