Skillintermediate

PromQL Query Generator

This skill provides a comprehensive, interactive workflow for generating production-ready PromQL queries with best practices built-in. Generate queries for monitoring dashboards, alerting rules, and ad-hoc analysis with an emphasis on user collaboration and planning before code generation.

Claude Code Knowledge Pack7/10/2026

Overview

PromQL Query Generator

Overview

When to Use This Skill

Invoke this skill when:

Creating new PromQL queries from scratch
Building monitoring dashboards (Grafana, Prometheus UI, etc.)
Implementing alerting rules for Prometheus Alertmanager
Analyzing metrics for troubleshooting or capacity planning
Converting monitoring requirements into PromQL expressions
Learning PromQL or teaching others
The user asks to "create", "generate", "build", or "write" PromQL queries
Working with Prometheus metrics (counters, gauges, histograms, summaries)
Implementing RED (Rate, Errors, Duration) or USE (Utilization, Saturation, Errors) metrics

Interactive Query Planning Workflow

CRITICAL: This skill emphasizes interactive planning before query generation. Always engage the user in a collaborative planning process to ensure the generated query matches their exact intentions.

Follow this workflow when generating PromQL queries:

Stage 1: Understand the Monitoring Goal

Start by understanding what the user wants to monitor or measure. Ask clarifying questions to gather requirements:

Primary Goal: What are you trying to monitor or measure?
- Request rate (requests per second)
- Error rate (percentage of failed requests)
- Latency/duration (response times, percentiles)
- Resource usage (CPU, memory, disk, network)
- Availability/uptime
- Queue depth, saturation, throughput
- Custom business metrics
Use Case: What will this query be used for?
- Dashboard visualization (Grafana, Prometheus UI)
- Alerting rule (firing when threshold exceeded)
- Ad-hoc troubleshooting/analysis
- Recording rule (pre-computed aggregation)
- Capacity planning or SLO tracking
Context: Any additional context?
- Service/application name
- Team or project
- Priority level
- Existing metrics or naming conventions

Use the AskUserQuestion tool to gather this information if not provided.

When to Ask vs. Infer: If the user's initial request already clearly specifies the goal, use case, and context (e.g., "Create an alert for P95 latency > 500ms for payment-service"), you may acknowledge these details in your response instead of re-asking. Only ask clarifying questions for information that is missing or ambiguous.

Stage 2: Identify Available Metrics

Determine which metrics are available and relevant:

Metric Discovery: What metrics are available?
- Ask the user for metric names
- If uncertain, suggest common naming patterns
- Check for metric type indicators in the name:
  - _total suffix → Counter
  - _bucket, _sum, _count suffix → Histogram
  - No suffix → Likely Gauge
  - _created suffix → Counter creation timestamp
Metric Type Identification: Confirm the metric type(s)
- Counter: Cumulative metric that only increases (or resets to zero)
  - Examples: http_requests_total, errors_total, bytes_sent_total
  - Use with: rate(), irate(), increase()
- Gauge: Point-in-time value that can go up or down
  - Examples: memory_usage_bytes, cpu_temperature_celsius, queue_length
  - Use with: avg_over_time(), min_over_time(), max_over_time(), or directly
- Histogram: Buckets of observations with cumulative counts
  - Examples: http_request_duration_seconds_bucket, response_size_bytes_bucket
  - Use with: histogram_quantile(), rate()
- Summary: Pre-calculated quantiles with count and sum
  - Examples: rpc_duration_seconds{quantile="0.95"}
  - Use _sum and _count for averages; don't average quantiles
Label Discovery: What labels are available on these metrics?
- Common labels: job, instance, environment, service, endpoint, status_code, method
- Ask which labels are important for filtering or grouping

Use the AskUserQuestion tool to confirm metric names, types, and available labels.

Stage 3: Determine Query Parameters

Gather specific requirements for the query.

Pre-confirmation for User-Provided Parameters

IMPORTANT: When the user has already specified parameters in their initial request (e.g., "5-minute window", "500ms threshold", "> 5% error rate"), you MUST:

Acknowledge the provided values explicitly in your response
Present them as pre-filled defaults in AskUserQuestion with the first option being "Use specified values"
Allow quick confirmation rather than re-asking for information already given

Example: If user says "alert when P95 latency exceeds 500ms", use:

AskUserQuestion:
- Question: "Confirm the alert threshold?"
- Options:
  1. "500ms (as specified)" - Use the threshold from your request
  2. "Different threshold" - Let me specify a different value

This respects the user's input and speeds up the workflow while still allowing modifications.

Time Range: What time window should the query cover?
- Instant value (current)
- Rate over time ([5m], [1h], [1d])
- For rate calculations: typically [1m] to [5m] for real-time, [1h] to [1d] for trends
- Rule of thumb: Rate range should be at least 4x the scrape interval
Label Filtering: Which labels should filter the data?
- Exact matches: job="api-server", status_code="200"
- Negative matches: status_code!="200"
- Regex matches: instance=~"prod-.*"
- Multiple conditions: {job="api", environment="production"}
Aggregation: Should the data be aggregated?
- No aggregation: Return all time series as-is
- Aggregate by labels: sum by (job, endpoint), avg by (instance)
- Aggregate without labels: sum without (instance, pod), avg without (job)
- Common aggregations: sum, avg, max, min, count, topk, bottomk
Thresholds or Conditions: Are there specific conditions?
- For alerting: threshold values (e.g., error rate > 5%)
- For filtering: only show series above/below a value
- For comparison: compare against historical data (offset)

Use the AskUserQuestion tool to gather or confirm these parameters. When the user has already provided values (e.g., "5-minute window", "> 5%"), present them as the default option for confirmation.

Stage 4: Present the Query Plan

BEFORE GENERATING ANY CODE, present a plain-English query plan and ask for user confirmation:

## PromQL Query Plan

Based on your requirements, here's what the query will do:

**Goal**: [Describe the monitoring goal in plain English]

**Query Structure**:
1. Start with metric: `[metric_name]`
2. Filter by labels: `{label1="value1", label2="value2"}`
3. Apply function: `[function_name]([metric][time_range])`
4. Aggregate: `[aggregation] by ([label_list])`
5. Additional operations: [any calculations, ratios, or transformations]

**Expected Output**:
- Data type: [instant vector/scalar]
- Labels in result: [list of labels]
- Value represents: [what the number means]
- Typical range: [expected value range]

**Example Interpretation**:
If the query returns `0.05`, it means: [plain English explanation]

**Does this match your intentions?**
- If yes, I'll generate the query and validate it
- If no, let me know what needs to change

Use the AskUserQuestion tool to confirm the plan with options:

"Yes, generate this query"
"Modify [specific aspect]"
"Show me alternative approaches"

When the user chooses:

"Modify [specific aspect]": ask one focused follow-up question about what to change (metric, labels, function, time range, threshold, or output shape), then present an updated plan before generating.
"Show me alternative approaches": provide at least two valid query plans with trade-offs (accuracy, cost, cardinality, readability), then ask the user to choose one before generating.

Stage 5: Generate the PromQL Query

Once the user confirms the plan, generate the actual PromQL query following best practices.

IMPORTANT: Consult Reference Files Before Generating

Before writing any query code, you MUST:

Identify the query category first (histogram, RED, USE, function-specific, optimization, etc.).
Read only the relevant reference section(s) using the Read tool:
- For histogram queries → Read references/metric_types.md (Histogram section)
- For error/latency patterns → Read references/promql_patterns.md (RED method section)
- For resource monitoring → Read references/promql_patterns.md (USE method section)
- For optimization questions → Read references/best_practices.md
- For specific functions → Read references/promql_functions.md
- Re-read a section only if requirements changed or you have not consulted it yet in the current thread.
If a needed reference cannot be read, state the issue and continue with best-effort generation using the most applicable documented pattern you already have.

Cite the applicable pattern or best practice in your response:

As documented in references/promql_patterns.md (Pattern 3: Latency Percentile):
# 95th percentile latency
histogram_quantile(0.95, sum by (le) (rate(...)))

Reference example files when generating similar queries:

Based on examples/red_method.promql (lines 64-82):
# P95 latency with proper histogram_quantile usage

This keeps generated queries aligned with documented patterns while avoiding unnecessary full-file rereads on iterative follow-ups.

Best Practices for Query Generation

Always Use Label Filters

# Good: Specific filtering reduces cardinality
rate(http_requests_total{job="api-server", environment="prod"}[5m])

# Bad: Matches all time series, high cardinality
rate(http_requests_total[5m])

Use Appropriate Functions for Metric Types

# Counter: Use rate() or increase()
rate(http_requests_total[5m])

# Gauge: Use directly or with *_over_time()
memory_usage_bytes
avg_over_time(memory_usage_bytes[5m])

# Histogram: Use histogram_quantile()
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

Apply Aggregations with by() or without()

# Aggregate by specific labels (keeps only these labels)
sum by (job, endpoint) (rate(http_requests_total[5m]))

# Aggregate without specific labels (removes these labels)
sum without (instance, pod) (rate(http_requests_total[5m]))

Use Exact Matches Over Regex When Possible

# Good: Faster exact match
http_requests_total{status_code="200"}

# Bad: Slower regex match when not needed
http_requests_total{status_code=~"200"}

Calculate Ratios Properly

# Error rate: errors / total requests
sum(rate(http_requests_total{status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Use Recording Rules for Complex Queries
- If a query is used frequently or is computationally expensive
- Pre-aggregate data to reduce query load
- Follow naming convention: level:metric:operations

Format for Readability

# Good: Multi-line for complex queries
histogram_quantile(0.95,
  sum by (le, job) (
    rate(http_request_duration_seconds_bucket{job="api-server"}[5m])
  )
)

Common Query Patterns

Pattern 1: Request Rate

# Requests per second
rate(http_requests_total{job="api-server"}[5m])

# Total requests per second across all instances
sum(rate(http_requests_total{job="api-server"}[5m]))

Pattern 2: Error Rate

# Error ratio (0 to 1)
sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api-server"}[5m]))

# Error percentage (0 to 100)
(
  sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{job="api-server"}[5m]))
) * 100

Pattern 3: Latency Percentile (Histogram)

# 95th percentile latency
histogram_quantile(0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{job="api-server"}[5m])
  )
)

Pattern 4: Resource Usage

# Current memory usage
process_resident_memory_bytes{job="api-server"}

# Average CPU usage over 5 minutes
avg_over_time(process_cpu_seconds_total{job="api-server"}[5m])

Pattern 5: Availability

# Percentage of up instances
(
  count(up{job="api-server"} == 1)
  /
  count(up{job="api-server"})
) * 100

Pattern 6: Saturation/Queue Depth

# Average queue length
avg_over_time(queue_depth{job="worker"}[5m])

# Maximum queue depth in the last hour
max_over_time(queue_depth{job="worker"}[1h])

Stage 6: Validate the Generated Query

ALWAYS attempt to validate the generated query first using the devops-skills:promql-validator skill:

After generating the query, automatically invoke:
Skill(devops-skills:promql-validator)

The devops-skills:promql-validator skill will:
1. Check syntax correctness
2. Validate semantic logic (correct functions for metric types)
3. Identify anti-patterns and inefficiencies
4. Suggest optimizations
5. Explain what the query does
6. Verify it matches user intent

Validation checklist:

Syntax is correct (balanced brackets, valid operators)
Metric type matches function usage
Label filters are specific enough
Aggregation is appropriate
Time ranges are reasonable
No known anti-patterns
Query is optimized for performance

If validation fails, fix issues and re-validate until all checks pass.

If the validator skill is unavailable, fails to run, or cannot complete after two fix/re-validate cycles:

Report the validator failure briefly (tool unavailable, timeout, parsing error, etc.).
Run a manual fallback check (syntax shape, metric/function compatibility, label filtering, aggregation, time range sanity).
Mark any unchecked areas as UNVERIFIED and ask the user whether to proceed with best-effort output or provide more context for another validation attempt.

IMPORTANT: Display Validation Results to User

After running validation, you MUST display the structured results to the user in this format:

## PromQL Validation Results

### Syntax Check
- Status: ✅ VALID / ⚠️ WARNING / ❌ ERROR / ⚠️ UNVERIFIED
- Issues: [list any syntax errors]

### Best Practices Check
- Status: ✅ OPTIMIZED / ⚠️ CAN BE IMPROVED / ❌ HAS ISSUES / ⚠️ UNVERIFIED
- Issu