LogQL Best Practices

This document outlines best practices for writing efficient, maintainable, and performant LogQL queries in Grafana Loki.

Query Structure and Performance

1. Use Specific Stream Selectors

Always use the most specific label selectors possible to reduce the number of streams Loki needs to search.

Good:

{namespace="production", app="api-server", environment="prod"}

Bad:

{namespace="production"}  # Too broad, searches many streams

Why: Loki indexes logs by label combinations (streams). More specific selectors mean fewer streams to search, resulting in faster queries.

2. Order Operations Efficiently

Apply filters in the most efficient order: stream selector → line filters → parser → label filters → aggregations.

Good:

{job="nginx"} |= "error" | json | status_code >= 500 | sum(count_over_time([5m]))

Bad:

{job="nginx"} | json | status_code >= 500 |= "error"  # Parse before line filter

Why: Line filters are fast and work on raw log lines. Parsers are more expensive. Apply cheap operations first to reduce data early.

3. Use Line Filters Before Parsing

Filter out irrelevant log lines before parsing to reduce computational overhead.

Good:

{app="api"} |= "error" | json | level="error"

Bad:

{app="api"} | json | level="error"  # Parses all logs, not just errors

Why: Line filters (|=, !=, |, !) are extremely fast string operations. Parsing (json, logfmt, regexp) is more expensive.

4. Avoid Complex Regex When Simple Matching Works

Use exact string matching when possible instead of regex.

Good:

{job="app"} |= "ERROR:"  # Fast string match

Bad:

{job="app"} |~ "ERROR:"  # Slower regex match for simple string

Why: Regex matching requires compilation and more complex pattern matching. Simple string contains is significantly faster.

5. Use Appropriate Time Ranges

Use the shortest time range that satisfies your requirements.

Good:

rate({app="api"}[1m])  # For real-time dashboards
rate({app="api"}[1h])  # For trend analysis

Bad:

rate({app="api"}[24h])  # Unnecessarily long for real-time monitoring

Why: Larger time ranges mean more data to process. Match the range to your use case.

Label Management

6. Understand Label vs Line Filter Trade-offs

Use labels for indexed dimensions, line filters for unique values.

Good (using line filter for unique ID):

{app="api"} |= "trace_id=abc123"

Bad (would create high cardinality if trace_id was a label):

{app="api", trace_id="abc123"}  # Don't do this!

Why: Labels create separate streams and indexes. High cardinality labels (user IDs, trace IDs, session IDs) create too many streams, degrading performance.

7. Keep Cardinality Low

Avoid using high-cardinality data as labels in stream selectors.

High cardinality fields (use line filters instead):

• user_id
• trace_id
• request_id
• session_id
• ip_address (individual IPs)
• timestamp

Good cardinality fields (suitable for labels):

• namespace
• app
• environment
• cluster
• level (error, warn, info)
• pod (in moderation)
• job
• host (in moderation)

Why: Each unique combination of labels creates a new stream. Too many streams overwhelm Loki's indexing.

8. Use Label Operations Wisely

Drop unnecessary labels to reduce series cardinality in metric queries.

Good:

{app="api"} | json | drop instance, pod | sum by (namespace, app) (rate([5m]))

Why: Fewer labels in results = fewer time series = better performance and lower memory usage.

Parsing Best Practices

9. Choose the Right Parser

Use the most appropriate parser for your log format.

Performance order (fastest to slowest): pattern > logfmt > json > regexp

Why this order matters:

• pattern: Simple string matching with placeholders, fastest execution
• logfmt: Optimized key=value parsing, very efficient
• json: Full JSON parsing, moderate overhead
• regexp: Regex compilation and matching, slowest but most flexible

Why: Simpler parsers are faster. JSON and logfmt are optimized. Pattern is faster than regex for simple cases.

9a. Use logfmt Parser Flags When Needed

The logfmt parser supports optional flags for handling edge cases:

--strict flag:

# Fail on malformed key=value pairs (stops scanning on error)
{app="api"} | logfmt --strict

# Use when you need to detect malformed log entries
{app="api"} | logfmt --strict | __error__ != ""

--keep-empty flag:

# Retain standalone keys as labels with empty string value
{app="api"} | logfmt --keep-empty

# Combine flags
{app="api"} | logfmt --strict --keep-empty

When to use:

• --strict: When log quality matters and you want to detect malformed entries
• --keep-empty: When logs have standalone keys (no values) that need to be preserved

Why: By default, logfmt is non-strict (skips invalid tokens) which is more lenient but may hide log quality issues.

9b. Use JSON Parser Parameter Extraction for Performance

Extract only the fields you need instead of parsing entire JSON:

Good (extract specific fields):

{app="api"} | json status="response.code", method="request.method"

Less efficient (parse all fields):

{app="api"} | json

Supported access patterns:

• Dot notation: | json method="request.method"
• Bracket notation: | json ua="headers[\\"User-Agent\\"]"
• Array access: | json first="items[0]"
• Combined: | json item="data.items[0].name"

Why: Extracting fewer fields reduces parsing overhead and memory usage.

10. Parse Only What You Need

If you only need specific fields, extract just those fields.

Good:

{app="api"} | json level, message, status_code

Better than:

{app="api"} | json  # Parses all fields

Why: Extracting fewer fields reduces parsing overhead and memory usage.

11. Use Pattern Parser for Simple Cases

Pattern parser is faster than regex for straightforward field extraction.

Good:

{job="nginx"} | pattern "<ip> - - [<timestamp>] \\"<method> <path> <_>\\" <status>"

Avoid (unless necessary):

{job="nginx"} | regexp "(?P<ip>\\\\S+) .* (?P<method>\\\\w+) (?P<path>\\\\S+).*"

Why: Pattern parser is simpler and faster for structured formats.

Aggregation Best Practices

12. Use Appropriate Aggregation Functions

Choose the right function for your metric type.

13. Aggregate Early and Often

Reduce data volume as early as possible.

Good:

sum by (namespace) (
  count_over_time({app="api"} | json | level="error" [5m])
)

Why: Aggregating reduces the number of time series, improving query performance.

14. Use by Instead of without When Possible

Explicitly specify labels to keep rather than labels to remove.

Good:

sum by (namespace, app) (rate({job="kubernetes-pods"}[5m]))

Less efficient:

sum without (pod, instance, node) (rate({job="kubernetes-pods"}[5m]))

Why: by is more explicit and often results in fewer output series.

Query Optimization

15. Avoid Expensive Operations in Inner Loops

Don't use regex or complex parsing inside frequently-evaluated contexts.

Good:

sum(rate({app="api"} |= "error" [5m]))  # Filter first

Bad:

sum(rate({app="api"} | regexp "complex.*pattern" [5m]))  # Regex on every line

16. Use Metric Queries for Dashboards

For dashboard panels, use metric queries (aggregations) rather than log queries.

Good (for time series panel):

rate({app="api"}[5m])

Bad (for time series panel):

{app="api"}  # Returns log lines, not metrics

Why: Metric queries return time series data suitable for graphing.

17. Limit Log Query Results

When querying for log lines (not metrics), limit the result set.

Important: The limit is an API parameter, not a LogQL pipeline operator. Set it via:

• API: /loki/api/v1/query_range?query={...}&limit=100
• Grafana UI: "Line limit" field in the query editor (default: 1000)
• logcli: --limit=100 flag

Good:

# Using logcli
logcli query '{app="api"} | json | level="error"' --limit=100

# Using API
curl -G "http://localhost:3100/loki/api/v1/query_range" \\
  --data-urlencode 'query={app="api"} | json | level="error"' \\
  --data-urlencode 'limit=100'

Why: Returning thousands of log lines is slow and resource-intensive. Always set appropriate limits for log queries.

18. Use __error__="" to Filter Parse Errors

When parsing, filter out lines that fail to parse to get clean results.

Good:

{app="api"} | json | __error__="" | level="error"

Why: Parse errors create __error__ labels. Filtering them out gives you only successfully parsed logs.

Alerting Best Practices

19. Use Metric Queries for Alerts

Alerts require numeric values. Always use metric queries (aggregations).

Good:

sum(rate({app="api"} | json | level="error" [5m])) > 10

Bad:

{app="api"} | json | level="error"  # Returns logs, not metrics

20. Include Meaningful Thresholds

Set explicit, meaningful thresholds for alerting.

Good:

(
  sum(rate({app="api"} | json | level="error" [5m]))
  /
  sum(rate({app="api"}[5m]))
) > 0.05  # Alert if error rate > 5%

Why: Thresholds should be based on SLOs or historical baselines.

21. Use absent_over_time for Missing Logs

Detect when logs stop coming (potential service outage).

Good:

absent_over_time({app="critical-service"}[5m])

Why: This returns 1 when no logs match in the time range, indicating a potential problem.

Security and Sensitive Data

22. Don't Log Sensitive Information

Avoid logging sensitive data that could appear in LogQL query results.

Avoid in logs:

• Passwords
• API keys
• Tokens
• Credit card numbers
• PII (personally identifiable information)

If you must log sensitive data:

• Use structured metadata (not indexed)
• Redact before ingestion
• Use Loki's data retention policies
• Restrict access with Loki's multi-tenancy

23. Use Structured Metadata for High-Cardinality Data

Store high-cardinality data as structured metadata, not labels.

Good:

# In your log shipper config
structured_metadata:
  trace_id: ${TRACE_ID}
  user_id: ${USER_ID}

Then query:

{app="api"} | trace_id="abc123"

Why: Structured metadata is not indexed, avoiding cardinality issues.

Maintenance and Debugging

24. Test Queries Incrementally

Build complex queries step by step, testing each stage.

Approach:

# Step 1: Test stream selector
{app="api"}

# Step 2: Add line filter
{app="api"} |= "error"

# Step 3: Add parser
{app="api"} |= "error" | json

# Step 4: Add label filter
{app="api"} |= "error" | json | status_code >= 500

# Step 5: Add aggregation
sum(count_over_time({app="api"} |= "error" | json | status_code >= 500 [5m]))

Why: Incremental testing helps identify issues early and understand query behavior.

25. Use line_format for Debugging

Format log output to see extracted fields during development.

Debugging query:

{app="api"} | json | line_format "level={{.level}} status={{.status_code}} message={{.message}}"

Why: Makes it easy to see what fields were extracted and their values.

26. Comment Complex Queries

Use LogQL comments to document complex queries.

Good:

# Calculate 5xx error rate as percentage
# Alerts when > 5% for SLO compliance
(
  sum(rate({app="api"} | json | status_code >= 500 [5m]))
  /
  sum(rate({app="api"}[5m]))
) * 100 > 5

Why: Comments help team members understand query intent and logic.

Performance Tuning

27. Use Query Splitting for Large Time Ranges

For very large time ranges, consider splitting queries or using downsampling.

Instead of:

sum(count_over_time({app="api"}[30d]))  # Very expensive

Consider:

• Using Loki's query splitting (automatic in recent versions)
• Using recording rules for frequently-queried metrics
• Adjusting retention policies

28. Leverage Loki's Query Parallelization

Recent Loki versions automatically parallelize queries. Structure queries to take advantage:

Good (parallelizable):

sum by (namespace) (rate({job="kubernetes-pods"}[5m]))

Why: Loki can process different namespaces in parallel.

29. Use Appropriate Step Sizes

For metric queries over long time ranges, use appropriate step sizes.

Good:

# For 24h dashboard, use 1m step
rate({app="api"}[5m])  # With 1m step in Grafana

# For 7d dashboard, use 5m or 15m step
rate({app="api"}[15m])  # With 5m step

Why: Smaller steps = more data points = slower queries. Match resolution to your needs.

Structured Metadata (Loki 3.x)

35. Use Structured Metadata for High-Cardinality Data

Structured metadata is metadata attached to logs without indexing. Introduced in Loki 3.0.

What it is:

• Metadata attached to logs that is NOT indexed
• Ideal for high-cardinality data (trace_id, user_id, request_id, pod names)
• Avoids index bloat and cardinality explosion
• Automatically extracted as labels in query results

Key differences from labels:

• Labels are indexed → fast stream selection, but high cardinality is expensive
• Structured metadata is NOT indexed → no cardinality impact, but requires scanning

Query syntax:

# Filter by structured metadata (AFTER stream selector, not inside it!)
{app="api"} | trace_id="abc123"

# Combine multiple structured metadata filters
{app="api"} | trace_id="abc123" | user_id="user456"

# Use with other filters
{app="api"} | trace_id="abc123" | json | level="error"

WRONG (structured metadata is not a label):

{app="api", trace_id="abc123"}