Loki Configuration Best Practices

This document outlines best practices for configuring and deploying Grafana Loki in production environments.

Important Notice (Loki 3.4+): Promtail has been deprecated and its code merged into Grafana Alloy. For new log collection deployments, use Grafana Alloy instead of Promtail.

Schema Configuration

Use TSDB with v13 Schema (CRITICAL)

Always use the latest schema for new deployments:

schema_config:
  configs:
    - from: "2025-01-01"  # Use deployment date
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: loki_index_
        period: 24h

Why:

• TSDB is the modern, performant index store
• v13 schema provides best performance and features
• Cannot be changed after deployment without migration
• Daily period (24h) is recommended for most use cases

Important: Set from date to your deployment date, not a past date.

Deployment Modes

Choose the Right Deployment Mode

Monolithic:

• Single binary with all components
• Easy to operate
• Limited scalability
• Good for getting started

Simple Scalable:

• Separates read, write, and backend
• Horizontal scaling
• Production-ready
• Recommended for most use cases

Microservices:

• Full component separation
• Maximum scalability
• Independent scaling per component
• Requires more operational overhead

Storage Configuration

Storage Backend Selection

Filesystem:

• Development and testing only
• Requires persistent volumes
• Not recommended for production at scale

Object Storage (S3, GCS, Azure):

• Recommended for production
• Cost-effective at scale
• Durable and highly available
• Use IAM roles/service accounts for authentication

Best practices:

common:
  storage:
    s3:
      s3: s3://region/bucket-name
      s3forcepathstyle: false
      # Use IAM roles instead of access keys
  replication_factor: 3  # Always use 3 for production

Replication and High Availability

Always Use Replication Factor 3

common:
  replication_factor: 3

Why:

• Data durability: tolerates 2 node failures
• Query reliability: ensures data availability
• Industry standard for distributed systems

Enable Zone-Aware Replication

For multi-AZ deployments:

ingester:
  lifecycler:
    ring:
      zone_awareness_enabled: true

Why:

• Distributes replicas across availability zones
• Survives entire AZ failures
• Better fault tolerance

Native OTLP Ingestion (Loki 3.0+)

Configure OTLP Attributes

If using OpenTelemetry, configure how OTLP attributes are mapped:

limits_config:
  allow_structured_metadata: true

  otlp_config:
    resource_attributes:
      ignore_defaults: false  # Set true to completely override defaults
      attributes_config:
        - action: index_label
          attributes:
            - service.name
            - service.namespace
            - deployment.environment
            # NOTE: Do NOT include high-cardinality attributes as index labels!
        - action: structured_metadata
          attributes:
            - k8s.pod.name           # High cardinality - use structured_metadata
            - service.instance.id    # High cardinality - use structured_metadata
    log_attributes:
      - action: structured_metadata
        attributes:
          - trace_id
          - span_id

⚠️ CRITICAL: Label Cardinality Best Practices (Updated 2025)

DO NOT use these high-cardinality attributes as index labels:

• k8s.pod.name - Changes frequently, creates too many streams
• service.instance.id - High cardinality

Instead, store them as structured_metadata. This is now the recommended approach. See: https://grafana.com/docs/loki/latest/get-started/labels/remove-default-labels/

Recommended index labels (low-cardinality):

• service.name, service.namespace, deployment.environment
• cloud.region, cloud.availability_zone
• k8s.cluster.name, k8s.namespace.name, k8s.container.name
• k8s.deployment.name, k8s.statefulset.name, k8s.daemonset.name

Configuring Default Resource Attributes:

For more control over which OTLP resource attributes become labels:

distributor:
  otlp_config:
    default_resource_attributes_as_index_labels:
      - service.name
      - service.namespace
      - deployment.environment
      - k8s.cluster.name
      - k8s.namespace.name
      # EXCLUDES: k8s.pod.name, service.instance.id

Why:

• Native OTLP support eliminates the need for Loki Exporter (deprecated)
• Control which attributes become labels vs structured metadata
• Low-cardinality attributes should be index_label
• High-cardinality attributes should be structured_metadata
• Use ignore_defaults: true for complete control over attribute mapping

OTLP Endpoint: POST /otlp/v1/logs

OpenTelemetry Collector Configuration:

exporters:
  otlphttp:
    endpoint: http://loki:3100/otlp
    # Note: lokiexporter is DEPRECATED - use otlphttp instead

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Pattern Ingester (Loki 3.0+)

Enable Pattern Detection

pattern_ingester:
  enabled: true

Why:

• Automatic log pattern detection
• Powers Explore Logs / Grafana Drilldown features
• Identifies recurring patterns for anomaly detection
• Minimal resource overhead

Caching Configuration

Configure Memcached for Production

# Chunk cache
chunk_store_config:
  chunk_cache_config:
    memcached:
      batch_size: 256
      parallelism: 10
    memcached_client:
      host: memcached-chunks.loki.svc.cluster.local
      service: memcached-client
      timeout: 500ms

# Results cache
query_range:
  cache_results: true
  results_cache:
    cache:
      memcached_client:
        host: memcached-results.loki.svc.cluster.local
        timeout: 500ms

Important Notes:

• TSDB does NOT need index cache - only chunks and results cache
• Use separate Memcached instances for chunks and results
• Size chunk cache based on query hot data volume
• Size results cache based on repeated query patterns

Helm Chart Caching:

memcached:
  chunk_cache:
    enabled: true
  results_cache:
    enabled: true

memcachedChunks:
  enabled: true
  replicas: 2
  resources:
    requests:
      memory: 1Gi
    limits:
      memory: 2Gi

Limits Configuration

Set Appropriate Ingestion Limits

limits_config:
  ingestion_rate_mb: 50  # Adjust based on expected load
  ingestion_burst_size_mb: 100  # 2x rate for bursts
  max_line_size: 256KB
  max_line_size_truncate: true

Why:

• Prevents resource exhaustion
• Protects against misconfigured clients
• Allows burst traffic while limiting sustained overload

Control Stream Cardinality

limits_config:
  max_streams_per_user: 10000
  max_global_streams_per_user: 100000

Why:

• High cardinality kills performance
• Each label combination creates a stream
• Limit prevents accidental label explosion

Best practice: Use line filters for high-cardinality data (user IDs, trace IDs) instead of labels.

Configure Retention

compactor:
  retention_enabled: true
  retention_delete_delay: 2h

limits_config:
  retention_period: 30d  # Adjust based on requirements

Why:

• Controls storage costs
• Meets compliance requirements
• Automatic cleanup of old data

Chunk Management

Optimize Chunk Settings

ingester:
  chunk_encoding: snappy
  chunk_target_size: 1572864  # 1.5MB
  chunk_idle_period: 30m
  max_chunk_age: 2h

Why:

• snappy: Best balance of speed vs compression
• 1.5MB target: Optimal chunk size (requires 5-10x raw data)
• 30m idle: Flushes inactive chunks to storage
• 2h max age: Prevents memory buildup

Important: More streams = more chunks in memory. Keep stream cardinality low.

Query Performance

Configure Query Concurrency

querier:
  max_concurrent: 4  # Per querier instance
  query_timeout: 5m

Recommendations:

• Start with 4 concurrent queries
• Increase based on CPU/memory resources
• Monitor query latency and adjust

Enable Query Parallelization

query_range:
  parallelise_shardable_queries: true
  split_queries_by_interval: 15m  # For large time ranges

Why:

• Distributes query load across queriers
• Faster results for large time ranges
• Better resource utilization

Security

Enable Multi-Tenancy

auth_enabled: true

Production recommendation:

• Always use auth_enabled: true
• Deploy authenticating reverse proxy (nginx, Envoy)
• Enforce X-Scope-OrgID header
• Isolate tenant data

Use TLS for Inter-Component Communication

server:
  http_tls_config:
    cert_file: /path/to/cert.pem
    key_file: /path/to/key.pem
  grpc_tls_config:
    cert_file: /path/to/cert.pem
    key_file: /path/to/key.pem

Why:

• Encrypts data in transit
• Prevents eavesdropping
• Required for compliance (PCI, HIPAA, etc.)

Secure Credentials

Never hardcode credentials:

# BAD
common:
  storage:
    s3:
      access_key_id: AKIAIOSFODNN7EXAMPLE
      secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# GOOD
common:
  storage:
    s3:
      # Uses IAM role automatically

Best practices:

• Use IAM roles for AWS
• Use service accounts for GCP
• Use managed identities for Azure
• Store secrets in Kubernetes Secrets or Vault
• Reference secrets via environment variables

Monitoring and Observability

Enable Metrics

Loki exports Prometheus metrics automatically. Scrape them:

# In Prometheus config
- job_name: 'loki'
  static_configs:
    - targets: ['loki:3100']

Key metrics to monitor:

• loki_ingester_chunks_flushed_total: Chunk flush rate
• loki_ingester_memory_streams: Active streams (watch for growth)
• loki_request_duration_seconds: Query latency
• loki_distributor_ingester_append_failures_total: Ingestion failures
• loki_boltdb_shipper_request_duration_seconds: Index query time

Set Up Alerts

Critical alerts:

# High ingestion failure rate
- alert: LokiIngestionFailureRate
  expr: sum(rate(loki_distributor_ingester_append_failures_total[5m])) > 10

# Too many streams (cardinality explosion)
- alert: LokiHighStreamCardinality
  expr: loki_ingester_memory_streams > 100000

# Compaction not running
- alert: LokiCompactionNotRunning
  expr: time() - loki_boltdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds > 3600

Resource Planning

Ingester Resources

Memory requirements:

• Base: ~1GB per ingester
• Add: 1-2KB per active stream
• Add: Chunk buffer (depends on throughput)

Example: 10,000 streams = ~1GB + 20MB = ~1.2GB minimum

Kubernetes recommendations:

resources:
  requests:
    memory: "4Gi"
    cpu: "1"
  limits:
    memory: "8Gi"
    cpu: "2"

Querier Resources

Memory requirements:

• Base: ~500MB per querier
• Add: Depends on query complexity and concurrency

CPU requirements:

• Varies with query load
• More CPU = faster queries

Kubernetes recommendations:

resources:
  requests:
    memory: "2Gi"
    cpu: "1"
  limits:
    memory: "4Gi"
    cpu: "2"

Storage Requirements

Estimate storage:

Daily storage = (ingestion rate MB/s) × 86400 seconds × compression ratio

Compression ratios:

• Text logs: 5-10x (snappy)
• JSON logs: 3-7x (snappy)
• Structured logs: 2-5x (snappy)

Example: 10 MB/s ingestion with 5x compression:

10 MB/s × 86400 × 0.2 = ~170 GB/day

Operational Best Practices

Use Health Checks

Configure Kubernetes probes:

livenessProbe:
  httpGet:
    path: /ready
    port: 3100
  initialDelaySeconds: 45

readinessProbe:
  httpGet:
    path: /ready
    port: 3100
  initialDelaySeconds: 45

Enable Graceful Shutdown

server:
  graceful_shutdown_timeout: 30s

Why:

• Allows in-flight requests to complete
• Prevents data loss during restarts
• Smooth rolling updates

Use Configuration Management

Best practices:

• Store configs in Git
• Use configuration as code (Terraform, Helm)
• Validate configs before applying
• Test in staging before production
• Document all customizations

Regular Maintenance

Weekly:

• Review metrics and alerts
• Check for errors in logs
• Verify compaction is running

Monthly:

• Review and adjust limits based on actual usage
• Analyze storage growth trends
• Update Loki to latest stable version

Quarterly:

• Review architecture for scale
• Optimize queries and cardinality
• Conduct disaster recovery tests

Common Anti-Patterns

Don't Use High-Cardinality Labels

BAD:

# Don't use user_id, trace_id, request_id as labels
{app="api", user_id="12345"}  # Creates too many streams

GOOD:

# Use structured metadata or line filters instead
{app="api"} | json | user_id="12345"

Don't Ignore Limits

BAD:

limits_config:
  max_streams_per_user: 0  # Unlimited - dangerous!

GOOD:

limits_config:
  max_streams_per_user: 10000  # Reasonable limit

Don't Skip Replication

BAD:

common:
  replication_factor: 1  # Single copy - data loss risk

GOOD:

common:
  replication_factor: 3  # Durability and availability

Don't Use Filesystem Storage in Production

BAD:

common:
  storage:
    filesystem:
      chunks_directory: /loki/chunks  # Not scalable

GOOD:

common:
  storage:
    s3:
      s3: s3://region/bucket  # Scalable and durable

Don't Disable Authentication in Multi-Tenant Environments

BAD:

auth_enabled: false  # No tenant isolation

GOOD:

auth_enabled: true  # Proper tenant isolation

Configuration Validation

Before Deployment

1. Validate syntax:

   loki -config.file=loki.yaml -verify-config
   

2. Review configuration:

   loki -config.file=loki.yaml -print-config-stderr
   

3. Test ingestion: Send test logs and verify they appear

4. Test queries: Run sample LogQL queries

After Deployment

1. Check health:

   curl http://loki:3100/ready
   

2. Monitor metrics: Review Prometheus metrics

3. Verify data ingestion: Check ingester and distributor logs

4. Test query performance: Run representative queries

Troubleshooting Guide

High Memory Usage

Symptoms:

• OOMKilled pods