Troubleshoot Elasticsearch OpenTelemetry integration

How to check

$
sudo systemctl status otelcol-contrib

Resolution

• If the service is inactive, start it: sudo systemctl start otelcol-contrib
• If the service failed, fix configuration errors and restart: sudo systemctl restart otelcol-contrib

How to check

$
sudo journalctl -u otelcol-contrib.service -f

Resolution

Review the log output and resolve the root cause (for example, connection problems, authentication failures, or permission issues).

Error sample: dial tcp [::1]:9200: connect: connection refused

Resolution

• Ensure the endpoint in config.yaml matches the Elasticsearch host and port.
• Confirm Elasticsearch is running and reachable from the collector host.

Error sample: permanent error: 403 Forbidden

Resolution

1. Verify NEWRELIC_LICENSE_KEY in /etc/systemd/system/otelcol-contrib.service.d/environment.conf.

2. Reload systemd and restart the collector:

   bash

   コピー

   $
   sudo systemctl daemon-reload
   $
   sudo systemctl restart otelcol-contrib

Error sample: permission denied or cannot open file

Resolution

1. Add the collector user to the Elasticsearch group:
   bash

   コピー

   $
   sudo usermod -a -G elasticsearch otelcol-contrib

2. Restart the collector: sudo systemctl restart otelcol-contrib

How to check

$
# Unsecured cluster
$
curl -I http://localhost:9200
$

$
# With authentication
$
curl -u username:password -k https://localhost:9200

Resolution

Verify the cluster is healthy, credentials are valid, and firewall or security settings permit access.

Resolution

• Ensure the resourcedetection processor is included in every metrics pipeline.
• Verify elasticsearch.cluster.name is set via the resource/cluster_name_override processor.

Resolution

• Confirm filelog receiver paths are correct and absolute.
• Check that the logs pipeline includes both the filelog receiver and the otlphttp exporter.

How to check

$
# Verify your Elasticsearch pods have the required label
$
kubectl get pods -n <namespace> -l app=elasticsearch --show-labels

Resolution

If no pods are returned, your Elasticsearch pods are missing the required app=elasticsearch label. The receiver_creator cannot discover pods without matching labels.

• For StatefulSet/Deployment, add the label in the pod template:

  spec:
    template:
      metadata:
        labels:
          app: elasticsearch

  コピー

• For existing pods, add the label and restart:

  bash

  コピー

  $
  kubectl label pods -l <your-selector> app=elasticsearch -n <namespace>
  $
  kubectl rollout restart statefulset/elasticsearch -n <namespace>

• If using custom labels, update the receiver rule in values.yaml to match your labels:

  rule: type == "pod" && labels["app"] == "your-custom-label"

  コピー

How to check

$
kubectl get pods -n newrelic
$
kubectl describe pod <collector-pod-name> -n newrelic

Resolution

• Check pod events for errors: kubectl describe pod

• Review collector logs:

  bash

  コピー

  $
  kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector

• Verify the secret exists:

  bash

  コピー

  $
  kubectl get secret newrelic-licenses -n newrelic

• Check resource limits aren't too low

How to check

$
# Check collector logs for discovery errors
$
kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep "receiver_creator"

Resolution

• Verify RBAC permissions are correctly set:

  bash

  コピー

  $
  kubectl get clusterrole | grep opentelemetry
  $
  kubectl describe clusterrole <role-name>

• Ensure the collector has permissions to watch pods, nodes, and endpoints

• Check that k8s_observer extension is enabled in the config

How to check

$
# Check network policies
$
kubectl get networkpolicies -n <namespace>
$

$
# Test connectivity from collector to Elasticsearch
$
kubectl exec -n newrelic <collector-pod> -- curl http://<es-pod-ip>:9200

Resolution

• Verify network policies allow traffic from the newrelic namespace to your Elasticsearch namespace

• Check if Elasticsearch pods are exposing the correct port (default: 9200)

• Ensure no firewall rules block inter-pod communication

Error sample: permanent error: 403 Forbidden

Resolution

• Verify the secret contains the correct license key:

  bash

  コピー

  $
  kubectl get secret newrelic-licenses -n newrelic -o jsonpath='{.data.NEWRELIC_LICENSE_KEY}' | base64 -d

• Ensure the OTLP endpoint is correct for your region

• Check that the secret is mounted in the collector pod:

  bash

  コピー

  $
  kubectl describe pod <collector-pod> -n newrelic | grep -A5 "Environment"

Resolution

• Verify you're using mode: daemonset (deployment mode cannot access node logs)

• Check volume mounts are correctly configured:

  bash

  コピー

  $
  kubectl describe pod <collector-pod> -n newrelic | grep -A10 "Mounts"

• Verify the filelog receiver path matches your Elasticsearch pod logs:

  bash

  コピー

  $
  kubectl exec -n newrelic <collector-pod> -- ls /var/log/pods/*/elasticsearch*/*.log

• Ensure the collector has read permissions on host log directories

Resolution

• Verify K8S_CLUSTER_NAME environment variable is set in values.yaml

• Check the resource/cluster processor is in the metrics pipeline

• Query to verify:

  FROM Metric SELECT * WHERE metricName LIKE 'elasticsearch.%' LIMIT 1

  コピー

  Check if k8s.cluster.name attribute is present