Troubleshoot Elasticsearch OpenTelemetry integration

How to check

$
sudo systemctl status otelcol-contrib

Resolution

• If the service is inactive, start it: sudo systemctl start otelcol-contrib
• If the service failed, fix configuration errors and restart: sudo systemctl restart otelcol-contrib

How to check

$
sudo journalctl -u otelcol-contrib.service -f

Resolution

Review the log output and resolve the root cause (for example, connection problems, authentication failures, or permission issues).

Error sample: dial tcp [::1]:9200: connect: connection refused

Resolution

• Ensure the endpoint in config.yaml matches the Elasticsearch host and port.
• Confirm Elasticsearch is running and reachable from the collector host.

Error sample: permanent error: 403 Forbidden

Resolution

1. Verify NEWRELIC_LICENSE_KEY in /etc/systemd/system/otelcol-contrib.service.d/environment.conf.

2. Reload systemd and restart the collector:

   bash

   Copiar

   $
   sudo systemctl daemon-reload
   $
   sudo systemctl restart otelcol-contrib

Error sample: permission denied or cannot open file

Resolution

1. Add the collector user to the Elasticsearch group:
   bash

   Copiar

   $
   sudo usermod -a -G elasticsearch otelcol-contrib

2. Restart the collector: sudo systemctl restart otelcol-contrib

How to check

$
# Unsecured cluster
$
curl -I http://localhost:9200
$

$
# With authentication
$
curl -u username:password -k https://localhost:9200

Resolution

Verify the cluster is healthy, credentials are valid, and firewall or security settings permit access.

Resolution

• Ensure the resourcedetection processor is included in every metrics pipeline.
• Verify elasticsearch.cluster.name is set via the resource/cluster_name_override processor.

Resolution

• Confirm filelog receiver paths are correct and absolute.
• Check that the logs pipeline includes both the filelog receiver and the otlphttp exporter.

How to check

$
# Verify your Elasticsearch pods have the required label
$
kubectl get pods -n <namespace> -l app=elasticsearch --show-labels

Resolution

If no pods are returned, your Elasticsearch pods are missing the required app=elasticsearch label. The receiver_creator cannot discover pods without matching labels.

• For StatefulSet/Deployment, add the label in the pod template:

  spec:
    template:
      metadata:
        labels:
          app: elasticsearch

  Copiar
• For existing pods, add the label and restart:
  bash

  Copiar

  $
  kubectl label pods -l <your-selector> app=elasticsearch -n <namespace>
  $
  kubectl rollout restart statefulset/elasticsearch -n <namespace>

• If using custom labels, update the receiver rule in values.yaml to match your labels:

  rule: type == "pod" && labels["app"] == "your-custom-label"

  Copiar

How to check

$
kubectl get pods -n newrelic
$
kubectl describe pod <collector-pod-name> -n newrelic

Resolution

• Check pod events for errors: kubectl describe pod

• Review collector logs:

  bash

  Copiar

  $
  kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector

• Verify the secret exists:

  bash

  Copiar

  $
  kubectl get secret newrelic-licenses -n newrelic

• Check resource limits aren't too low

How to check

$
# Check collector logs for discovery errors
$
kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep "receiver_creator"

Resolution

• Verify RBAC permissions are correctly set:

  bash

  Copiar

  $
  kubectl get clusterrole | grep opentelemetry
  $
  kubectl describe clusterrole <role-name>

• Ensure the collector has permissions to watch pods, nodes, and endpoints

• Check that k8s_observer extension is enabled in the config

How to check

$
# Check network policies
$
kubectl get networkpolicies -n <namespace>
$

$
# Test connectivity from collector to Elasticsearch
$
kubectl exec -n newrelic <collector-pod> -- curl http://<es-pod-ip>:9200

Resolution

• Verify network policies allow traffic from the newrelic namespace to your Elasticsearch namespace

• Check if Elasticsearch pods are exposing the correct port (default: 9200)

• Ensure no firewall rules block inter-pod communication

Error sample: permanent error: 403 Forbidden

Resolution

• Verify the secret contains the correct license key:

  bash

  Copiar

  $
  kubectl get secret newrelic-licenses -n newrelic -o jsonpath='{.data.NEWRELIC_LICENSE_KEY}' | base64 -d

• Ensure the OTLP endpoint is correct for your region

• Check that the secret is mounted in the collector pod:

  bash

  Copiar

  $
  kubectl describe pod <collector-pod> -n newrelic | grep -A5 "Environment"

Resolution

• Verify you're using mode: daemonset (deployment mode cannot access node logs)

• Check volume mounts are correctly configured:

  bash

  Copiar

  $
  kubectl describe pod <collector-pod> -n newrelic | grep -A10 "Mounts"

• Verify the filelog receiver path matches your Elasticsearch pod logs:

  bash

  Copiar

  $
  kubectl exec -n newrelic <collector-pod> -- ls /var/log/pods/*/elasticsearch*/*.log

• Ensure the collector has read permissions on host log directories

Resolution

• Verify K8S_CLUSTER_NAME environment variable is set in values.yaml

• Check the resource/cluster processor is in the metrics pipeline

• Query to verify:

  FROM Metric SELECT * WHERE metricName LIKE 'elasticsearch.%' LIMIT 1

  Copiar

  Check if k8s.cluster.name attribute is present

How to check

FROM Span SELECT count(*) WHERE es.cluster.name = '<elasticsearch-cluster-name>' SINCE 30 minutes ago

Resolution

• Native OTLP trace export requires Elasticsearch 9.4 or later. Confirm your version with curl http://localhost:9200.
• Verify all three settings are present in elasticsearch.yml: telemetry.tracing.enabled: true, telemetry.export.endpoint, and telemetry.tracing.sample_rate.
• Confirm the JVM system property -Dtelemetry.otel.traces.enabled=true is set (in jvm.options or ES_JAVA_OPTS), then restart Elasticsearch.
• The default telemetry.tracing.sample_rate is 0.001 (0.1%). Raise it while validating so spans appear quickly.

Resolution

• Confirm the collector has an otlp receiver with the gRPC protocol enabled on 0.0.0.0:4317, and that the endpoint is reachable from the Elasticsearch nodes.
• Confirm a traces pipeline exists and includes the otlp receiver and the otlphttp exporter.
• Check the collector logs for OTLP receiver or export errors.

Cause

A self-loop appears when Elasticsearch spans reach New Relic that can't be attributed to a calling application. There are two common sources:

• Parentless (root) Elasticsearch spans — spans with no parent application span, such as Kubernetes health probes or the Elasticsearch metrics receiver's own scrape calls (_nodes/stats, _cluster/health, and similar). Because they aren't part of an application's trace, New Relic has no calling service and draws the relationship on the cluster itself.

• The node's own address — Elasticsearch server spans carry http.request.headers.host (and server.address), which New Relic resolves back to the cluster.

  Resolution

  Add both processors to the traces pipeline, then restart the collector. filter/drop_rootless_es drops the parentless Elasticsearch spans (application-driven spans always have a parent, so they're kept), and transform/strip_es_host removes the address attributes:

  processors:
    filter/drop_rootless_es:
      error_mode: ignore
      traces:
        span:
          - 'instrumentation_scope.name == "elasticsearch" and IsRootSpan()'
    transform/strip_es_host:
      error_mode: ignore
      trace_statements:
        - context: span
          statements:
            - delete_key(attributes, "http.request.headers.host") where instrumentation_scope.name == "elasticsearch"
            - delete_key(attributes, "server.address") where instrumentation_scope.name == "elasticsearch"
  # traces pipeline: processors: [filter/drop_rootless_es, transform/strip_es_host, batch]

  Copiar

  Existing self-loops clear on their own once the corrected spans arrive and the previous relationship expires (this can take up to about 75 minutes).

Resolution

• Make sure your applications are instrumented and propagate the W3C traceparent header on their Elasticsearch calls. Most modern Elasticsearch clients do this automatically when the application is instrumented with OpenTelemetry.
• Open a distributed trace that includes an Elasticsearch call and confirm the application spans and Elasticsearch spans share the same trace.
• Give the relationship a few minutes to build. New Relic re-evaluates entity relationships periodically.

Resolution

• The traces path keys on es.cluster.name (stamped by Elasticsearch on its spans) and the metrics path on elasticsearch.cluster.name. The elasticsearchreceiver reports elasticsearch.cluster.name automatically from the value Elasticsearch returns, so both derive from the same cluster.name and resolve to one entity — no extra collector configuration needed.
• Don't force the metrics cluster name to a fixed value. If your collector config sets it with a resource processor (for example a resource/cluster_name_override that hardcodes the name), remove that override — otherwise metrics and traces resolve to two different entities and the correlated entity shows no metrics data.
• Make sure your Elasticsearch cluster.name is unique within your New Relic account so it maps to a single, unambiguous entity.