Monitorear Elasticsearch autohospedado con OpenTelemetry

Monitoree su clúster de Elasticsearch autohospedado instalando el OpenTelemetry Collector directamente en servidores o máquinas virtuales. New Relic ofrece opciones de implementación flexibles para adaptarse a la configuración de su infraestructura y a sus requisitos de monitoreo.

You can choose between three collector options:

NRDOT: Distribución de New Relic de OpenTelemetry
OTel Collector Contrib: OpenTelemetry Collector estándar con componentes aportados por la comunidad
Prometheus Receiver: For environments already running a Prometheus Elasticsearch exporter

Opciones de instalación

Seleccione la distribución del recopilador que se ajuste a sus necesidades:

Antes de que empieces

Before configuring Elasticsearch monitoring with NRDOT, ensure you have:

NRDOT collector installed:

You must have NRDOT collector installed and running as a systemd service on your host
If not installed, follow the official NRDOT installation guide to install the collector using the package manager (DEB/RPM) which sets up the systemd service automatically
Privilegios de acceso requeridos:
Tu New Relic
Privilegios de root o sudo en la máquina host
Privilegios de administrador de clúster de Elasticsearch con privilegio de clúster monitor o manage (consulte la documentación de privilegios de seguridad de Elasticsearch para obtener detalles)
Requisitos del sistema:
Versión 7.16 o superior de Elasticsearch - Esta integración requiere un clúster moderno de Elasticsearch
Conectividad de red - HTTPS saliente (puerto 443) al endpoint de ingesta OTLPde New Relic
Valores de configuración listos:
Punto de conexión de Elasticsearch - La URL de su clúster de Elasticsearch (p. ej., http://localhost:9200)
Nombre del clúster - Un identificador único para su clúster en New Relic

Configurar el monitoreo de Elasticsearch

Once the NRDOT collector is installed, replace the collector's default configuration file with the Elasticsearch monitoring configuration. This will enable Elasticsearch metrics collection. Host metrics and logs are optional and can be added separately.

El archivo de configuración se encuentra en: /etc/nrdot-collector/config.yaml

Sugerencia

Respalde su configuración predeterminada: Antes de modificar el archivo de configuración, cree una copia de seguridad para conservar los ajustes predeterminados:

bash

$sudo cp /etc/nrdot-collector/config.yaml /etc/nrdot-collector/config.yaml.backup

Para configurar el colector:

Abra el archivo de configuración con un editor de texto con privilegios de root o sudo:
bash
```
$sudo nano /etc/nrdot-collector/config.yaml
```

Elimine todo el contenido existente y reemplácelo con la siguiente configuración para el monitoreo de Elasticsearch:

Importante

Reemplace el valor endpoint con el endpoint de su clúster de Elasticsearch y actualice elasticsearch.cluster.name en el bloque del procesador con un nombre único para identificar su clúster en New Relic.

receivers:
  elasticsearch:
    endpoint: "http://localhost:9200"
    collection_interval: 15s
    metrics:
      elasticsearch.os.cpu.usage:
        enabled: true
      elasticsearch.cluster.data_nodes:
        enabled: true
      elasticsearch.cluster.health:
        enabled: true
      elasticsearch.cluster.in_flight_fetch:
        enabled: true
      elasticsearch.cluster.nodes:
        enabled: true
      elasticsearch.cluster.pending_tasks:
        enabled: true
      elasticsearch.cluster.shards:
        enabled: true
      elasticsearch.cluster.state_update.time:
        enabled: true
      elasticsearch.index.documents:
        enabled: true
      elasticsearch.index.operations.merge.current:
        enabled: true
      elasticsearch.index.operations.time:
        enabled: true
      elasticsearch.node.cache.count:
        enabled: true
      elasticsearch.node.cache.evictions:
        enabled: true
      elasticsearch.node.cache.memory.usage:
        enabled: true
      elasticsearch.node.shards.size:
        enabled: true
      elasticsearch.node.cluster.io:
        enabled: true
      elasticsearch.node.documents:
        enabled: true
      elasticsearch.node.disk.io.read:
        enabled: true
      elasticsearch.node.disk.io.write:
        enabled: true
      elasticsearch.node.fs.disk.available:
        enabled: true
      elasticsearch.node.fs.disk.total:
        enabled: true
      elasticsearch.node.http.connections:
        enabled: true
      elasticsearch.node.ingest.documents.current:
        enabled: true
      elasticsearch.node.ingest.operations.failed:
        enabled: true
      elasticsearch.node.open_files:
        enabled: true
      elasticsearch.node.operations.completed:
        enabled: true
      elasticsearch.node.operations.current:
        enabled: true
      elasticsearch.node.operations.get.completed:
        enabled: true
      elasticsearch.node.operations.time:
        enabled: true
      elasticsearch.node.shards.reserved.size:
        enabled: true
      elasticsearch.index.shards.size:
        enabled: true
      elasticsearch.os.cpu.load_avg.1m:
        enabled: true
      elasticsearch.os.cpu.load_avg.5m:
        enabled: true
      elasticsearch.os.cpu.load_avg.15m:
        enabled: true
      elasticsearch.os.memory:
        enabled: true
      jvm.gc.collections.count:
        enabled: true
      jvm.gc.collections.elapsed:
        enabled: true
      jvm.memory.heap.max:
        enabled: true
      jvm.memory.heap.used:
        enabled: true
      jvm.memory.heap.utilization:
        enabled: true
      jvm.threads.count:
        enabled: true
      elasticsearch.index.segments.count:
        enabled: true
      elasticsearch.index.operations.completed:
        enabled: true
      elasticsearch.node.script.cache_evictions:
        enabled: false
      elasticsearch.node.cluster.connections:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.preprocessed:
        enabled: false
      elasticsearch.node.thread_pool.tasks.queued:
        enabled: false
      elasticsearch.cluster.published_states.full:
        enabled: false
      jvm.memory.pool.max:
        enabled: false
      elasticsearch.node.script.compilation_limit_triggered:
        enabled: false
      elasticsearch.node.shards.data_set.size:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.current:
        enabled: false
      elasticsearch.cluster.state_update.count:
        enabled: false
      elasticsearch.node.fs.disk.free:
        enabled: false
      jvm.memory.nonheap.used:
        enabled: false
      jvm.memory.pool.used:
        enabled: false
      elasticsearch.node.translog.size:
        enabled: false
      elasticsearch.node.thread_pool.threads:
        enabled: false
      elasticsearch.cluster.state_queue:
        enabled: false
      elasticsearch.node.translog.operations:
        enabled: false
      elasticsearch.memory.indexing_pressure:
        enabled: false
      elasticsearch.node.ingest.documents:
        enabled: false
      jvm.classes.loaded:
        enabled: false
      jvm.memory.heap.committed:
        enabled: false
      elasticsearch.breaker.memory.limit:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.replica_rejections:
        enabled: false
      elasticsearch.breaker.memory.estimated:
        enabled: false
      elasticsearch.cluster.published_states.differences:
        enabled: false
      jvm.memory.nonheap.committed:
        enabled: false
      elasticsearch.node.translog.uncommitted.size:
        enabled: false
      elasticsearch.node.script.compilations:
        enabled: false
      elasticsearch.node.pipeline.ingest.operations.failed:
        enabled: false
      elasticsearch.indexing_pressure.memory.limit:
        enabled: false
      elasticsearch.breaker.tripped:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.primary_rejections:
        enabled: false
      elasticsearch.node.thread_pool.tasks.finished:
        enabled: false
processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}
  cumulativetodelta: {}
  attributes/cluster_state_aggregate:
    include:
      match_type: strict
      metric_names:
        - elasticsearch.cluster.state_update.time
    actions:
      - key: type
        action: delete
      - key: state
        action: delete
  filter/critical_operations:
    metrics:
      datapoint:
        # Filters to keep only: query, index, get, merge
        # Affects only 4 metrics: *.operations.completed and *.operations.time
        # All other metrics pass through unchanged
        - 'attributes["operation"] == "query" or attributes["operation"] == "index" or attributes["operation"] == "get" or attributes["operation"] == "merge" or attributes["operation"] == nil'
  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert
  resourcedetection:
    detectors: [ system ]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true
  batch:
    timeout: 30s
    send_batch_size: 2048
    send_batch_max_size: 4096
  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete
  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")
exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
    compression: gzip
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, attributes/cardinality_reduction, filter/critical_operations, attributes/cluster_state_aggregate, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

(Opcional) Para Elasticsearch seguro con autenticación y SSL, modifique la configuración del receptor:

receivers:
  elasticsearch:
    endpoint: "https://localhost:9200"
    username: "your_elasticsearch_username"
    password: "your_elasticsearch_password"
    tls:
      ca_file: "/etc/elasticsearch/certs/http_ca.crt"
      insecure_skip_verify: false
    collection_interval: 15s

(Opcional) Para recopilar métricas del host, agregue el receptor hostmetrics:

receivers:
  hostmetrics:
    collection_interval: 60s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization: {enabled: true}
          system.cpu.time: {enabled: true}
      load:
        metrics:
          system.cpu.load_average.1m: {enabled: true}
          system.cpu.load_average.5m: {enabled: true}
          system.cpu.load_average.15m: {enabled: true}
      memory:
        metrics:
          system.memory.usage: {enabled: true}
          system.memory.utilization: {enabled: true}
      disk:
        metrics:
          system.disk.io: {enabled: true}
          system.disk.operations: {enabled: true}
      filesystem:
        metrics:
          system.filesystem.usage: {enabled: true}
          system.filesystem.utilization: {enabled: true}
      network:
        metrics:
          system.network.io: {enabled: true}
          system.network.packets: {enabled: true}
      process:
        metrics:
          process.cpu.utilization:
            enabled: true

Y agregue a los pipelines de servicio:

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp]

(Optional) To collect Elasticsearch logs, add the filelog receiver. Ensure the user running the collector service (nrdot-collector) has read access to your Elasticsearch log files:

Si ejecuta Elasticsearch en Linux (Host):

receivers:
  filelog:
    include:
      - /var/log/elasticsearch/elasticsearch.log
      - /var/log/elasticsearch/*.log

Si ejecuta Elasticsearch en Docker:

receivers:
  filelog:
    include:
      - /var/lib/docker/containers/*/*.log
    operators:
      - type: move
        from: attributes.log
        to: body

Y agregue a los pipelines de servicio:

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resource/cluster_name_override]
      exporters: [otlphttp]

(Opcional) Para agregar etiquetas de metadatos personalizadas a sus métricas, utilice el procesador resource/static_override:

processors:
  resource/static_override:
    attributes:
      - key: env
        value: "production"
        action: upsert
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, resource/static_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

Almacene el archivo de configuración.
Configura las variables de entorno:
Crea un directorio de anulación de systemd:
bash
```
$sudo mkdir -p /etc/systemd/system/nrdot-collector.service.d
```
Cree el archivo de configuración del entorno:
bash
```
$cat <<EOF | sudo tee /etc/systemd/system/nrdot-collector.service.d/environment.conf
$[Service]
$Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
$Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE"
$Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"
$EOF
```
Actualice la configuración con sus valores:
- Reemplace https://otlp.nr-data.net:4318 con el punto final de su región
- Reemplace YOUR_LICENSE_KEY_HERE con su clave de licencia de New Relic real
- Reemplace 100 con el límite de memoria deseado en MiB para el colector (predeterminado: 100 MiB). Ajuste según las necesidades de su entorno

Reinicie el colector NRDOT para aplicar los cambios:

bash

$sudo systemctl daemon-reload
$sudo systemctl restart nrdot-collector.service

Verificar la recopilación de datos

Verify that the NRDOT collector is running and collecting data without errors:

Verifique el estado del servicio del colector:
bash
```
$sudo systemctl status nrdot-collector.service
```
Monitoree los logs del recopilador en busca de errores:
bash
```
$sudo journalctl -u nrdot-collector.service -f
```
Busque conexiones exitosas a Elasticsearch y New Relic. Si ve errores, consulte la guía de solución de problemas.

Antes de que empieces

Antes de configurar el OTel Collector Contrib, asegúrese de tener:

Privilegios de acceso requeridos:

Tu New Relic
Privilegios de root o sudo en la máquina host
Privilegios de administrador de clúster de Elasticsearch con privilegio de clúster monitor o manage (consulte la documentación de privilegios de seguridad de Elasticsearch para obtener detalles)
Requisitos del sistema:
Versión 7.16 o superior de Elasticsearch - Esta integración requiere un clúster moderno de Elasticsearch
Conectividad de red - HTTPS saliente (puerto 443) al endpoint de ingesta OTLPde New Relic
Valores de configuración listos:
Punto de conexión de Elasticsearch - La URL de su clúster de Elasticsearch (p. ej., http://localhost:9200)
Nombre del clúster - Un identificador único para su clúster en New Relic
Importante
Debe tener OpenTelemetry Collector Contrib instalado en su host antes de continuar. Recomendamos instalar mediante paquetes oficiales (.deb o .rpm) para garantizar que la unidad de servicio systemd se cree correctamente.

Configurar el monitoreo de Elasticsearch

Una vez que OTel Collector Contrib esté instalado, reemplace el archivo de configuración predeterminado del collector con la configuración de monitoreo de Elasticsearch. Esto habilitará la recopilación de métricas de Elasticsearch. Las métricas y los logs del host son opcionales y se pueden agregar por separado.

El archivo de configuración se encuentra en: /etc/otelcol-contrib/config.yaml

Sugerencia

Respalde su configuración predeterminada: Antes de modificar el archivo de configuración, cree una copia de seguridad para conservar los ajustes predeterminados:

bash

$sudo cp /etc/otelcol-contrib/config.yaml /etc/otelcol-contrib/config.yaml.backup

Para configurar el colector:

Abra el archivo de configuración con un editor de texto con privilegios de root o sudo:
bash
```
$sudo nano /etc/otelcol-contrib/config.yaml
```

Elimine todo el contenido existente y reemplácelo con la siguiente configuración para el monitoreo de Elasticsearch:

Importante

receivers:
  elasticsearch:
    endpoint: "http://localhost:9200"
    collection_interval: 15s
    metrics:
      elasticsearch.os.cpu.usage:
        enabled: true
      elasticsearch.cluster.data_nodes:
        enabled: true
      elasticsearch.cluster.health:
        enabled: true
      elasticsearch.cluster.in_flight_fetch:
        enabled: true
      elasticsearch.cluster.nodes:
        enabled: true
      elasticsearch.cluster.pending_tasks:
        enabled: true
      elasticsearch.cluster.shards:
        enabled: true
      elasticsearch.cluster.state_update.time:
        enabled: true
      elasticsearch.index.documents:
        enabled: true
      elasticsearch.index.operations.merge.current:
        enabled: true
      elasticsearch.index.operations.time:
        enabled: true
      elasticsearch.node.cache.count:
        enabled: true
      elasticsearch.node.cache.evictions:
        enabled: true
      elasticsearch.node.cache.memory.usage:
        enabled: true
      elasticsearch.node.shards.size:
        enabled: true
      elasticsearch.node.cluster.io:
        enabled: true
      elasticsearch.node.documents:
        enabled: true
      elasticsearch.node.disk.io.read:
        enabled: true
      elasticsearch.node.disk.io.write:
        enabled: true
      elasticsearch.node.fs.disk.available:
        enabled: true
      elasticsearch.node.fs.disk.total:
        enabled: true
      elasticsearch.node.http.connections:
        enabled: true
      elasticsearch.node.ingest.documents.current:
        enabled: true
      elasticsearch.node.ingest.operations.failed:
        enabled: true
      elasticsearch.node.open_files:
        enabled: true
      elasticsearch.node.operations.completed:
        enabled: true
      elasticsearch.node.operations.current:
        enabled: true
      elasticsearch.node.operations.get.completed:
        enabled: true
      elasticsearch.node.operations.time:
        enabled: true
      elasticsearch.node.shards.reserved.size:
        enabled: true
      elasticsearch.index.shards.size:
        enabled: true
      elasticsearch.os.cpu.load_avg.1m:
        enabled: true
      elasticsearch.os.cpu.load_avg.5m:
        enabled: true
      elasticsearch.os.cpu.load_avg.15m:
        enabled: true
      elasticsearch.os.memory:
        enabled: true
      jvm.gc.collections.count:
        enabled: true
      jvm.gc.collections.elapsed:
        enabled: true
      jvm.memory.heap.max:
        enabled: true
      jvm.memory.heap.used:
        enabled: true
      jvm.memory.heap.utilization:
        enabled: true
      jvm.threads.count:
        enabled: true
      elasticsearch.index.segments.count:
        enabled: true
      elasticsearch.index.operations.completed:
        enabled: true
      elasticsearch.node.script.cache_evictions:
        enabled: false
      elasticsearch.node.cluster.connections:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.preprocessed:
        enabled: false
      elasticsearch.node.thread_pool.tasks.queued:
        enabled: false
      elasticsearch.cluster.published_states.full:
        enabled: false
      jvm.memory.pool.max:
        enabled: false
      elasticsearch.node.script.compilation_limit_triggered:
        enabled: false
      elasticsearch.node.shards.data_set.size:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.current:
        enabled: false
      elasticsearch.cluster.state_update.count:
        enabled: false
      elasticsearch.node.fs.disk.free:
        enabled: false
      jvm.memory.nonheap.used:
        enabled: false
      jvm.memory.pool.used:
        enabled: false
      elasticsearch.node.translog.size:
        enabled: false
      elasticsearch.node.thread_pool.threads:
        enabled: false
      elasticsearch.cluster.state_queue:
        enabled: false
      elasticsearch.node.translog.operations:
        enabled: false
      elasticsearch.memory.indexing_pressure:
        enabled: false
      elasticsearch.node.ingest.documents:
        enabled: false
      jvm.classes.loaded:
        enabled: false
      jvm.memory.heap.committed:
        enabled: false
      elasticsearch.breaker.memory.limit:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.replica_rejections:
        enabled: false
      elasticsearch.breaker.memory.estimated:
        enabled: false
      elasticsearch.cluster.published_states.differences:
        enabled: false
      jvm.memory.nonheap.committed:
        enabled: false
      elasticsearch.node.translog.uncommitted.size:
        enabled: false
      elasticsearch.node.script.compilations:
        enabled: false
      elasticsearch.node.pipeline.ingest.operations.failed:
        enabled: false
      elasticsearch.indexing_pressure.memory.limit:
        enabled: false
      elasticsearch.breaker.tripped:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.primary_rejections:
        enabled: false
      elasticsearch.node.thread_pool.tasks.finished:
        enabled: false
processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}
  cumulativetodelta: {}
  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert
  resourcedetection:
    detectors: [ system ]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true
  batch:
    timeout: 10s
    send_batch_size: 1024
  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete
  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")
exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

(Opcional) Para Elasticsearch seguro con autenticación y SSL, modifique la configuración del receptor:

receivers:
  elasticsearch:
    endpoint: "https://localhost:9200"
    username: "your_elasticsearch_username"
    password: "your_elasticsearch_password"
    tls:
      ca_file: "/etc/elasticsearch/certs/http_ca.crt"
      insecure_skip_verify: false
    collection_interval: 15s

(Opcional) Para recopilar métricas del host, agregue el receptor hostmetrics:

receivers:
  hostmetrics:
    collection_interval: 60s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization: {enabled: true}
          system.cpu.time: {enabled: true}
      load:
        metrics:
          system.cpu.load_average.1m: {enabled: true}
          system.cpu.load_average.5m: {enabled: true}
          system.cpu.load_average.15m: {enabled: true}
      memory:
        metrics:
          system.memory.usage: {enabled: true}
          system.memory.utilization: {enabled: true}
      disk:
        metrics:
          system.disk.io: {enabled: true}
          system.disk.operations: {enabled: true}
      filesystem:
        metrics:
          system.filesystem.usage: {enabled: true}
          system.filesystem.utilization: {enabled: true}
      network:
        metrics:
          system.network.io: {enabled: true}
          system.network.packets: {enabled: true}
      process:
        metrics:
          process.cpu.utilization:
            enabled: true

Y agregue a los pipelines de servicio:

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp]

(Opcional) Para recopilar logs de Elasticsearch, agregue el receptor filelog. Asegúrese de que el usuario que ejecuta el servicio del recopilador (otelcol-contrib) tenga acceso de lectura a sus archivos de log de Elasticsearch:

Si ejecuta Elasticsearch en Linux (Host):

receivers:
  filelog:
    include:
      - /var/log/elasticsearch/elasticsearch.log
      - /var/log/elasticsearch/*.log

Si ejecuta Elasticsearch en Docker:

receivers:
  filelog:
    include:
      - /var/lib/docker/containers/*/*.log
    operators:
      - type: move
        from: attributes.log
        to: body

Y agregue a los pipelines de servicio:

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resource/cluster_name_override]
      exporters: [otlphttp]

(Opcional) Para agregar etiquetas de metadatos personalizadas a sus métricas, utilice el procesador resource/static_override:

processors:
  resource/static_override:
    attributes:
      - key: env
        value: "production"
        action: upsert
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, resource/static_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

Almacene el archivo de configuración.
Configura las variables de entorno:
Crea un directorio de anulación de systemd:
bash
```
$sudo mkdir -p /etc/systemd/system/otelcol-contrib.service.d
```
Cree el archivo de configuración del entorno:
bash
```
$cat <<EOF | sudo tee /etc/systemd/system/otelcol-contrib.service.d/environment.conf
$[Service]
$Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
$Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE"
$Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"
$EOF
```
Actualice la configuración con sus valores:
- Reemplace https://otlp.nr-data.net:4318 con el punto final de su región
- Reemplace YOUR_LICENSE_KEY_HERE con su clave de licencia de New Relic real
- Reemplace 100 con el límite de memoria deseado en MiB para el colector (predeterminado: 100 MiB). Ajuste según las necesidades de su entorno

Reinicie el OTel Collector Contrib para aplicar los cambios:

bash

$sudo systemctl daemon-reload
$sudo systemctl restart otelcol-contrib.service

Verificar la recopilación de datos

Verifique que el OTel Collector Contrib se esté ejecutando y recopilando datos sin errores:

Verifique el estado del servicio del colector:
bash
```
$sudo systemctl status otelcol-contrib.service
```
Monitoree los logs del recopilador en busca de errores:
bash
```
$sudo journalctl -u otelcol-contrib.service -f
```
Busque conexiones exitosas a Elasticsearch y New Relic. Si ve errores, consulte la guía de solución de problemas.

Use this approach if you already have a Prometheus Elasticsearch exporter running in your environment, or if you're migrating from a Prometheus-based monitoring stack.

Sugerencia

Recommended: If you don't already have a Prometheus exporter running, use the NRDOT Collector or OTel Collector Contrib tabs instead. They connect directly to the Elasticsearch API without needing an additional exporter component.

Install and configure the Elasticsearch exporter

The Prometheus Elasticsearch exporter exposes Elasticsearch metrics in Prometheus format on port 9114.

Download the latest release:

bash

$wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.8.0/elasticsearch_exporter-1.8.0.linux-amd64.tar.gz
$tar xzf elasticsearch_exporter-*.tar.gz
$cd elasticsearch_exporter-*

Start the exporter (point it to your Elasticsearch instance):
bash
```
$./elasticsearch_exporter --es.uri=http://localhost:9200
```
Sugerencia
For secured Elasticsearch, use: --es.uri=https://username:password@localhost:9200 --es.ssl-skip-verify
Verify the exporter is running:
bash
```
$curl http://localhost:9114/metrics | grep elasticsearch_cluster_health
```
You should see Prometheus-formatted metrics like elasticsearch_cluster_health_status, elasticsearch_cluster_health_number_of_nodes, etc.

Configurar el recolector

This configuration works with both NRDOT and OTel Collector Contrib. Place it in the appropriate config location for your collector:

NRDOT: /etc/nrdot-collector/config.yaml

OTel Collector Contrib: /etc/otelcol-contrib/config.yaml

If you don't have a collector installed yet, follow the installation steps in the NRDOT Collector or OTel Collector Contrib tabs above first, then return here to apply this configuration.

This configuration scrapes metrics from the Elasticsearch exporter and translates them to OpenTelemetry-compatible metric names that power the New Relic Elasticsearch dashboard.

receivers:
  prometheus/elasticsearch:
    config:
      scrape_configs:
        - job_name: 'elasticsearch'
          scrape_interval: 15s
          static_configs:
            - targets: ['localhost:9114']  # <-- elasticsearch_exporter address

processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}

  metricstransform/prom_to_otel:
    transforms:
      - include: ^elasticsearch_cluster_health_number_of_data_nodes$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.data_nodes
      - include: ^elasticsearch_cluster_health_status$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.health
        operations:
          - action: update_label
            label: color
            new_label: status
      - include: ^elasticsearch_cluster_health_number_of_in_flight_fetch$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.in_flight_fetch
      - include: ^elasticsearch_cluster_health_number_of_nodes$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.nodes
      - include: ^elasticsearch_cluster_health_number_of_pending_tasks$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.pending_tasks
      - include: ^elasticsearch_cluster_health_active_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: active
      - include: ^elasticsearch_cluster_health_active_primary_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: active_primary
      - include: ^elasticsearch_cluster_health_initializing_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: initializing
      - include: ^elasticsearch_cluster_health_relocating_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: relocating
      - include: ^elasticsearch_cluster_health_unassigned_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: unassigned
      - include: ^elasticsearch_indices_docs$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.documents
        operations:
          - action: add_label
            new_label: state
            new_value: active
      - include: ^elasticsearch_indices_docs$
        match_type: regexp
        action: insert
        new_name: elasticsearch.index.documents
        operations:
          - action: add_label
            new_label: state
            new_value: active
          - action: add_label
            new_label: aggregation
            new_value: primary_shards
          - action: add_label
            new_label: elasticsearch.index.name
            new_value: _all
      - include: ^elasticsearch_indices_docs_deleted$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.documents
        operations:
          - action: add_label
            new_label: state
            new_value: deleted
      - include: ^elasticsearch_indices_fielddata_evictions$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.evictions
        operations:
          - action: add_label
            new_label: cache_name
            new_value: fielddata
      - include: ^elasticsearch_indices_query_cache_evictions$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.evictions
        operations:
          - action: add_label
            new_label: cache_name
            new_value: query
      - include: ^elasticsearch_indices_fielddata_memory_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.memory.usage
        operations:
          - action: add_label
            new_label: cache_name
            new_value: fielddata
      - include: ^elasticsearch_indices_query_cache_memory_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.memory.usage
        operations:
          - action: add_label
            new_label: cache_name
            new_value: query
      - include: ^elasticsearch_transport_rx_size_bytes_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cluster.io
        operations:
          - action: add_label
            new_label: direction
            new_value: received
      - include: ^elasticsearch_transport_tx_size_bytes_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cluster.io
        operations:
          - action: add_label
            new_label: direction
            new_value: sent
      - include: ^elasticsearch_filesystem_data_available_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.fs.disk.available
      - include: ^elasticsearch_filesystem_data_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.fs.disk.total
      - include: ^elasticsearch_process_open_files_count$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.open_files
      - include: ^elasticsearch_indices_indexing_index_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: index
      - include: ^elasticsearch_indices_search_query_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: query
      - include: ^elasticsearch_indices_get_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: get
      - include: ^elasticsearch_indices_merges_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: merge
      - include: ^elasticsearch_indices_indexing_index_time_seconds_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.time
        operations:
          - action: add_label
            new_label: operation
            new_value: index
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_indices_search_query_time_seconds$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.time
        operations:
          - action: add_label
            new_label: operation
            new_value: query
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_thread_pool_active_count$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.operations.current
        operations:
          - action: update_label
            label: type
            new_label: operation
      - include: ^elasticsearch_os_cpu_percent$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.usage
      - include: ^elasticsearch_os_load1$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.1m
      - include: ^elasticsearch_os_load5$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.5m
      - include: ^elasticsearch_os_load15$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.15m
      - include: ^elasticsearch_jvm_gc_collection_seconds_count$
        match_type: regexp
        action: update
        new_name: jvm.gc.collections.count
      - include: ^elasticsearch_jvm_gc_collection_seconds_sum$
        match_type: regexp
        action: update
        new_name: jvm.gc.collections.elapsed
        operations:
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_jvm_memory_max_bytes$
        match_type: regexp
        action: update
        new_name: jvm.memory.heap.max
      - include: ^elasticsearch_jvm_memory_used_bytes$
        match_type: regexp
        action: update
        new_name: jvm.memory.heap.used
      - include: ^elasticsearch_thread_pool_threads_count$
        match_type: regexp
        action: update
        new_name: jvm.threads.count
        operations:
          - action: aggregate_labels
            label_set: [name, cluster]
            aggregation_type: sum
      - include: ^elasticsearch_indices_segments_count$
        match_type: regexp
        action: update
        new_name: elasticsearch.index.segments.count

  filter/allowed_metrics:
    metrics:
      include:
        match_type: strict
        metric_names:
          - elasticsearch.cluster.data_nodes
          - elasticsearch.cluster.health
          - elasticsearch.cluster.in_flight_fetch
          - elasticsearch.cluster.nodes
          - elasticsearch.cluster.pending_tasks
          - elasticsearch.cluster.shards
          - elasticsearch.index.documents
          - elasticsearch.index.segments.count
          - elasticsearch.node.cache.evictions
          - elasticsearch.node.cache.memory.usage
          - elasticsearch.node.cluster.io
          - elasticsearch.node.documents
          - elasticsearch.node.fs.disk.available
          - elasticsearch.node.fs.disk.total
          - elasticsearch.node.open_files
          - elasticsearch.node.operations.completed
          - elasticsearch.node.operations.current
          - elasticsearch.node.operations.time
          - elasticsearch.os.cpu.usage
          - elasticsearch.os.cpu.load_avg.1m
          - elasticsearch.os.cpu.load_avg.5m
          - elasticsearch.os.cpu.load_avg.15m
          - jvm.gc.collections.count
          - jvm.gc.collections.elapsed
          - jvm.memory.heap.max
          - jvm.memory.heap.used
          - jvm.threads.count

  transform/rename_node_name:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["elasticsearch.node.name"], attributes["name"]) where attributes["name"] != nil
          - delete_key(attributes, "name") where attributes["elasticsearch.node.name"] != nil

  groupbyattrs/node:
    keys:
      - elasticsearch.node.name
      - elasticsearch.cluster.name

  cumulativetodelta: {}

  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert

  transform/scope_override:
    metric_statements:
      - context: scope
        statements:
          - set(name, "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/elasticsearchreceiver")
          - set(version, "")

  resourcedetection:
    detectors: [system]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true

  batch:
    timeout: 30s
    send_batch_size: 2048
    send_batch_max_size: 4096

  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete

  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")

exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
    compression: gzip
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    metrics/elasticsearch:
      receivers: [prometheus/elasticsearch]
      processors:
        - memory_limiter
        - metricstransform/prom_to_otel
        - filter/allowed_metrics
        - transform/rename_node_name
        - groupbyattrs/node
        - transform/scope_override
        - resourcedetection
        - resource/cluster_name_override
        - attributes/cardinality_reduction
        - cumulativetodelta
        - transform/metadata_nullify
        - batch
      exporters: [otlphttp]

Replace the following values in the configuration:

<elasticsearch-cluster-name>: Your Elasticsearch cluster name for identification in New Relic.
localhost:9114: The address of your elasticsearch_exporter if running on a different host or port.

Set up environment variables

Create a systemd override to inject the required environment variables. Replace <collector-service> with your collector service name (nrdot-collector or otelcol-contrib):

bash

$sudo mkdir -p /etc/systemd/system/<collector-service>.service.d

Create the file /etc/systemd/system/<collector-service>.service.d/environment.conf:

[Service]
Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
Environment="NEWRELIC_LICENSE_KEY=YOUR_NEWRELIC_LICENSE_KEY"
Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"

Reemplace YOUR_NEWRELIC_LICENSE_KEY con su .

Sugerencia

For EU accounts, use NEWRELIC_OTLP_ENDPOINT=https://otlp.eu01.nr-data.net:4318

Iniciar el colector

Replace <collector-service> with your collector service name (nrdot-collector or otelcol-contrib):

bash

$sudo systemctl daemon-reload
$sudo systemctl enable <collector-service>
$sudo systemctl restart <collector-service>

Check the collector status:

bash

$sudo systemctl status <collector-service>
$sudo journalctl -u <collector-service> -f

Verifique los datos en New Relic

After a few minutes, verify that data is flowing to New Relic:

FROM Metric SELECT count(*)
WHERE metricName LIKE 'elasticsearch.%'
AND elasticsearch.cluster.name = '<elasticsearch-cluster-name>'
SINCE 10 minutes ago

Sugerencia

Correlacione APM con Elasticsearch: Para conectar su aplicación APM y el clúster de Elasticsearch, incluya el atributo de recurso es.cluster.name="your-cluster-name" en sus métricas de APM. Esto permite la visibilidad entre servicios y una solución de problemas más rápida dentro de New Relic.

Ver sus datos de Elasticsearch

Una vez que el recopilador se esté ejecutando y enviando datos, puede explorar sus métricas de Elasticsearch, crear consultas personalizadas y configurar dashboards de monitoreo en New Relic.

Para obtener información detallada sobre cómo acceder a sus datos, escribir consultas NRQL y configurar alertas, consulte Buscar y consultar datos de Elasticsearch.

Resolución de problemas

Si encuentra problemas durante la instalación o no ve datos en New Relic, consulte nuestra completa guía de solución de problemas para obtener soluciones paso a paso a los problemas comunes.

Te ofrecemos esta traducción automática para facilitar la lectura.

Monitorear Elasticsearch autohospedado con OpenTelemetry

Opciones de instalación

Antes de que empieces

Configurar el monitoreo de Elasticsearch

Sugerencia

Importante

Verificar la recopilación de datos

Antes de que empieces

Importante

Configurar el monitoreo de Elasticsearch

Sugerencia

Importante

Verificar la recopilación de datos

Sugerencia

Install and configure the Elasticsearch exporter

Sugerencia

Configurar el recolector

Full Prometheus receiver configuration

Set up environment variables

Sugerencia

Iniciar el colector

Verifique los datos en New Relic

Sugerencia

Ver sus datos de Elasticsearch

Resolución de problemas

Te ofrecemos esta traducción automática para facilitar la lectura.

Monitorear Elasticsearch autohospedado con OpenTelemetry

Opciones de instalación .css-21sua1{background:none;border:none;width:0;padding:0;}

Antes de que empieces

Configurar el monitoreo de Elasticsearch

Sugerencia

Importante

Verificar la recopilación de datos

Antes de que empieces

Importante

Configurar el monitoreo de Elasticsearch

Sugerencia

Importante

Verificar la recopilación de datos

Sugerencia

Install and configure the Elasticsearch exporter

Sugerencia

Configurar el recolector

Set up environment variables

Sugerencia

Iniciar el colector

Verifique los datos en New Relic

Sugerencia

Ver sus datos de Elasticsearch

Resolución de problemas

Opciones de instalación