Prometheus High Availability (HA)

If you are using our Prometheus remote write integration in a high-availability (HA) configuration, you need to make sure your Prometheus servers aren't sending multiple copies of the same metrics to New Relic. This document describes how you can configure your remote write integration so that New Relic does not keep duplicated metrics.

Tip

This page applies only if you run your own Prometheus servers in a high-availability configuration and forward metrics with remote write. If you don't already run Prometheus, the Prometheus agent for Kubernetes is a fully New Relic-managed alternative that doesn't require HA deduplication; see Send Prometheus metric data to New Relic to compare your options.

Quick reference

New Relic deduplicates HA replicas using three values. Set all three on every replica in a high-availability cluster:

Values	Where you set it	Value across replicas in the same HA cluster	Example
`prometheus`	Prometheus external label	Same for all replicas	`monitoring-cluster`
`prometheus_replica`	Prometheus external label	Unique per replica	`replica-1`, `replica-2`
`prometheus_server`	`remote_write` URL query parameter	Same for all replicas	`prod-monitoring`

How it works: New Relic organizes replicas into HA clusters based on your account, the prometheus label, and the prometheus_server parameter. In each cluster, New Relic designates one replica as the leader and stores only the data from that leader. The leader is identified by the prometheus_replica label.

Set up deduplication

Prometheus Operator

Prometheus Operator version 0.19.0 or higher adds the prometheus and prometheus_replica external labels for you, whether you use the operator directly or via the helm chart:

prometheus is set to <prometheus deployment namespace>/<prometheus deployment name>. For example, a deployment named prometheus-cluster1 in the monitoring namespace produces monitoring/prometheus-cluster1.
prometheus_replica is set to each replica's pod name, in the format replica-<replica number> (for example, replica-1).

Set prometheus_server yourself in the remoteWrite configuration on the Prometheus resource, using the same value for every replica. For example, this resource runs a two-replica HA pair with a single remoteWrite block, so prometheus_server stays consistent across both replicas:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: monitoring-cluster
  namespace: monitoring
spec:
  replicas: 2
  remoteWrite:
    - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=prod-monitoring
      authorization:
        credentials:
          name: nr-license-key
          key: licenseKey

The operator then sets prometheus to monitoring/monitoring-cluster and prometheus_replica to each replica's pod name automatically — you only supply prometheus_server. If you use the kube-prometheus-stack helm chart, set the same remoteWrite block under prometheus.prometheusSpec.

Standalone Prometheus

Add the external labels to each replica's configuration file, and use an identical remote_write URL — including the prometheus_server parameter — across all replicas.

Replica 1 (prometheus.yml)

global:
  external_labels:
    prometheus: monitoring-cluster
    prometheus_replica: replica-1

remote_write:
  - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=prod-monitoring
    authorization:
      credentials: YOUR_LICENSE_KEY

Replica 2 (prometheus.yml)

global:
  external_labels:
    prometheus: monitoring-cluster
    prometheus_replica: replica-2

remote_write:
  - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=prod-monitoring
    authorization:
      credentials: YOUR_LICENSE_KEY

Only prometheus_replica differs between the two replicas; prometheus and prometheus_server are identical.

Limitations

Keep prometheus_server identical across replicas. Because New Relic uses prometheus_server to group replicas, giving replicas different values — for example, per host or per pod — places each in its own HA group. Each replica becomes the sole member of its group, is always elected leader, and retains all its data. The result is duplicate data in New Relic with no error or warning, since each replica appears as a healthy, standalone leader. To distinguish individual replicas, use the prometheus_replica label instead.

An account can have up to 1,500 unique Prometheus HA clusters, counted per unique combination of account, prometheus label, and prometheus_server parameter. Exceeding this limit causes data from additional HA clusters to be dropped, and New Relic generates PrometheusHAClusterLimit NrIntegrationError events. Keeping prometheus_server consistent across replicas also prevents a single HA group from consuming multiple slots against this limit.

Troubleshooting

Prometheus Operator does not set prometheus_server — that value comes entirely from your remoteWrite configuration on the Prometheus resource. When all replicas share a single remoteWrite block (the normal case), this is consistent by default. If you inject per-replica remoteWrite overrides, ensure prometheus_server is the same across all of them.

If you still see duplicate copies of replica data, ensure you do not have replicaExternalLabelName or prometheusExternalLabelName in your Prometheus spec or chart configuration because these overrides change the label name.

For the most consistent deduplication, keep your Prometheus scrape_interval at 60 seconds or less. New Relic identifies the active replica in each HA cluster from its most recently received data, and Prometheus remote write forwards new samples about once per scrape. If a cluster sends data less frequently than that, New Relic may not consistently recognize the same leader and can retain duplicated or interleaved data from more than one replica, with no error surfaced.