eBPF agent best practices guide

New Relic eBPF agent uses eBPF technology to provide APM functionality in a single agent with zero code instrumentation. This approach empowers platform engineering teams by eliminating the need for coordination with application teams for monitoring deployment.

When to use eBPF APM

• Large-scale deployments: When you have many applications that need monitoring deployed at scale and require "good enough" metrics without the overhead of individual language agents.
• Unknown or unmodifiable workloads: When the workload you want to monitor is written in an unknown programming language and/or cannot be modified.
• Platform engineering efficiency: When you want to deploy monitoring at scale without coordinating with individual application teams.
• Linux-focused environments: When you don't need to monitor Windows platforms, as eBPF works excellently on Linux in both Kubernetes and host environments.
• No distributed tracing requirement: When your monitoring needs don't require distributed tracing capabilities.

eBPF vs traditional APM comparison

Understanding the differences between eBPF APM and traditional APM agents helps you choose the right approach:

Data source perspective

eBPF APM shifts the monitoring perspective from the application layer to the kernel layer:

Best practices for deployment

1. Supplement the traditional APM

Use eBPF agent to supplement APM language agents for comprehensive coverage. This gives you full APM coverage with coexistence between eBPF APM and APM agents, without double data ingestion.

Recommended approach:

• APM language agents: Use for your most critical applications that require deep-level insights, distributed tracing, or custom instrumentation.
• eBPF APM: Use to cover everything else, including uninstrumented services, third-party apps, and for continuously discovering/reporting new services.

2. Add network metrics for deeper context

The eBPF agent can also provide granular network metrics (TCP, DNS, etc.) to give you visibility outside your application's boundary. This capability is complementary and can be used with or without eBPF APM. For more information, refer to network-metrics.

The following deployment options are available:

3. Create a unified monitoring approach

By following the practices above, you can use the single eBPF agent to handle both infrastructure and application monitoring needs. This unifies your observability, allowing the agent to automatically gather critical application performance data (eBPF APM) and network metrics (eBPF network metrics) to populate existing dashboards, providing deep context without code changes or restarts.

Benefits:

• APM insights that populate New Relic APM UI automatically.
• Network insights from the same agent.

Implementation recommendations

• Start with eBPF APM for scale: If you need to deploy monitoring at scale across many applications and want "good enough" metrics without complex coordination, start with eBPF APM.

• Add network metrics for complete visibility: Once eBPF APM is deployed, consider adding eBPF network metrics to gain visibility beyond application boundaries for comprehensive troubleshooting capabilities.

Data sampling and fine-tuning

APM language agents cap transaction events with a fixed reservoir, by default 10,000 events per minute (configured with transaction_events.max_samples_stored). The eBPF agent does not use a single fixed per-minute limit. Instead, it samples each telemetry type differently, so understanding how each type behaves helps you control data volume and cost without losing the signals you care about.

samplingLatency

samplingErrorRate

p50

max_unlinked_spans

unlinked_spans_error_quota_percentage

100

30

10000

How span sampling works

The eBPF agent does not limit spans to a fixed number per minute. For each protocol, it exports only the spans that cross a threshold you set:

• Latency threshold (samplingLatency): exports spans slower than the chosen percentile for a given route. The default is p50 (the median). Raise it toward p90 or p99 to keep only the slowest tail and reduce data volume; lower it toward p1 or p0 to keep nearly all spans and increase visibility. Any percentile from p0 to p99 is accepted.
• Error-rate threshold (samplingErrorRate, HTTP): a value from 1 to 100. When a route's error rate exceeds this threshold, spans for that route are exported so failures are never sampled away. Leave it empty to disable error-based export.

Controlling unlinked spans

Spans that the agent captures but cannot associate with a parent are called unlinked spans. They are capped per protocol by max_unlinked_spans (default 100; set to 0 to disable the cap). The unlinked_spans_error_quota_percentage setting (default 30) reserves part of that budget for error spans, so failures stay visible even when normal spans arrive first.

Tuning examples

Sampling is configured in the Helm values.yaml file (Kubernetes) or in /etc/newrelic-ebpf-agent/newrelic-ebpf-agent.yaml (Linux host). The keys are identical for both.

protocols:
  global:
    max_unlinked_spans: 100          # 0 disables the cap
    unlinked_spans_error_quota_percentage: 30
  http:
    enabled: true
    spans:
      enabled: true
      samplingLatency: "p90"         # keep only slower requests, less data
      samplingErrorRate: "5"         # always keep routes with >5% errors
logDataFilters:
  applicationLogReporting:
    maxSamplesPerMinute: 10000         # APM-equivalent reservoir for logs

Recommendations

• To maximize trace visibility: lower samplingLatency toward p0 and set a low samplingErrorRate, but expect higher ingest.
• To reduce data: raise samplingLatency (for example, p90 or p99), disable unwanted protocols all together (or just spans.enabled: false to keep the protocol metrics only), and lower maxSamplesPerMinute for logs.
• Keep metrics in mind: metrics are unaffected by span sampling. Disable an entire protocol only when you don't need its data at all.

Related articles