Kubernetes APM auto-attach

The Kubernetes APM auto-attach, formerly known as the Kubernetes agent operator, streamlines full-stack observability for Kubernetes environments by automating APM instrumentation alongside Kubernetes agent deployment. By enabling auto instrumentation, developers no longer need to manually manage APM agents. The Kubernetes APM auto-attach will automatically install, upgrade and remove APM agents.

It currently supports Java, .NET, Node.js, Python, Ruby, and PHP.

How it works

• The MutatingWebHook, upon installation, becomes involved in intercepting API requests for deploying pods onto nodes.

• Reflecting the configurations specified, it mutates the pod specification to add a NR init container and environment variables.

• Following the establishment of the pod, the New Relic APM Agent is seamlessly integrated into the application housed within it.

Before you begin

Before installing the operator, check the following:

• Helm: You must install it to use the charts. See the Helm documentation if you need help getting started.

• Kubectl: You have to configure it to communicate with your cluster.

Installation

Depending on what you need, you can choose to install the Kubernetes APM auto-attach independently or together with our Kubernetes integrations.

We strongly recommend to install it together with the Kubernetes integration to take advantage of our entire full stack observability experience.

Bundle installation in addition to the Kubernetes integration (recommended)

The Kubernetes APM auto-attach chart is part of the nri-bundle chart, which manages the installation of all the components needed to enable a full Kubernetes observability.

Add the k8s-agents-operator.enabled=true parameter to your helm command or include it in the values.yaml file. See the Install the Kubernetes integration page for more information about using Helm or check out the nri-bundle chart.

See this sample of Helm commands using parameters:

$
helm repo add newrelic https://helm-charts.newrelic.com
$

$
helm upgrade --install newrelic-bundle newrelic/nri-bundle \
>
    --set global.licenseKey=YOUR_NEW_RELIC_INGEST_LICENSE_KEY \
>
    --set global.cluster=CLUSTER_NAME \
>
    --namespace=newrelic \
>
    --set newrelic-infrastructure.privileged=true \
>
    --set global.lowDataMode=true \
>
    --set kube-state-metrics.enabled=true \
>
    --set kubeEvents.enabled=true \
>
    --set k8s-agents-operator.enabled=true \
>
    --create-namespace

Standalone installation

To install the Kubernetes APM auto-attach with the default configuration, run these commands:

$
helm repo add k8s-agents-operator https://newrelic.github.io/k8s-agents-operator
$
helm upgrade --install k8s-agents-operator k8s-agents-operator/k8s-agents-operator \
>
    --namespace newrelic \
>
    --create-namespace \
>
    --set licenseKey=YOUR_NEW_RELIC_INGEST_LICENSE_KEY

For a complete list of configuration options, see the README chart.

Configure auto-instrumentation

After APM auto-attach is all set up in your cluster, the next step is just to roll out the configs required to get it operational. That involves having at least one instrumentation Custom Resource (CR) active in the cluster.

Here's what the instrumentation CR lets you map out:

• Name of the instrumentation CR
• Where it will apply the instrumentation CR (thanks to podLabelSelector and namespaceLabelSelector)
• APM agent (one per CR)
• APM agent version
• APM config parameters (env vars)
• License key (optional)

The manifest file needs to injected in the same namespace (newrelic by default) where you installed APM auto-attach.

$
kubectl apply -f ./values.yaml -n newrelic

How to use selectors

To know when the instrumentation CR is going to inject APM agents, we need to use selectors. There are 2 label selectors available that you can use together (they act as a logical AND (&&) operator) or by separate depending on your needs.

• PodLabelSelector informs the APM Auto-attach which pods needs to be instrumented.

  Example using matchLabel (select pods containing an specific tag and value):

  ...
    podLabelSelector:
      matchLabels:
        app.kubernetes.io/name: flask-hello-world
    ...

  카피

• NameSpaceLabelSelector defines at the namespace level the auto-instrumented pods.

  Example using matchExpressions (select namespace containing an specific tag and value):

  ...
    namespaceLabelSelector:
      matchExpressions:
        - key: "kubernetes.io/metadata.name"
          operator: "In"
          values: ["backend"]
    ...

  카피

  TIP

  Keep in mind that apply the kubernetes.io/metadata.name label is the same as selecting based on the namespace's name.

Both selectors support matchLabel and matchExpressions.

...
  selector:
    matchLabels:
      tier: frontend
    matchExpressions:
      - {key: name, operator: In, values: [payroll, web]}
      - {key: environment, operator: NotIn, values: [dev]}
  ...

APM agent

You've got to specify the APM agent and its version within the instrumentation CR. We recommend using the latest version to take advantage of the newest features available.

See this example:

...
  spec:
    agent:
      language: dotnet
      image: newrelic/newrelic-dotnet-init:latest
  ...

APM configuration parameters

The instrumentation CR provides the capability to inject environment variables in the pod to streamline the configuration of the APM agents. See this example:

...
  spec:
      env:
      # Example overriding the appName configuration by using a label of the pod
        - name: NEW_RELIC_APP_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['app.kubernetes.io/name']
  ...

In the above example, we show you how you can configure the agent settings globally using environment variables. See each agent's configuration documentation for available configuration options:

• Java
• Node
• Python
• .NET
• Ruby
• PHP

License keys (optional)

When you install it, a 라이선스 키 is created and it's the license by default. Follow these steps, if you need to send the APM telemetry to a different account:

• To create a secret containing a new license key, run this command:

  bash

  카피

  $
  kubectl create secret generic newrelic-key-secret \
  >
    --namespace my-monitored-namespace \
  >
    --from-literal=new_relic_license_key=<NEW RELIC INGEST LICENSE KEY>

• To reference the secret from the instrumentation CR, run this command:

  ...
  spec:
    licenseKeySecret: the-name-of-the-custom-secret
  ...

  카피

Instrumentation CR examples

apiVersion: newrelic.com/v1alpha2
kind: Instrumentation
metadata:
  name: newrelic-instrumentation-python
  namespace: newrelic
spec:
  agent:
    language: python
    image: newrelic/newrelic-python-init:latest
    env:
      - name: NEW_RELIC_APP_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['app']
  podLabelSelector:
    matchExpressions:
      - key: "app"
        operator: "In"
        values: ["flask-hello-world","flask-hello-world-v2"]

apiVersion: newrelic.com/v1alpha2
kind: Instrumentation
metadata:
  name: newrelic-instrumentation-java
  namespace: newrelic
spec:
  agent:
    language: java
    image: newrelic/newrelic-java-init:latest
  namespaceLabelSelector:
    matchExpressions:
      - key: "kubernetes.io/metadata.name"
        operator: "In"
        values: ["java"]

apiVersion: newrelic.com/v1alpha2
kind: Instrumentation
metadata:
  name: newrelic-instrumentation-ruby
  namespace: newrelic
spec:
  agent:
    language: ruby
    image: newrelic/newrelic-ruby-init:latest
  namespaceLabelSelector:
    matchExpressions:
      - key: "Ruby"
        operator: "Exists"
  licenseKeySecret: the-name-of-the-custom-secret

Update APM instrumentation in applications

By default, the Kubernetes APM auto-attach automatically installs the latest available version of the corresponding APM agent.

Once the monitoring of an application starts, it's not automatically updated to a newer version unless you choose to update. You can update the application by redeploying the pods or restarting your deployment.

Remove APM instrumentation in applications

To remove the APM instrumentation from an application, you must change the matching label selector inside either the podLabelSelector or namespaceLabelSelector used or delete the instrumentation CR. Then, restart the deployment. The remove process takes just a few seconds.

Update the Kubernetes APM auto-attach

Bundle installation

Run an update of the nri-bundle chart with the following parameter:

$
k8s-agents-operator.enabled=true

Standalone installation

Run the helm upgrade command to update to a newer version of the Kubernetes APM auto-attach.

$
helm upgrade k8s-agents-operator newrelic/k8s-agents-operator -n newrelic

Uninstalling the Kubernetes APM auto-attach

Bundle installation

Uninstall the nri-bundle chart or if you only want to remove Kubernetes APM auto-attach, run a helm upgrade with the following parameter:

$
k8s-agents-operator.enabled=false

Standalone installation

To uninstall and delete the Kubernetes APM auto-attach, run this command:

$
helm uninstall k8s-agents-operator -n newrelic

Find and use data

• Get insights of your applications and resolve incidents using the APM summary page.

• Check out the Kubernetes summary page. It provides Kubernetes insights in the context of your monitored applications.

Certificates

The Kubernetes APM auto-attach can support cert-manager if preferred.

• Run this command to install the cert-manager Helm chart:

  bash

  카피

  $
  helm install cert-manager jetstack/cert-manager \
  >
    --namespace cert-manager \
  >
    --create-namespace \
  >
    --set crds.enabled=true

• In your values.yaml file, set admissionWebhooks.autoGenerateCert.enabled: false and admissionWebhooks.certManager.enabled: true. Then, install the chart as normal.

Available chart releases

Run this command to see the available charts:

$
helm search repo k8s-agents-operator

Frequently asked questions

Troubleshooting

If your applications are not instrumented, you should check the following:

• Please be sure to redeploy or deploy new applications after you've installed Kubernetes APM auto-attach. Notice that only auto-instruments new applications are deployed in the cluster.

• Run this command to check that the secret is installed in the app's namespace:

  bash

  카피

  $
  kubectl get secrets -n NAMESPACE

• Check that the pod has the required labels that enable automatic instrumentation through CR when using podLabelSelector. Similarly, check that the namespace has the required labels when using namespaceLabelSelector inside the CR.

  bash

  카피

  $
  kubectl get pod POD_NAME -n NAMESPACE  -o jsonpath='{.metadata.annotations}'

• Run this command to get logs from the APM auto-attach pod:

  bash

  카피

  $
  kubectl logs AGENT_OPERATOR_POD -n newrelic

• Run this command to ensure the init container has been injected and sucessfully executed inside the application's pod.

  bash

  카피

  $
  kubectl describe pod POD_NAME -n NAMESPACE

How to migrate from previous versions that required annotations

Starting with version 0.14, annotations within the application deployment manifest are no longer necessary for applications to be auto-instrumented.

It's advised to uninstall any versions preceding 0.14 and proceed with the installation of the latest release. Utilizing the label selectors within the instrumentation CR will enable the precise deployment of APM agents, thereby obviating the requirement for annotations.

Support

Kubernetes APM auto-attach supports the following languages and their minimum supported versions according to our standard APM agent support policy:

• Java: 8.12
• .NET: 10.25
• Ruby: 9.10
• Node.js: 11.9
• Python: 9.10
• PHP: 11.12

For any issues:

• Review the issues section on GitHub for any similar problems or consider opening a new issue.

• You can reach out to the New Relic support team for assistance.