Secrets management

With secrets management, you can configure the agent and on-host integrations to use sensitive data (such as passwords) without having to write them as plain text into the configuration files. Currently, Hashicorp Vault, AWS KMS, CyberArk, New Relic CLI obfuscation and custom commands are supported.

Learn more in this NerdBytes video (5:12 minutes).

Define secrets

To use secrets in a configuration YAML file:

1. Define a variables section, where each entry is a name for a secret object.
2. In each entry, include the source of the secret and the proper configuration to retrieve those secrets.
3. In the general configuration section, set ${variable.property} placeholders that will be automatically replaced by the properties of the secret object. The secret object can be defined as a simple string or json object.

For example, the following configuration will retrieve an object named creds from Vault (you can define the object's name for the secret.) Let's assume that the stored object is a valid JSON with a property named user and another property named password. We want to use them to set the basic HTTP credentials of the status_url property from an Nginx on-host integration:

variables:
  creds:
    vault:
      http:
        url: http://my.vault.host/v1/newengine/data/secret
        headers:
          X-Vault-Token: my-vault-token
integrations:
  - name: nri-nginx
    env:
      METRICS: "true"
      STATUS_URL: http://${creds.user}:${creds.password}@example.com/status
      STATUS_MODULE: discover
      REMOTE_MONITORING: true
    interval: 30s
    labels:
      env: production
      role: load_balancer

Using environment variables

Environment variables can be used into the variables section with the {{MY_ENV_VAR}} notation. When doing so, environment variables are expanded and their value is replaced at runtime.

Use this method to avoid having sensitive values such as tokens or obfuscation keys included in the configuration file.

When using environment variables in on-host integration configuration files the passthrough_environment setting must be defined.

Secrets variables

Define secrets in each configuration under a variables section. Each entry is a user-defined secret name that will store the properties of the retrieved secrets. Each variable can contain the following properties:

AWS KMS secrets

To retrieve your secrets from Amazon KMS, you can set the following properties in your aws-kms section. Not all fields are required. For example, you will need either data, file, or http to provide the encoded KMS string.

The following example will allow retrieving a plain password string from AWS KMS. It must be decrypted from the provided data encoded string.

variables:
  myPassword:
    aws-kms:
      data: T0hBSStGTEVY
      region: ap-southeast-2
      credential_file: "./my-aws-credentials-file"
      config_file: "./my-aws-config-file"
      type: plain

Vault secrets

Vault must enable an http field containing the HTTP configuration used to connect to Vault. The http entry can contain the following entries:

tls_config properties

The following example will retrieve a secret using a Vault token from a secured server, and skip the server certificates verification:

variables:
  mydata:
    vault:
      http:
        url: https://my.vault.host/v1/newengine/data/secret
        headers:
          X-Vault-Token: my-vault-token
        tls_config:
          insecure_skip_verify: true

CyberArk command line interface

To retrieve secrets from the CyberArk command line interface (CLI) use the following configuration, all keys are required

The following example will retrieve a CyberArk secret using the command line interface:

variables:
  credentials:
    cyberark-cli:
      cli: /full/path/to/clipasswordsdk
      app-id: my-appid
      safe: my-safe
      folder: my-folder
      object: my-object

CyberArk REST API

To retrieve secrets using the CyberArk REST API there must be a http key containing the HTTP configuration. The http key contains these sub-keys, only url is required:

The following example will retrieve a secret using the CyberArk REST API, skipping server certificate verification:

variables:
  credentials:
    cyberark-api:
      http:
        url: https://hostname/AIMWebService/api/Accounts?AppID=myAppID&Query=Safe=mySafe;Object=myObject
        tls_config:
          insecure_skip_verify: true

Referring to the above example, call the username and password like this:

USERNAME: ${credentials.user}
   PASSWORD: ${credentials.password}

Note: For Microsoft SQL Server, if you use a Windows Authentication user login, you must prefix the USERNAME with your domain. For example:

USERNAME: <domain>\${credentials.user}

New Relic CLI Obfuscation

You can use the New Relic CLI obfuscate command to obscure sensitive information in the infrastructure agent or any on-host integration configuration file.

Steps:

• Install the New Relic CLI on any host (it can be your development host).
• Run the CLI obfuscate command in order to generate the obfuscated value:

newrelic agent config obfuscate --value '<plain_text_config_value>' --key '<obfuscation_key>'

• Copy the result of the cli command into the text argument in the obfuscated section as shown in the examples below.
• You can obfuscate JSON blocks if you need to reference multiple values in your configuration YML file. Note that you may need to escape the quotation marks "" with backslashes on Windows hosts. For example: '{\"attribute\":\"value\".... In addition, use Powershell on Windows hosts, rather than the Command Prompt window, since the Command Prompt window can handle quotation marks in unexpected ways.

newrelic agent config obfuscate --value '{"attribute":"value","attribute2":"value2"}' --key '<obfuscation_key>'

Integrations configuration example

The following example will allow retrieving the Nginx user and password that has been obfuscated using the New Relic CLI. This example uses dot notation when retrieving the credentials, since the obfuscated value was a JSON block:

variables:
  creds:
    obfuscated:
      key: 'random_key_used_in_cli'
      secret: 'obscured_output_from_cli'
integrations:
  - name: nri-nginx
    env:
      METRICS: "true"
      STATUS_URL: http://${creds.user}:${creds.password}@example.com/status
      STATUS_MODULE: discover
      REMOTE_MONITORING: true
    interval: 30s
    labels:
      env: production
      role: load_balancer

It's recommended to use environment variables for the obfuscation arguments as shown in the example below:

variables:
  creds:
    obfuscated:
      key: {{OBFUSCATION_KEY}}
      secret: {{OBFUSCATION_TEXT}}
integrations:
  - name: nri-nginx
    env:
      METRICS: "true"
      STATUS_URL: http://${creds.user}:${creds.password}@example.com/status
      STATUS_MODULE: discover
      REMOTE_MONITORING: true
    interval: 30s
    labels:
      env: production
      role: load_balancer

Agent configuration example

This example allows retrieving a string that contains the proxy details (user, password and host):

variables:
  obfuscated_proxy:
    obfuscated:
      key: 'random_key_used_in_cli'
      secret: 'obscured_output_from_cli'

proxy: ${obfuscated_proxy}

Custom Commands

You can load agent or integrations configurations from custom commands available in the same host. The infrastructure agent will detect when configurations should initially be loaded from a command output and keep values updated based on a ttl refresh time.

The expected output format for the commmand is either JSON or String:

JSON

The following JSON structure is expected as the main output (stdout) from the defined command:

{
  "data": {
    "myKey": "myValue"
  }, 
  "ttl": "24h"
}

• A top-level data field is required.
• A top-level ttl field is required.

It can then be used in any configuration file using the ${myVariable.key} notation. In the following example, a my-custom-auth command is executed with a domain parameter then loading the resulting token as an agent configuration:

variables:
  myToken:
    command:
      path: '/path/to/my-custom-auth-json'
      args: ["--domain", "myDomain", "--other_param", "otherValue"] 
      ttl: 24h
      
# my-custom-auth output example: {"data":{"token":"XXXXYYYYZZZZ"}, "ttl": "24h"}
# the `token` result can be included as ${myToken.token} in any configuration.
  
http:
  headers: 
    "Proxy-Authorization": ${myToken.token}

String (text/plain)

If the command output is a plain string, then it can be referenced in the configuration using the ${myVar} notation.

Example:

variables:
  myToken:
    command:
      path: '/path/to/my-custom-auth-string'
      args: ["arg1", "arg2"] 
      ttl: 24h
      
# my-custom-auth output example: "XXXXYYYYZZZZ"
# the `token` result can be included as ${myToken} on any configuration.
  
http:
  headers: 
    "Proxy-Authorization": ${myToken}