Amazon Managed Kafka (MSK) integration

Importante

Enable the AWS CloudWatch Metric Streams integration to monitor all CloudWatch metrics from your AWS services, including custom namespaces. Individual integrations are no longer our recommended option.

New Relic offers an integration for collecting your Amazon Web Services Managed Streaming for Apache Kafka data. This document explains how to activate this integration and describes the data that can be reported.

Activate integration

To enable this integration, follow standard procedures to connect AWS services to New Relic.

Configuration and polling

To change the polling frequency and filter data, use configuration options.

Default polling information for the Amazon Managed Kafka integration:

New Relic polling interval: 5 minutes
Amazon CloudWatch data interval: 1 minute

View and use data

To view your integration data, go to one.newrelic.com > All capabilities > Infrastructure > AWS and select an integration.

You can query and explore your data using the following event type:

Entity	Event Type	Provider
Cluster	`AwsMskClusterSample`	`AwsMskCluster`
Broker	`AwsMskBrokerSample`	`AwsMskBroker`
Topic	`AwsMskTopicSample`	`AwsMskTopic`

For more on how to use your data, see Understand and use integration data.

Metric data

This integration records Amazon Managed Kafka data for cluster, broker, and topic entities.

Managed Kafka Cluster data

Metric	Unit	Description
`activeControllerCount`	Count	Only one controller per cluster should be active at any given time.
`globalPartitionCount`	Count	Total number of partitions across all brokers in the cluster.
`globalTopicCount`	Count	Total number of topics across all brokers in the cluster.
`offlinePartitionsCount`	Count	Total number of partitions that are offline in the cluster.

Managed Kafka Broker data

Metric	Unit	Description
`bytesInPerSec`	BytesPerSecond	The number of bytes per second received from clients.
`bytesOutPerSec`	BytesPerSecond	The number of bytes per second sent to clients.
`cpuSystem`	Percent	The percentage of CPU in kernel space.
`cpuUser`	Percent	The percentage of CPU in user space.
`cpuIdle`	Percent	The percentage of CPU idle time.
`fetchConsumerLocalTimeMsMean`	Milliseconds	The mean time in milliseconds that the consumer request is processed at the leader.
`fetchConsumerRequestQueueTimeMsMean`	Milliseconds	The mean time in milliseconds that the consumer request waits in the request queue.
`fetchConsumerResponseQueueTimeMsMean`	Milliseconds	The mean time in milliseconds that the consumer request waits in the response queue.
`fetchConsumerResponseSendTimeMsMean`	Milliseconds	The mean time in milliseconds for the consumer to send a response.
`fetchConsumerTotalTimeMsMean`	Milliseconds	The mean total time in milliseconds that consumers spend on fetching data from the broker.
`fetchFollowerLocalTimeMsMean`	Milliseconds	The mean time in milliseconds that the follower request is processed at the leader.
`fetchFollowerRequestQueueTimeMsMean`	Milliseconds	The mean time in milliseconds that the follower request waits in the request queue.
`fetchFollowerResponseQueueTimeMsMean`	Milliseconds	The mean time in milliseconds that the follower request waits in the response queue.
`fetchFollowerResponseSendTimeMsMean`	Milliseconds	The mean time in milliseconds for the follower to send a response.
`fetchFollowerTotalTimeMsMean`	Milliseconds	The mean total time in milliseconds that followers spend on fetching data from the broker.
`fetchMessageConversionsPerSec`	CountPerSecond	The number of fetch message conversions per second for the broker.
`fetchThrottleTime`	Milliseconds	The average fetch throttle time in milliseconds.
`fetchThrottleByteRate`	BytesPerSecond	The number of throttled bytes per second.
`fetchThrottleQueueSize`	Count	The number of messages in the throttle queue.
`kafkaAppLogsDiskUsed`	Percent	The percentage of disk space used for application logs.
`kafkaDataLogsDiskUsed`	Percent	The percentage of disk space used for data logs.
`leaderCount`	Count	The number of leader replicas.
`memoryBuffered`	Bytes	The size in bytes of buffered memory for the broker.
`memoryCached`	Bytes	The size in bytes of cached memory for the broker.
`memoryFree`	Bytes	The size in bytes of memory that is free and available for the broker.
`memoryUsed`	Bytes	The size in bytes of memory that is in use for the broker.
`messagesInPerSec`	CountPerSecond	The number of incoming messages per second for the broker.
`networkProcessorAvgIdlePercent`	Percent	The average percentage of the time the network processors are idle.
`networkRxDropped`	Count	The number of dropped receive packages.
`networkRxErrors`	Count	The number of network receive errors for the broker.
`networkRxPackets`	Count	The number of packets received by the broker.
`networkTxDropped`	Count	The number of dropped transmit packages.
`networkTxErrors`	Count	The number of network transmit errors for the broker.
`networkTxPackets`	Count	The number of packets transmitted by the broker.
`partitionCount`	Count	The number of partitions for the broker.
`produceLocalTimeMsMean`	Milliseconds	The mean time in milliseconds for the follower to send a response.
`produceMessageConversionsPerSec`	CountPerSecond	The number of produce message conversions per second for the broker.
`produceMessageConversionsTimeMsMean`	Milliseconds	The mean time in milliseconds spent on message format conversions.
`produceRequestQueueTimeMsMean`	Milliseconds	The mean time in milliseconds that request messages spend in the queue.
`produceResponseQueueTimeMsMean`	Milliseconds	The mean time in milliseconds that response messages spend in the queue.
`produceResponseSendTimeMsMean`	Milliseconds	The mean time in milliseconds spent on sending response messages.
`produceTotalTimeMsMean`	Milliseconds	The mean produce time in milliseconds.
`produceThrottleTime`	Milliseconds	The average produce throttle time in milliseconds.
`produceThrottleByteRate`	BytesPerSecond	The number of throttled bytes per second.
`produceThrottleQueueSize`	Count	The number of messages in the throttle queue.
`requestBytesMean`	Count	The mean number of request bytes for the broker.
`requestExemptFromThrottleTime`	Milliseconds	The average time spent in broker network and I/O threads to process requests that are exempt from throttling.
`requestHandlerAvgIdlePercent`	Percent	The average percentage of the time the request handler threads are idle.
`requestThrottleQueueSize`	Count	The number of messages in the throttle queue.
`requestThrottleTime`	Milliseconds	The average request throttle time in milliseconds.
`requestTime`	Milliseconds	The average time spent in broker network and I/O threads to process requests.
`rootDiskUsed`	Percent	The percentage of the root disk used by the broker.
`swapUsed`	Bytes	The size in bytes of swap memory that is in use for the broker.
`swapFree`	Bytes	The size in bytes of swap memory that is available for the broker.
`underMinIsrPartitionCount`	Count	The number of under minIsr partitions for the broker.
`underReplicatedPartitions`	Count	The number of under-replicated partitions for the broker.
`zooKeeperRequestLatencyMsMean`	Milliseconds	Mean latency in milliseconds for ZooKeeper requests from broker.
`zooKeeperSessionState`	Other	Connection status of broker's ZooKeeper session which may be one of the following: NOT_CONNECTED: '0.0', ASSOCIATING: '0.1', CONNECTING: '0.5', CONNECTEDREADONLY: '0.8', CONNECTED: '1.0', CLOSED: '5.0', AUTH_FAILED: '10.0'.

Managed Kafka Topic data

Metric	Unit	Description
`bytesInPerSec`	BytesPerSecond	The number of bytes received per second.
`bytesOutPerSec`	BytesPerSecond	The number of bytes sent per second.
`fetchMessageConversionsPerSec`	CountPerSecond	The number of fetched messages converted per second.
`messagesInPerSec`	CountPerSecond	The number of messages received per second.
`produceMessageConversionsPerSec`	CountPerSecond	The number of conversions per second for produced messages.