AWS Kinesis Data Analytics monitoring integration

Important

Enable the AWS CloudWatch Metric Streams integration to monitor all CloudWatch metrics from your AWS services, including custom namespaces. Individual integrations are no longer our recommended option.

Our infrastructure monitoring integrations include an integration for reporting your AWS Kinesis Data Analytics data to our products. Read on to learn more about how to activate the integration and what data it collects.

Activate integration

To enable this integration follow standard procedures to Connect AWS services to New Relic.

Configuration and polling

You can change the polling frequency and filter data using configuration options.

Default polling information for the AWS Kinesis Data Analytics integration:

New Relic polling interval: 5 minutes
Amazon CloudWatch data interval: 1 minute

Find and use data

To find your integration data in our infrastructure UI, go to one.newrelic.com > All capabilities > Infrastructure > AWS and select an integration.

Data is attached to the following event type:

Entity	Event Type	Provider
Application	`AwsKinesisAnalyticsApplicationSample`	`AwsKinesisAnalyticsApplication`
ApplicationKpu	`AwsKinesisAnalyticsApplicationSample`	`AwsKinesisAnalyticsApplication`
Task	`AwsKinesisAnalyticsTaskSample`	`AwsKinesisAnalyticsTask`
TaskOperator	`AwsKinesisAnalyticsTaskOperatorSample`	`AwsKinesisAnalyticsTaskOperator`
TaskParallelism	`AwsKinesisAnalyticsTaskParallelismSample`	`AwsKinesisAnalyticsTaskParallelism`
TaskOperatorParallelism	`AwsKinesisAnalyticsTaskOperatorParallelismSample`	`AwsKinesisAnalyticsTaskOperatorParallelism`

For more on how to use your data, see Understand and use integration data.

Metric data

This integration collects AWS Kinesis Data Analytics data for Application, ApplicationKpu, Task, TaskOperator, TaskParallelism and TaskOperatorParallelism.

Kinesis Data Analytics Application data

Metric	Unit	Description
`bytesConsumedRate`	BytesPerSecond	The average number of bytes consumed per second for a topic.
`commitsFailed`	Count	The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.
`commitsSucceeded`	Count	The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.
`cpuUtilization`	Percent	Overall percentage of CPU utilization across task managers. For example, if there are five task managers, Kinesis Data Analytics publishes five samples of this metric per reporting interval.
`currentInputWatermark`	Milliseconds	The last watermark this application/operator/task/thread has received.
`currentOutputWatermark`	Milliseconds	The last watermark this application/operator/task/thread has emitted.
`downtime`	Milliseconds	For jobs currently in a failing/recovering situation, the time elapsed during this outage.
`fullRestarts`	Count	The total number of times this job has fully restarted since it was submitted. This metric does not measure fine-grained restarts.
`heapMemoryUtilization`	Percent	Overall heap memory utilization across task managers. For example, if there are five task managers, Kinesis Data Analytics publishes five samples of this metric per reporting interval.
`lastCheckpointDuration`	Milliseconds	The time it took to complete the last checkpoint.
`lastCheckpointSize`	Bytes	The total size of the last checkpoint.
`numRecordsInPerSecond`	CountPerSecond	The total number of records this operator or task has received per second.
`numRecordsOut`	Count	The total number of records this operator or task has emitted.
`numRecordsOutPerSecond`	CountPerSecond	The total number of records this operator or task has emitted per second.
`numberOfFailedCheckpoints`	Count	The number of times checkpointing has failed.
`numRecordsIn`	Count	The total number of records this operator or task has received.
`oldGenerationGcCount`	Count	The total number of old garbage collection operations that have occurred across all task managers.
`oldGenerationGcTime`	Milliseconds	The total time spent performing old garbage collection operations.
`recordsLagMax`	Count	The maximum lag in terms of number of records for any partition in this window.
`threadsCount`	Count	The total number of live threads used by the application.
`uptime`	Milliseconds	The time that the job has been running without interruption.
`bytes`	Bytes	The number of bytes read (per input stream) or written (per output stream).
`inputProcessing.DroppedRecords`	Count	The number of records returned by a Lambda function that were marked with Dropped status.
`inputProcessing.Duration`	Milliseconds	The time taken for each AWS Lambda function invocation performed by Kinesis Data Analytics.
`inputProcessing.OkBytes`	Count	The sum of bytes of the records returned by a Lambda function that were marked with Ok status.
`inputProcessing.OkRecords`	Count	The number of records returned by a Lambda function that were marked with Ok status.
`inputProcessing.ProcessingFailedRecords`	Count	The number of records returned by a Lambda function that were marked with ProcessingFailed status.
`inputProcessing.Success`	Count	The number of successful Lambda invocations by Kinesis Data Analytics.
`lambdaDelivery.DeliveryFailedRecords`	Count	The number of records returned by a Lambda function that were marked with DeliveryFailed status.
`lambdaDelivery.Duration`	Milliseconds	The time taken for each Lambda function invocation performed by Kinesis Data Analytics.
`lambdaDelivery.OkRecords`	Count	The number of records returned by a Lambda function that were marked with Ok status.
`millisBehindLatest`	Milliseconds	Indicates how far behind from the current time an application is reading from the streaming source.
`records`	Count	The number of records read (per input stream) or written (per output stream).
`success`	Count	1 for each successful delivery attempt to the destination configured for your application; 0 for every failed delivery attempt. The average value of this metric indicates how many successful deliveries are performed.
`bytesRequestedPerFetch`	Bytes	The bytes requested in a single call to getRecords.
`committedOffsets`	Other	The last successfully committed offsets to Kafka, for each partition. A particular partition's metric can be specified by topic name and partition id.
`currentOffsets`	Other	The consumer's current read offset, for each partition. A particular partition's metric can be specified by topic name and partition id.

Kinesis Data Analytics ApplicationKpu data

Metric	Unit	Description
`KPUs`	Count	The number of Kinesis Processing Units (KPUs) currently in use. Known issue: as this metric has a longer lag than the rest, in some cases, it's not shown

Metric

Unit

Description

KPUs

Count

The number of Kinesis Processing Units (KPUs) currently in use.

Known issue: as this metric has a longer lag than the rest, in some cases, it's not shown

Kinesis Data Analytics Task data

Metric	Unit	Description
`commitsFailed`	Count	The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.
`commitsSucceeded`	Count	The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.
`currentInputWatermark`	Milliseconds	The last watermark this task has received.
`currentOutputWatermark`	Milliseconds	The currently output watermark.
`numLateRecordsDropped`	Count	The number of records this task has dropped due to arriving late.
`numRecordsIn`	Count	The total number of records this task has received.
`numRecordsInPerSecond`	CountPerSecond	The total number of records this task has received per second.
`numRecordsOut`	Count	The total number of records this task has emitted.
`numRecordsOutPerSecond`	CountPerSecond	The total number of records this task has emitted per second.
`recordsLagMax`	Count	The maximum lag in terms of number of records for any partition in this window.

Kinesis Data Analytics TaskOperator data

Metric	Unit	Description
`bytesConsumedRate`	BytesPerSecond	The average number of bytes consumed per second for a topic.
`commitsFailed`	Count	The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.
`commitsSucceeded`	Count	The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.
`currentInputWatermark`	Milliseconds	The last watermark this task has received.
`currentOutputWatermark`	Milliseconds	The last watermark this application/operator/task/thread has emitted.
`numLateRecordsDropped`	Count	The number of records this operator or task has dropped due to arriving late.
`numRecordsIn`	Count	The total number of records this task has received.
`numRecordsInPerSecond`	CountPerSecond	The total number of records this task has received per second.
`numRecordsOut`	Count	The total number of records this task has emitted.
`numRecordsOutPerSecond`	CountPerSecond	The total number of records this task has emitted per second.
`recordsLagMax`	Count	The maximum lag in terms of number of records for any partition in this window.

Kinesis Data Analytics TaskParallelism data

Metric	Unit	Description
`currentInputWatermark`	Milliseconds	The last watermark this application/operator/task/thread has received.
`numLateRecordsDropped`	Count	The number of records this task has dropped due to arriving late.
`numRecordsIn`	Count	The total number of records this task has received.
`numRecordsInPerSecond`	CountPerSecond	The total number of records this task has received per second.
`numRecordsOut`	Count	The total number of records this task has emitted.
`numRecordsOutPerSecond`	CountPerSecond	The total number of records this task has emitted per second.

Kinesis Data Analytics TaskOperatorParallelism data

Metric	Unit	Description
`bytesConsumedRate`	BytesPerSecond	The average number of bytes consumed per second for a topic.
`commitsFailed`	Count	The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.
`commitsSucceeded`	Count	The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.
`currentInputWatermark`	Milliseconds	The last watermark this application/operator/task/thread has received.
`currentOutputWatermark`	Milliseconds	The last watermark this application/operator/task/thread has emitted.
`numRecordsIn`	Count	The total number of records this operator or task has received.
`numRecordsInPerSecond`	CountPerSecond	The total number of records this operator or task has received per second.
`numRecordsOut`	Count	The total number of records this operator or task has emitted.
`numRecordsOutPerSecond`	CountPerSecond	The total number of records this operator or task has emitted per second.
`recordsLagMax`	Count	The maximum lag in terms of number of records for any partition in this window.