New Relic integrations with the Google Cloud Platform (GCP) include one that reports Google Cloud Dataflow data to New Relic. This document explains how to activate the GCP Cloud Dataflow integration and describes the data it reports.
Features
Cloud Dataflow is Google Cloud's fully managed streaming and batch data-processing service based on Apache Beam. New Relic Dataflow integration collects worker resource, throughput, lag, and job-state metrics for each Dataflow job.
Activate integration
To enable the integration, follow standard procedures to connect your GCP service to New Relic:
Polling frequency
New Relic integrations query your GCP services according to a polling interval that varies by integration. The polling frequency for Google Cloud Dataflow is 5 minutes. The resolution is 1 data point every minute.
Important
Cloud Dataflow supports 1-minute polling intervals for WIF integrations in limited preview. To request access, contact New Relic support.
Workload Identity Federation
Find and use data
After you enable the integration, your Dataflow resources appear as entities in the New Relic entity explorer. To see dashboards and manage services, go to one.newrelic.com > All capabilities > Infrastructure > GCP.
All Dataflow metrics available in GCP Cloud Monitoring are collected as dimensional metrics in the Metric event type. Additional metrics beyond this table are collected automatically. See Google's Dataflow metrics documentation for the complete list.
Entities
Metric data
Key metrics — Job
Metric name | Unit | Description |
|---|---|---|
| Count | Number of vCPUs currently allocated to the Dataflow job. |
| Seconds | Total vCPU-seconds consumed by the Dataflow job. |
| Count | Number of elements produced by the Dataflow job, faceted by transform. |
| Seconds | Maximum time an element has been waiting in the Dataflow pipeline. |
| Seconds | Age of the data watermark for the Dataflow job. |
| Seconds | System lag observed at each stage of the Dataflow job. |
| Count | User-defined counters emitted by the Dataflow job, faceted by counter name. |
For the complete list of Dataflow metrics, see Google's Dataflow metrics documentation.
Service account or user account
Find and use data
To find your integration data, go to one.newrelic.com > All capabilities > Infrastructure > GCP and select an integration.
Data is attached to the following event type:
Entity | Event Type | Provider |
|---|---|---|
Job |
|
|
For more on how to use your data, see Understand and use integration data.
Metric data
This integration collects GCP Dataflow data for Job.
Dataflow Job data
Metric | Unit | Description |
|---|---|---|
| Bytes | The billable bytes of shuffle data processed by this Dataflow job. |
| Count | The number of vCPUs currently being used by this Dataflow job. This is the current number of workers times the number of vCPUs per worker. |
| Count | The current shuffle slots used by this Dataflow job. |
| Seconds | The age (time since event timestamp) up to which all data has been processed by the pipeline. |
| Seconds | Duration that the current run of this pipeline has been in the Running state so far, in seconds. When a run completes, this stays at the duration of that run until the next run starts. |
| Count | Number of elements added to the pcollection so far. |
| Bytes | An estimated number of bytes added to the pcollection so far. Dataflow calculates the average encoded size of elements in a pcollection and mutiplies it by the number of elements. |
| Count | Has this job failed. |
| Seconds | The age (time since event timestamp) up to which all data has been processed by this stage of the pipeline. |
| Seconds | The current maximum duration that an item of data has been processing or awaiting processing in seconds, per pipeline stage. |
| Seconds | The current maximum duration that an item of data has been processing or awaiting processing, in seconds. |
| Other | The total GB seconds of memory allocated to this Dataflow job. |
| Other | The total GB seconds for all persistent disk used by all workers associated with this Dataflow job. |
| Bytes | The total bytes of shuffle data processed by this Dataflow job. |
| Bytes | The total bytes of streaming data processed by this Dataflow job. |
| Seconds | The total vCPU seconds used by this Dataflow job. |
| Count | A user-defined counter metric. |