New Relic integrations with the Google Cloud Platform (GCP) include one that reports Google Cloud Composer data to New Relic. This document explains how to activate the GCP Cloud Composer integration and describes the data it reports.
Features
Cloud Composer is Google Cloud's fully managed workflow orchestration service built on Apache Airflow. New Relic Cloud Composer integration collects environment, workload, and workflow metrics — DAG runs, task instances, scheduler heartbeats, and worker resource usage.
Activate integration
To enable the integration, follow standard procedures to connect your GCP service to New Relic:
Polling frequency
New Relic integrations query your GCP services according to a polling interval that varies by integration. The polling frequency for Google Cloud Composer is 5 minutes. The resolution is 1 data point every minute.
Workload Identity Federation
Find and use data
After you enable the integration, your Cloud Composer resources appear as entities in the New Relic entity explorer. To see dashboards and manage services, go to one.newrelic.com > All capabilities > Infrastructure > GCP.
All Cloud Composer metrics available in GCP Cloud Monitoring are collected as dimensional metrics in the Metric event type. Additional metrics beyond this table are collected automatically. See Google's Cloud Composer metrics documentation for the complete list.
Entities
Metric data
Key metrics — Environment
Metric name | Unit | Description |
|---|---|---|
| Boolean | Whether the Composer environment is currently healthy. |
| Seconds | Total time spent parsing DAGs in the environment. |
| Percent | CPU utilization of the environment's metadata database. |
| Percent | Memory utilization of the environment's metadata database. |
| Count | Number of worker pod evictions in the environment. |
Key metrics — Workflow
Metric name | Unit | Description |
|---|---|---|
| Count | Number of DAG runs, faceted by workflow state. |
| Seconds | Duration of DAG runs across the workflow. |
| Count | Number of task instances across the workflow, faceted by state. |
| Seconds | Task instance execution time distribution. |
Key metrics — Workload
Metric name | Unit | Description |
|---|---|---|
| Seconds | CPU-seconds consumed by the Composer workload. |
| Bytes | Memory currently in use by the Composer workload. |
| Count | Number of pods backing the Composer workload. |
For the complete list of Cloud Composer metrics, see Google's Cloud Composer metrics documentation.
Service account or user account
Find and use data
To find your integration data, go to one.newrelic.com > All capabilities > Infrastructure > GCP and select the integration.
Data is attached to the following event types:
Entity | Event Type | Provider |
|---|---|---|
Environment |
|
|
Workflow |
|
|
For more on how to use your data, see Understand and use integration data.
Metric data
This integration collects GCP Composer data for Environment and Workflow.
Composer Environment data
Metric | Unit | Description |
|---|---|---|
| Count | Number of Composer API requests seen so far. |
| Milliseconds | Distribution of Composer API call latencies. |
| Count | Number of errors raised during parsing DAG files. |
| Count | Number of currently running DAG parsing processes. |
| Count | Number of file processors terminated due to processing timeout. |
| Seconds | Number of seconds taken to scan and import all DAG files once. |
| Count | The current dag bag size. |
| Count | Number of cores reserved for the database instance. |
| Seconds | CPU usage time of the database instance. |
| Count | CPU utilization ratio (from 0.0 to 1.0) of the database instance. |
| Bytes | Used disk space in bytes on the database instance. |
| Bytes | Maximum data disk size in bytes of the database instance. |
| Count | Disk quota usage ratio (from 0.0 to 1.0) of the database instance. |
| Bytes | Memory usage of the database instance in bytes. |
| Bytes | Maximum RAM size in bytes of the database instance. |
| Count | Memory utilization ratio (from 0.0 to 1.0) of the database instance. |
| Count | Number of open slots on executor. |
| Count | Number of running tasks on executor. |
| Count | Overall task instances. |
| Count | Number of Celery workers. |
| Count | Scheduler heartbeats. |
| Count | Number of tasks in queue. |
| Count | The number of Airflow worker pods evictions. |
| Count | Number of zombie tasks killed. |
Composer Workflow data
Metric | Unit | Description |
|---|---|---|
| Count | Number of workflow runs completed so far. |
| Seconds | Duration of workflow run completion. |
| Count | Number of workflow tasks completed so far. |
| Seconds | Duration of task completion. |