BigQuery is Google Cloud's serverless data warehouse. New Relic BigQuery integration collects query, execution, storage, and slot utilization metrics at both project and dataset scope.
New Relic integrations query your GCP services according to a polling interval that varies by integration. The polling frequency for Google BigQuery is 5 minutes. The resolution is 1 data point every minute.
Important
BigQuery supports 1-minute polling intervals for WIF integrations in limited preview. To request access, contact New Relic support.
Activate Table metrics
The collection of BigQuery table metrics is disabled by default. This affects the metrics listed under GcpBigQueryTableSample. To collect those metrics, turn on the "Enable table metrics" setting under the configuration menu of the BigQuery integration.
Workload Identity Federation
Find and use data
After you enable the integration, your BigQuery resources appear as entities in the New Relic entity explorer. To see dashboards and manage services, go to one.newrelic.com > All capabilities > Infrastructure > GCP.
All BigQuery metrics available in GCP Cloud Monitoring are collected as dimensional metrics in the Metric event type. Additional metrics beyond this table are collected automatically. See Google's BigQuery metrics documentation for the complete list.
Entities
Entity
Entity type
Resource type
Project
GCPBIGQUERYPROJECT
bigquery_project
Dataset
GCPBIGQUERYDATASET
bigquery_dataset
Table
GCPBIGQUERYTABLE
bigquery_table
Metric data
Key metrics — Project
Metric name
Unit
Description
gcp.bigquery.query.count
Count
Number of queries executed in the project, faceted by query_type and state.
gcp.bigquery.query.execution_times
Seconds
Query execution time distribution across the project.
gcp.bigquery.slots.allocated_for_project
Count
Number of slots currently allocated to the project.
gcp.bigquery.storage.uploaded_bytes
Bytes
Bytes uploaded to BigQuery in the project.
gcp.bigquery.storage.uploaded_row_count
Count
Number of rows uploaded to BigQuery in the project.
Key metrics — Dataset
Metric name
Unit
Description
gcp.bigquery.storage.stored_bytes
Bytes
Total bytes stored in the dataset.
gcp.bigquery.storage.table_count
Count
Number of tables in the dataset.
gcp.bigquery.storage.stored_logical_bytes
Bytes
Logical bytes stored in the dataset (uncompressed size).
gcp.bigquery.storage.stored_physical_bytes
Bytes
Physical bytes stored in the dataset (compressed on-disk size).
Key metrics — Table
Metric name
Unit
Description
gcp.bigquery.storage.stored_bytes
Bytes
Number of logical bytes stored in the table. GCP labels the top 100 tables per dataset (by size) with a table label on this metric; the rest are grouped together with no table label.
gcp.bigquery.storage.uploaded_bytes
Bytes
Bytes uploaded to the table.
gcp.bigquery.storage.uploaded_bytes_billed
Bytes
Bytes billed for data uploaded to the table. Currently only the streaming API is billed.
New Relic also collects metrics for the following BigQuery resource types. These resource types don't create entities in the entity explorer, so query their metrics directly on the Metric event type.
Metric name
Unit
Description
gcp.bigquery.query.count
Count
In flight queries.
gcp.bigquery.query.execution_times
Seconds
Distribution of execution times for queries that completed successfully within the last sampling interval. Incomplete and failed queries are not included.
gcp.bigquery.query.scanned_bytes
Bytes
Scanned bytes.
gcp.bigquery.query.scanned_bytes_billed
Bytes
Scanned bytes billed.
gcp.bigquery.slots.allocated
Count
Number of BigQuery slots currently allocated for project. Slot allocation can be broken down based on reservation and job type. The accuracy of this metric can be poor, especially the break-down by reservation. Consider using resource charts instead.
gcp.bigquery.slots.allocated_for_project
Count
Number of BigQuery slots currently allocated for query jobs in the project.
Number of BigQuery slots currently allocated across projects in the reservation. Note that the metric data is only reported while at least one project has been assigned to the reservation and is consuming slots. As an alternative, consider querying reservations information from INFORMATION_SCHEMA, which doesn't have these limitations.
gcp.bigquery.slots.total_available
Count
(Deprecated) Total number of BigQuery slots available for the project. If you are using the BigQuery Reservations API, consider using bigquery.googleapis.com/slots/assigned and bigquery.googleapis.com/slots/max_assigned instead.
Metric name
Unit
Description
gcp.bigquery.job.continuous.backlog_records
Count
The estimated number of backlog records for each stage of the continuous job.
The estimated number of bytes processed for each stage of the continuous job.
gcp.bigquery.job.continuous.output_watermark
Microseconds
The most recent timestamp, in microseconds since the epoch, up to which all data has been processed by this stage of the continuous job.
gcp.bigquery.job.continuous.records_read
Count
The number of input records read for each stage of the continuous job.
gcp.bigquery.job.continuous.records_written
Count
The number of output records written for each stage of the continuous job.
gcp.bigquery.job.continuous.slots.used_time
Milliseconds
The total slot milliseconds used by the continuous job.
Metric name
Unit
Description
gcp.bigquerystorage.write.concurrent_connections
Count
Concurrent connection per project for BigQuery Write API. To be consistent with concurrent connection quota, the metric is by default on the project that initiated the AppendRowsRequests (associated with the API key or the service account).
To view metric data for your Google BigQuery integration in New Relic, create NRQL queries for GcpBigQueryProjectSample, GcpBigQueryDatasetSample, and GcpBigQueryTableSampleevents and their attributes.
GcpBigQueryProjectSample
Query GcpBigQueryProjectSample events in New Relic to view data for the following metrics. Resolution is one data point per minute with an 8-minute delay.
Attribute
Description
query.Count
In-flight queries.
query.ExecutionTimes
Distribution of query execution times.
slots.Allocated
Number of slots currently allocated for project.
slots.AllocatedForProject
Number of slots currently allocated for query jobs in the project.
slots.AllocatedForProjectAndJobType
Number of slots currently allocated for the project and job type.
slots.AllocatedForReservation
Number of slots currently allocated for project in the reservation.
slots.TotalAllocatedForReservation
Number of slots currently allocated across projects in the reservation.
slots.TotalAvailable
Total number of slots available for the project.
GcpBigQueryDatasetSample
Query GcpBigQueryDatasetSample events in New Relic to view data for the following metrics. Resolution is one data point every 30 minutes with a 3-hour delay.
Attribute
Description
storage.StoredBytes
Amount of stored bytes.
storage.Tables
Number of tables.
GcpBigQueryTableSample
Query GcpBigQueryTableSample events to view data for the following metrics. Resolution is one data point per minute with a 6-hour delay.
Attribute
Description
storage.UploadedBytes
Uploaded bytes.
storage.UploadedBytesBilled
Uploaded bytes billed. Currently only streaming API is billed.