What BigQuery API method is being referred to in error message? - google-bigquery

Question

We have a Continuous Integration (CI) pipeline that creates resources in GCP, runs some tests upon them, then destroys those resources. This pipeline runs tens of times per day. Some of the resources are BigQuery datasets. We use terraform to create and destroy those resources.
Occasionally the destroying of these resources fails with error:
Error: Error when reading or editing Resource projects/projectname/datasets/p00005167_datasetname for IAM Member (role "serviceAccount:sa-name#projectname.iam.gserviceaccount.com", "projects/projectname/roles/rolename"): Error applying IAM policy for Bigquery Dataset projectname/p00005167_datasetname: Error creating DatasetAccess: googleapi: Error 403: Exceeded rate limits: too many dataset metadata update operations for this dataset. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas, rateLimitExceeded
I can see the same errors appearing in Logs Explorer which are of the form:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 8,
"message": "Exceeded rate limits: too many dataset metadata update operations for this dataset. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas"
},
"authenticationInfo": {
"principalEmail": "deployer-dev#projectname.iam.gserviceaccount.com",
"serviceAccountDelegationInfo": [
{
"firstPartyPrincipal": {
"principalEmail": "deployer-dev#projectname.iam.gserviceaccount.com"
}
}
]
},
"requestMetadata": {
"callerIp": "10.51.0.118",
"callerSuppliedUserAgent": "Terraform/0.14.7 (+https://www.terraform.io) Terraform-Plugin-SDK/2.5.0 terraform-provider-google/3.84.0,gzip(gfe)",
"callerNetwork": "//compute.googleapis.com/projects/projectname/global/networks/__unknown__",
"requestAttributes": {},
"destinationAttributes": {}
},
"serviceName": "bigquery.googleapis.com",
"methodName": "datasetservice.update",
"authorizationInfo": [
{
"resource": "projects/projectname/datasets/p00005167_datasetname",
"permission": "bigquery.datasets.update",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/projectname/datasets/p00005167_datasetname",
"serviceData": {
"#type": "type.googleapis.com/google.cloud.bigquery.logging.v1.AuditData",
"datasetUpdateRequest": {
"resource": {
"datasetName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname"
},
"info": {
"friendlyName": "currents",
"description": "blah",
},
"acl": {
"entries": [
{
"role": "projects/projectname/roles/rolename",
"userEmail": "sa-name#projectname.iam.gserviceaccount.com",
"specialGroup": "PROJECT_OWNERS",
"viewName": {}
},
{
"role": "WRITER",
"specialGroup": "PROJECT_WRITERS",
"viewName": {}
},
{
"role": "OWNER",
"specialGroup": "PROJECT_OWNERS",
"viewName": {}
},
{
"role": "OWNER",
"userEmail": "deployer-dev#projectname.iam.gserviceaccount.com",
"specialGroup": "PROJECT_OWNERS",
"viewName": {}
},
{
"role": "READER",
"specialGroup": "PROJECT_READERS",
"viewName": {}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersBehaviorsSubscriptiongroupStatechange_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersCampaignsEnrollincontrol_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersBehaviorsCustomevent_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesContentcardClick_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesSmsInboundreceive_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesPushnotificationSend_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesPushnotificationBounce_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesSmsDeliveryfailure_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersBehaviorsInstallattribution_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesEmailBounce_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersBehaviorsAppNewsfeedimpression_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersBehaviorsLocation_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesEmailUnsubscribe_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesInappmessageClick_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesContentcardImpression_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesEmailSend_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesEmailClick_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersBehaviorsUninstall_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesEmailMarkasSpam_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersCampaignsConversion_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesSmsRejection_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesSmsSend_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesInappmessageImpression_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesContentcardDismiss_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesEmailDelivery_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesWebhookSend_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersBehaviorsPurchase_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesPushnotificationIosforeground_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesNewsfeedcardClick_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersCanvasEntry_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesSmsDelivery_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesEmailSoftbounce_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersBehaviorsAppFirstsession_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesNewsfeedcardImpression_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersMessagesEmailOpen_red_v"
}
},
{
"specialGroup": "PROJECT_OWNERS",
"viewName": {
"projectId": "projectname",
"datasetId": "p00005167_datasetname_presentation_red",
"tableId": "UsersBehaviorsAppSessionstart_red_v"
}
}
]
}
}
},
"datasetUpdateResponse": {
"resource": {
"datasetName": {},
"info": {},
"acl": {}
}
}
}
},
"insertId": "ivcvsse1mrrw",
"resource": {
"type": "bigquery_resource",
"labels": {
"project_id": "projectname"
}
},
"timestamp": "2021-10-15T14:25:33.764732Z",
"severity": "ERROR",
"logName": "projects/projectname/logs/cloudaudit.googleapis.com%2Factivity",
"receiveTimestamp": "2021-10-15T14:25:34.287118294Z"
}
I have a number of avenues of investigation I'm pursuing to try and get to the bottom of this however one thing I'd like to clarify is the API method that is being called. The log message states:
"serviceName": "bigquery.googleapis.com",
"methodName": "datasetservice.update",
however if I search for datasetservice.update the closest match I find is Method: datasets.update and the required response body as documented at that URL doesn't seem to match the response body in the log message. For example, the protoPayload.serviceData.datasetUpdateRequest object in the log message contains an object called acl however there is no such object documented in the response body at Method: datasets.update .
Hence I'm slightly confused. Is it correct to say that Method: datasets.update is the API method being called here?

Answer 1

Your suspicion is correct.
The methodName is reporting internal details as part of the audit log entry, namely that there's a datasetService. It services the datasets.* methods: https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets
My terraform-fu is pretty weak as I rarely interact with it, but this may represent some underspecified dependencies; it may be that terraform is making a bunch of calls that partially update the access list for example. A series of repeated permission updates to the dataset would be sufficient to trigger the ratelimit error.