I am trying to use the bq command line tool to load data into BigQuery from GCS bucket and I receive the following error message:
BigQuery error in load operation: Access Denied: Job mythical-maxim-293:bqjob_r11765e0cd9ceb52b_000001427694f0e1_1: RUN_JOB
I was using service account (with private key) for authentication. I followed the following links for granting the service account access level:
https://developers.google.com/bigquery/loading-data-into-bigquery
https://developers.google.com/bigquery/access-control
The service account email was granted WRITE access with the BigQuery dataset and READ access with the GCS bucket.
Note: Adding the service account email as a writer to the project, solved the issue but this is not feasible for my case. I am not allowed to request project level write access but BigQuery and GCS (readonly).
Thanks!
In order to run the job, the service account must be given at least READ permissions on the project. This is because whoever runs a job in the project can do things that cost the project owner money (e.g. run queries).
To add the service account to the project, go to https://cloud.google.com/console, then click on "Permissions", then "Add member".
You must provide the WRITE permission on the dataset.
https://cloud.google.com/bigquery/loading-data-into-bigquery#access
This is bad, as WRITE permission imply that you have READ permission. But, for bigquery READ is paid and Load is free. For doing free task, access to paid service should be necessary.
Google must correct this.
Related
I am trying to run a Google Vertex AI pipeline to query from a BigQuery table. In the pipeline, I am using the right project and the service account(which has bigquery.jobs.create access). But I see when it runs, it is accessing another project e1cd7306fb577e88gq-uq. I am not able to figure out where from this project is coming from. I am running the pipeline from Vertex AI user managed notebook
pandas_gbq.exceptions.GenericGBQException: Reason: 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/e1cd7306fb577e88gq-uq/jobs?prettyPrint=false: Access Denied: Project e1cd7306fb577e88gq-uq: User does not have bigquery.jobs.create permission in project e1cd7306fb577e88gq-uq.
The service agent or service account running your code does have the required permission, but your code is trying to access a resource in the wrong project. Due to the way Vertex AI runs your training code, this problem can occur inadvertently if you don't explicitly specify a project ID or project number in your code.
You can explicitly select the project you want this way:
import os
from google.cloud import bigquery
project_number = os.environ["CLOUD_ML_PROJECT_ID"]
client = bigquery.Client(project=project_number)
You can read more about training code requirements here.
Say that I have two projects in GCP, Projects A and B. Project A has a BigQuery table, and project B needs to read that data using the BigQuery Storage Read API.
If I create a service account in project B, then go to project A and assign this user the role BigQuery Read Session User as well as add them to the dataset ACL, they will be able to stream the table content. Which project will receive the bill for the data extracted? Project A where the read session is created, or Project B which is the home of the acting service account?
To be clear, I would like for Project B to pay for the load they generate.
I have tried to find a way to be explicit about this, but as far as I can tell there is no way to specify billing project when creating a read session. I have also checked what happens when I try to create a read session with the "parent project" set to Project B while the table location says Project A, and this just leads to the table not being found at all.
In Storage Read API pricing, BigQuery charges for the number of bytes processed (also referred to as bytes read). In your scenario, Project A has the Bigquery table and where the Read Session happens and you just attached Project B's service account as BigQuery Read Session User in Project A, hence the billed amount will go to Project A.
As an alternative, you can check using Billing Reports the cost trends for Bigquery Storage API with the following filters:
I just started a new project on Google Cloud, set up some bigquery datasets and tables. I now want to set up some scheduled queries. I have already enabled BigQuery Data Transfer API. My query is valid (it's just SELECT * FROM table). I can't find anything about this error online.
See screenshot
UPDATE: I've experimented a bit and it seems to be an organization wide issue. All projects, new and old within my organization get this same error when trying to schedule a query. I tried for a project in a different organization and did not have the issue. What could be causing this error for ALL projects in an organization?
UPDATE 2:
By querying a table that is not empty the error change to "Error creating scheduled query: Yn" instead of "Error creating scheduled query: er" (when the scheduled query would have queried an empty table).
I faced the same issue than you, and basically I just needed to run the query first before creating the the scheduled query... And that did the trick.
from the BQ FAQs :
"Scheduled queries use features of BigQuery Data Transfer Service. Verify that you have completed all actions required in Enabling BigQuery Data Transfer Service."
basically, what this means is that you need to enable the data transfer api in your project, AND give the user who creates the scheduled query a BQ admin role in order to have the right permissions to access that transfer service.
If done right, you should get a popup when creating the scheduled query to confirm that the data transfer service has access to your uses account (if you block popups you might not see this message and get stuck)
If this error only occurs in your organisation, I believe it might be caused by a organisation policy on Google Cloud. I would encourage you to double check if there is any org policy causing this error. If that's not the case, open a support ticket with GCP.
What worked for me was signing in through Incognito Mode with just my account and attempting to save the scheduled query. I have multiple Google Accounts signed it at one time and for whatever reason, BigQuery throws this generic error after authorization is successful and BigQuery is granted the access it requested.
You need to make sure that you are creating the query under the project targeted not in any other projects because it won't appear
Also you need to enable the API as one of the above answers
This eventually worked for me when i ran this in an cognito window
I already have access to Google analytics provided by my client and the bigquery has been configured to the project. But i want to know if i can create jobs. How do i find the role assigned to my id ?
i want to know if i can create jobs
Below is simple way to get this:
Just open Web UI and try to switch to project of your interest
a. If you do have it in the list of available projects – just select it and then run (just in case) some simple query (SELECT 1)
If it is run successfully - you can create jobs in this project (because any query is in reality a job)
b. If it is not in the initial list – select “Display Project” and enter project of your interest and also check “Make this my current project” box. If result is successful – most likely you again lucky and can create jobs in this project (but still – run some simple query to be 110% sure
How do i find the role assigned to my id
This would be more involved – you will need to use respective IAM (Google Identity and Access Management) APIs
For example you can use testIamPermissions() API that allows you to test Cloud IAM permissions on a user for a resource. It takes the resource URL and a set of permissions as input parameters, and returns the set of permissions that the caller is allowed.
The permission you should look for is bigquery.jobs.create, but yo can pass to this API list of any permissions you want to check if you have
I am trying to export a table from big query to google cloud storage from console/command line. The console job runs for a few minutes and errors out without any error code and the command line job also after running for sometime gives the below error:
BigQuery error in extract operation: Error processing job 'data-flow-experiment:bqjob_r308ff0f73d1820a6_00000157f77e8ab9_1': Backend error. Job aborted.
Job id of the command line is given above.
The billing is enabled for the project and the big query service is also enabled.
Also I get the below error when I try to create a bucket in the Google Cloud Storage:
AccessDeniedException: 403 The account for the specified project is read only.
Though the IAM user I am using has owner access and I have created buckets using this account previously and have also extracted tables in the past.
Please guide.
For the bigquery issue:
Do you happen to have a timestamp column which have out-of-range values (say, far far far into the future)?
If so, you can just wait for two more days, as the fix is