I am trying to update a deployment I have made using the GCP deployment manager, however, I get an error saying the datasets in the deployment already exist. Is there a way I can tell my deployment to create the dataset when it doesn't exist and do nothing if it does. I thought that was the point of the update command. Below is the error I am getting:
code: u'RESOURCE_ERROR'
location: u'dep23/dataset'
message: u'Unexpected response from resource of type bigquery.v2.dataset: 409 {"code":409,"errors":[{"domain":"global","message":"Already Exists: Dataset my-project:dataset","reason":"duplicate"}],"message":"Already Exists: Dataset my=project:dataset","statusMessage":"Conflict","requestPath":"https://www.googleapis.com/bigquery/v2/projects/my-project/datasets"}'>
It seems like the resource has to be created by the deployment manager or else you are going to have various issues. I had to delete my dataset and re-create it using the deployment manager and then it started working.
Related
In my current dbt project, I run everything on the same Google cloud project (let's say project : dataA). Since datasets becomes a lot, I decide to split the project into 2: The current project for import of raw data and a new project (for example : dataB) for production environment where I stock all data marts.
I use a service account to manage the lecture or editing data sources for both two projects. And I am sure that there is no issues on rights. The profile setting is quite similar to my current settings which work fine.
But I am experiencing some Database Error issues from dbt say that I don't have Insufficient Permission.
Does anyone have an idea about the reason of the issue? And how to fix it?
Many thanks!
I am trying to run a Google Vertex AI pipeline to query from a BigQuery table. In the pipeline, I am using the right project and the service account(which has bigquery.jobs.create access). But I see when it runs, it is accessing another project e1cd7306fb577e88gq-uq. I am not able to figure out where from this project is coming from. I am running the pipeline from Vertex AI user managed notebook
pandas_gbq.exceptions.GenericGBQException: Reason: 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/e1cd7306fb577e88gq-uq/jobs?prettyPrint=false: Access Denied: Project e1cd7306fb577e88gq-uq: User does not have bigquery.jobs.create permission in project e1cd7306fb577e88gq-uq.
The service agent or service account running your code does have the required permission, but your code is trying to access a resource in the wrong project. Due to the way Vertex AI runs your training code, this problem can occur inadvertently if you don't explicitly specify a project ID or project number in your code.
You can explicitly select the project you want this way:
import os
from google.cloud import bigquery
project_number = os.environ["CLOUD_ML_PROJECT_ID"]
client = bigquery.Client(project=project_number)
You can read more about training code requirements here.
I was following this step by step to connect data from BQ to AWS Glue and store in S3 , everything works ok until i tried to run the job, where the job keeps failing with:
An error occurred while calling o76.getSource. The type of table {datasetName}.{table_name} is currently not supported: EXTERNAL
I can't seem to find any similar error online, also can't find further helpful info from the log, it seems that it's stuck at the issue with the BQ table, I was following exactly as what the author did here in the blog with the key-value pair to indicate project ID and dataset/table (image refers to blog's author table name).
Does anybody know what's causing this?
I just started a new project on Google Cloud, set up some bigquery datasets and tables. I now want to set up some scheduled queries. I have already enabled BigQuery Data Transfer API. My query is valid (it's just SELECT * FROM table). I can't find anything about this error online.
See screenshot
UPDATE: I've experimented a bit and it seems to be an organization wide issue. All projects, new and old within my organization get this same error when trying to schedule a query. I tried for a project in a different organization and did not have the issue. What could be causing this error for ALL projects in an organization?
UPDATE 2:
By querying a table that is not empty the error change to "Error creating scheduled query: Yn" instead of "Error creating scheduled query: er" (when the scheduled query would have queried an empty table).
I faced the same issue than you, and basically I just needed to run the query first before creating the the scheduled query... And that did the trick.
from the BQ FAQs :
"Scheduled queries use features of BigQuery Data Transfer Service. Verify that you have completed all actions required in Enabling BigQuery Data Transfer Service."
basically, what this means is that you need to enable the data transfer api in your project, AND give the user who creates the scheduled query a BQ admin role in order to have the right permissions to access that transfer service.
If done right, you should get a popup when creating the scheduled query to confirm that the data transfer service has access to your uses account (if you block popups you might not see this message and get stuck)
If this error only occurs in your organisation, I believe it might be caused by a organisation policy on Google Cloud. I would encourage you to double check if there is any org policy causing this error. If that's not the case, open a support ticket with GCP.
What worked for me was signing in through Incognito Mode with just my account and attempting to save the scheduled query. I have multiple Google Accounts signed it at one time and for whatever reason, BigQuery throws this generic error after authorization is successful and BigQuery is granted the access it requested.
You need to make sure that you are creating the query under the project targeted not in any other projects because it won't appear
Also you need to enable the API as one of the above answers
This eventually worked for me when i ran this in an cognito window
I am trying to export a table from big query to google cloud storage from console/command line. The console job runs for a few minutes and errors out without any error code and the command line job also after running for sometime gives the below error:
BigQuery error in extract operation: Error processing job 'data-flow-experiment:bqjob_r308ff0f73d1820a6_00000157f77e8ab9_1': Backend error. Job aborted.
Job id of the command line is given above.
The billing is enabled for the project and the big query service is also enabled.
Also I get the below error when I try to create a bucket in the Google Cloud Storage:
AccessDeniedException: 403 The account for the specified project is read only.
Though the IAM user I am using has owner access and I have created buckets using this account previously and have also extracted tables in the past.
Please guide.
For the bigquery issue:
Do you happen to have a timestamp column which have out-of-range values (say, far far far into the future)?
If so, you can just wait for two more days, as the fix is