I just started a new project on Google Cloud, set up some bigquery datasets and tables. I now want to set up some scheduled queries. I have already enabled BigQuery Data Transfer API. My query is valid (it's just SELECT * FROM table). I can't find anything about this error online.
See screenshot
UPDATE: I've experimented a bit and it seems to be an organization wide issue. All projects, new and old within my organization get this same error when trying to schedule a query. I tried for a project in a different organization and did not have the issue. What could be causing this error for ALL projects in an organization?
UPDATE 2:
By querying a table that is not empty the error change to "Error creating scheduled query: Yn" instead of "Error creating scheduled query: er" (when the scheduled query would have queried an empty table).
I faced the same issue than you, and basically I just needed to run the query first before creating the the scheduled query... And that did the trick.
from the BQ FAQs :
"Scheduled queries use features of BigQuery Data Transfer Service. Verify that you have completed all actions required in Enabling BigQuery Data Transfer Service."
basically, what this means is that you need to enable the data transfer api in your project, AND give the user who creates the scheduled query a BQ admin role in order to have the right permissions to access that transfer service.
If done right, you should get a popup when creating the scheduled query to confirm that the data transfer service has access to your uses account (if you block popups you might not see this message and get stuck)
If this error only occurs in your organisation, I believe it might be caused by a organisation policy on Google Cloud. I would encourage you to double check if there is any org policy causing this error. If that's not the case, open a support ticket with GCP.
What worked for me was signing in through Incognito Mode with just my account and attempting to save the scheduled query. I have multiple Google Accounts signed it at one time and for whatever reason, BigQuery throws this generic error after authorization is successful and BigQuery is granted the access it requested.
You need to make sure that you are creating the query under the project targeted not in any other projects because it won't appear
Also you need to enable the API as one of the above answers
This eventually worked for me when i ran this in an cognito window
Related
I'm trying to create a scheduled job based out of saved query in big query. I'm following steps as per documentation here
After entering all the filelds, I'm getting response "Error creating scheduled query: Aq" with no additional details.
In on BigQuery documentation also, I didn't find anything.
I could run the query which I'm trying to create a schedule for, is running fine and giving appropriate results.
It turned out that, when schedule query is created for first time, the console is taking permission by take user to new tab in browser. Browser settings was suppressing it.
After acknowledging the required permissions, I could create scheduled query successfully.
I am having a problem with a data transfer from Google Ads. When I schedule the backfill I get the following error for some dates:
Invalid value: Load configuration must specify at least one source URI
When I check the log inside of the details of execution I get the following message:
Failed to start job for table p_ClickStats_5419416216$20201117 with error INVALID_ARGUMENT: Invalid value: Load configuration must specify at least one source URI
The weird part is that this happens for random dates which I had transfered before in a previous transfer. Did anyone have a problem similar to that?
I had the very same problem with the same error. I was using a free trial account. I shared my project under another account that was an upgraded account with billing set-up (but still with the promotional credits). So far, I have not had the same issue. Try to upgrade your account to one where you set up billing and try the data transfer again. Don't use the purely free trial account. You can share the project to the other account and set it up there, the data transfer. Backfilling for me has also worked and looks like no more duplicate runs either. Maybe free account trial is a limited version.
I am trying to export a table from big query to google cloud storage from console/command line. The console job runs for a few minutes and errors out without any error code and the command line job also after running for sometime gives the below error:
BigQuery error in extract operation: Error processing job 'data-flow-experiment:bqjob_r308ff0f73d1820a6_00000157f77e8ab9_1': Backend error. Job aborted.
Job id of the command line is given above.
The billing is enabled for the project and the big query service is also enabled.
Also I get the below error when I try to create a bucket in the Google Cloud Storage:
AccessDeniedException: 403 The account for the specified project is read only.
Though the IAM user I am using has owner access and I have created buckets using this account previously and have also extracted tables in the past.
Please guide.
For the bigquery issue:
Do you happen to have a timestamp column which have out-of-range values (say, far far far into the future)?
If so, you can just wait for two more days, as the fix is
I am trying to use the bq command line tool to load data into BigQuery from GCS bucket and I receive the following error message:
BigQuery error in load operation: Access Denied: Job mythical-maxim-293:bqjob_r11765e0cd9ceb52b_000001427694f0e1_1: RUN_JOB
I was using service account (with private key) for authentication. I followed the following links for granting the service account access level:
https://developers.google.com/bigquery/loading-data-into-bigquery
https://developers.google.com/bigquery/access-control
The service account email was granted WRITE access with the BigQuery dataset and READ access with the GCS bucket.
Note: Adding the service account email as a writer to the project, solved the issue but this is not feasible for my case. I am not allowed to request project level write access but BigQuery and GCS (readonly).
Thanks!
In order to run the job, the service account must be given at least READ permissions on the project. This is because whoever runs a job in the project can do things that cost the project owner money (e.g. run queries).
To add the service account to the project, go to https://cloud.google.com/console, then click on "Permissions", then "Add member".
You must provide the WRITE permission on the dataset.
https://cloud.google.com/bigquery/loading-data-into-bigquery#access
This is bad, as WRITE permission imply that you have READ permission. But, for bigquery READ is paid and Load is free. For doing free task, access to paid service should be necessary.
Google must correct this.
So, we have this one project which uses Cloud Storage and BigQuery as services. All has been well.
Then, I wanted to add Cloud SQL to this project to try it out. It asked for a unique Project ID so I gave it one. (The Project ID is different than the Project Number.)
Ever since then, I've been having a difficult time accessing my BigQuery tables. When I go to the BigQuery web interface, the URL contains the Project ID instead of the original Project Number. It shows the list of datasets, but now shows the Project Number before each dataset name and the datasets are greyed out and inaccessible. If I manually change the URL to contain the Project Number instead of the Project ID, it appears to work although it shows the list of datasets in the left nav twice, one set greyed out and inaccessible and the other set seemingly accessible.
At the same time, some code that I've been successfully using in Apps Script that accesses BigQuery is now regularly failing with a generic "We're sorry, a server error occurred. Please wait a bit and try again." I'm not sure if this is related to the Project ID/Project Number confusion, or if it's just a Red Herring.
Since we actively use the Cloud Storage service of this project, I am trying to be cautious with further experimentation with this project. I'm not sure if I should delete the Cloud SQL service in this project to get it back to the way it was, or if this is a known issue with some back-end solution. Please advise.
After setting the project id, there can be a delay where BigQuery picks up the change. It should happen within 15 minutes or so, but sometimes it takes longer.
If you send the project ID I can make sure it has been updated.