I integrated my Firestore solution with BigQuery. With every Firestore Insert/Update/Delete operation the Data gets transferred to BigQuery. I am trying to schedule a Query that runs daily and creates a view of the Firestore data.
Everytime no matter what Query I try to schedule I get the following error : "Error creating scheduled query: dq"
The queries I try to schedule, run perfectly from the editor and I am able successfully to insert the data with an insert statement into the destination table. I am also the owner of the project so I am supposed to have all permission rights.
I appreciate your help!
Example Query Below:
"select
table1.column1,
table1.column2
from projectid.datasetID.exampletable table1"
Schedule Configuration below:
enter image description here
enter image description hereenter image description here
I had the same issue and was able to resolve by trying a different browser ( chrome) and then leaving the Data location region as default in the New schedules query menu.
Related
I need to understand under what circumstance does the protoPayload.resourceName with full table path i.e., projects/<project_id>/datasets/<dataset_id>/tables/<table_id> appear in the Log Explorer as shown in the example below.
The below entries were generated by a composer dag running a kubernetespodoperator executing some dbt commands on some models. On the basis of this, I have a sink linked to pub/sub for further processing.
As seen in the image the resourceName value is appearing as-
projects/gcp-project-name/datasets/dataset-name/tables/table-name
I have shaded the actual values of projectid, datasetid, and tablename.
I can't run the similar dag job with kuberenetesoperator on test tables owing to environment restrictions. So I tried running some update queries and insert queries using BigQuery Editor. Here is how value of protoPayload.resourceName comes as -
projects/gcp-project-name/jobs/bxuxjob_
I tried same queries using Composer DAG using BigQueryInsertJobOpertor. Here is how the value of protoPayload.resourceName comes as -
projects/gcp-project-name/jobs/airflow_<>_
Here is my question. What operation/operations in BigQuery will give me protoPayload.resourceName as the one that I am expecting i.e. -
projects/<project_id>/datasets/<dataset_id>/tables/<table_id>
I have linked my google analytics account with big query and trying to create a schedule in big query to get my desired output on regular basis, but while saving the schedule query the "Schedule query error" occurred. As this is something which I could not understand.
I want to run query multiple time so that I could get updated live report of my data.
Link shared for your reference.
click here
I've got data buckets setup in GCS and using BigQuery to run all my .csv files from that bucket to build a table. That works flawlessly. I made a simple deduplication query that when manually run, selects only distinct rows and creates a new table with "DeDupe" appended (Code below). That runs flawlessly.
CREATE OR REPLACE TABLE
`project-name-123456.dataset_2022.dataset 2022 DeDuped` AS
SELECT
DISTINCT *
FROM
`project-name-123456.dataset_2022.dataset 2022`
The issue I am having is with scheduling that query. Every time it tries to run I get the error "Error status: Not found: Dataset project-name-123456:dataset_2022 was not found in location US; JobID: project-name-123456:628d7766-0000-2d36-a82f-94eb2c0a664a"
The only thing I can figure is that I have my data location for the dataset as "us-central1" as it has a free tier. And when I go to my scheduled query, whether I select the same data location, or "Default" it always changes to "US Multiple".
Is there a way to fix this?
Or do I need to create my dataset in "US Multiple"?
Trying to cut down on costs as much as possible by keeping it in the us-central1
EDIT: Seems like I just needed to delete and recreate the scheduled query again. Chatted with Google Support and they sorted it. Sorry all!
I'm trying to create an on-demand scheduled query in the BigQuery UI, so that I can connect it to a Pub/Sub topic to run when a data import is complete each day, but I'm unable to create the scheduled query. I get the pop-up requesting permissions, I click allow and the only message I see is "Scheduled Query Error".
I double checked my permissions, I'm the project owner, and the bigquery.admin, and I have all the permissions listed. I tried using a daily schedule, tried using a table that exists, a table that doesn't exist, tried using a service account, and double checked that I have the transfers enabled, but I always receive the same error message with no details on why it isn't working.
Any ideas on what I'm doing wrong or what I might be able to do to fix it?
For some reason you need to create an empty table first with the schema and mention that as table_id. Must be a GCP bug.
I had exactly this generic error message 'Scheduled query error', and it was driving me mad. I checked all permissions and locations etc. I turns out that the 'Time Travel Window' was set to less than 48 hours for the dataset I was trying to write into. Which apparently BigQuery doesn't like.
To fix this I executed this query
ALTER SCHEMA PROJECTNAME.DATASETNAME SET OPTIONS( max_time_travel_hours = 72);
I clicked a table on bigquery dashboard, got this error:
However, I can get data when I do a select on this table. (That means the table does exist)
I already have the highest admin privilege so it shouldn't be a permission issue.
I created this table with python script, which collects data, writes into a csv file, and upload the csv file to bigquery everyday. After I created the table I once changed the schema both in the script and on the dashboard. Not sure if that's the cause, but the table loading error occurred several days after I changed the schema.
If you have Addblock extensions, this might be the root cause of this issue. Thus, try disabling it, then try running your query again.
Hope it helps.