I can successfully run Bigquery Scheduled Queries with #Run_time, #run_date parameters.
You can review Google's inadequate documentation on https://cloud.google.com/bigquery/docs/scheduling-queries
But when I try, manual run fails; "Error in starting transfer runs: Request contains an invalid argument. Dismiss" not any detail :(
Example code: (please be attentions i use #run_date)
Destination table: test_{run_time|"%Y%m%d"}
The parameter named table serves to create a different table for each day.
For example
test_20181112
test_20181113 etc.
SELECT
#run_date AS mydate,
title,
author,
text
FROM `bigquery-public-data.hacker_news.stories`
LIMIT
10
I think the problem is caused by the #run_date parameter in the query during manual operation.
My project is a little more complicated, I've added this code so everyone can try it easily.
As I mentioned above, this scheduled task works correctly in the initial setup. But when I try to run manual, it gives an error.
Can you show me the way?
Thanks for your helps.
I think that there is a bug in manunal run.
You should choose carefully the start date and the end date ( the same as a previous run ) in order to not get this error
I just wanted to add something here, since this is a "gotcha" in the BigQuery UI: using today's date as the end date in a scheduled query run will cause an issue, but setting it to one day ahead (i.e. tomorrow) should allow you to create a request with #run_date set to today.
Related
SELECT
#run_date,
Country_PG,
SUM(Weighted_Revenue) AS Weighted_Pipe
FROM
`revenue.data`
GROUP BY
1,2
When I run this query I am getting an undeclared query error for #run_date.
How do I solve this error?
For your information: this is a scheduled query.
When you are running the query manually, it is expected to fail if #run_date is not defined (by you).
When you are running it as scheduled query, based on documentation #run_date should be passed in by BigQuery, if you got error telling #run_date is not defined, create a bug to BigQuery here.
When you attempt to test a scheduled query with either the parameter #run_date or #run_time in the UI it will return the error: "Undeclared query parameters"
However, when run as part of a backfill or at the scheduled time it will work correctly.
This is a bit ugly from a UI standpoint, but in order to test the scheduled query you should use the BQ command line as stated in the documentation
At last BigQuery supports using ; in the queries, so I can write more than one query in one "block", if I seperate them with semicolon.
If I run the code manually, it works. But I cannot schedule that.
When I want to schedule, I have two choices:
(New) Web UI: I must give a destination table. If I don't do it, I could not save the scheduled query. But all my queries are updates and inserts with different "destination tables". Like these:
UPDATE project.exampledataset.a
SET date = current_date()
WHEN TRUE
;
INSERT INTO project.otherdataset.b
SELECT c,d
FROM project.otherdataset.c
So I cannot even make a scheduling in the Web UI.
Classic UI: I tried this, because the official documentary states, that I should leave the "destination table" blank, and Classic UI allows it. I can setup the scheduling, but it doesn't run, when it should. I get the error message in email "Error status: Dataset specified in the query ('') is not consistent with Destination dataset 'exampledataset'."
AIK scripting (and using semicolon) is a very new feature in BigQuery, but I hope someone can help me.
Yes, I know that I could schedule every query one by one, but I would like to resolve it with one big script.
Looks like the scheduled query was defined earlier with destination dataset defined with APPEND/TRUNCATE type transaction. While updating the same scheduled query to a DML query, GUI doesn't show the dataset field / table name to update to NULL. Hence this error is coming considering the previously set dataset and table name in the scheduled query.
Hence the fix is to delete the scheduled query and create it from scratch with DML query option. It worked for me.
Scripting is supported in scheduled query now. However, scripting query, when being scheduled, doesn't support setting a destination table for now. You still need to use DDL/DML to make change to existing table.
E.g.:
CREATE OR REPLACE TABLE destinationTable AS
SELECT *
FROM sourceTable
WHERE date >= maxDate
As of 2022, the BQ Console UI will let you create a new scheduled query without a destination dataset, but it won't let you update a prior SELECT to use DDL/DML block syntax. However, you can use the BigQuery Data Transfer API to update the destinationDatasetId field, via transferconfigs/patch. Use transferconfigs/list to get the configId for a given scheduled query.
Note that you can either use the in-browser API Explorer, if you have the appropriate credentials, or write a programmatic solution. Also seems useful for setting/updating any other fields, including renaming scheduled queries.
We got an error that {"message":"Invalid snapshot time 1472342794519, unable to read before 1472342794785","reason":"invalid"}. Other QA describes that such an error happens when table decorators' parameters are invalid, however, our query does not have table decorators.
The query uses TABLE_DATE_RANGE, but its arguments are date timestamp, so the lower digits must be 0s, not like that in the above error.
Retrying the same query succeeded.
I can provide the job ID, but because it includes internal information of our company. I apologize that I cannot directly write it here.
The tables that the TABLE_DATE_RANGE wildcard evaluates to are resolved as of the time of the start of the query. Looking at the timestamps, it looks like the table was deleted right after the job started execution. This causes the table resolution to throw that error.
Since yesterday, 1-09-2012, I can't run any queries over a table that has been created from the result of another query.
example query:
SELECT region FROM [project.table] LIMIT 1000
result:
Query Failed
Error: Field 'region' is incompatible with the table schema.
49077933619
These kinds of queries passed successfully every day, last couple of weeks. Has anybody else encountered a similar problem?
We added some additional schema checking on friday. I was unable to reproduce the problem but I'll look into your examples (I was able to find your failed job in the logs). I'm in the process of turning off the additional schema checking in the meantime. Please try again and let us know if the problem continues.
We have a bunch of T-SQL scripts dependent on today's date and when they run. If one doesn't run on the week it should, we end up temporarily setting the system time a day before, run the script, then set it back.
Is there anyway to temporarily set the system date for a script without changing the original script, like when you execute it or only for that session?
You could store the actual date in a table / temp table.
THen retrieve or update that date rather then making a call to GetDate().
I've found an answer by someone else, here I share it: "The date is tied to the OS date and time. See here: http://msdn.microsoft.com/en-us/library/ms188383.aspx".
You could refer to this other question Simulate current date on a SQL Server instance?