I am developing a scheduled query where I am using the WITH statement to join and filtrate several tables from BigQuery. To filtrate the dates, I would like to declare the following variables:
DECLARE initial, final DATE;
SET initial = DATE_TRUNC(DATE_TRUNC(CURRENT_DATE(), MONTH)+7,ISOWEEK);
SET final = LAST_DAY(DATE_TRUNC(CURRENT_DATE(), MONTH)+7, ISOWEEK);
However, when executing this query, I am getting two results; one for the variables declared (which I am not interested in having them as output), and the WITH statement that is selected at the end (which as the results that I am interested in).
The principal problem is that, whenever I try t connect this scheduled query to a table in Google Data Studio I get the following error:
Invalid value: configuration.query.destinationTable cannot be set for scripts;
How can I declare a variable without getting it as a result at the end?
Here you have a sample of the code I am trying work in:
DECLARE initial, final DATE;
SET initial = DATE_TRUNC(DATE_TRUNC(CURRENT_DATE(), MONTH)+7,ISOWEEK);
SET final = LAST_DAY(DATE_TRUNC(CURRENT_DATE(), MONTH)+7, ISOWEEK);
WITH HelloWorld AS (
SELECT shop_date, revenue
FROM fulltable
WHERE shop_date >= initial
AND shop_date <= final
)
SELECT * from HelloWorld;
with initial1 as ( select DATE_TRUNC(DATE_TRUNC(CURRENT_DATE(), MONTH)+7,ISOWEEK) as initial2),
final1 as ( select LAST_DAY(DATE_TRUNC(CURRENT_DATE(), MONTH)+7, ISOWEEK) as final2),
HelloWorld AS (
SELECT shop_date, revenue
FROM fulltable
WHERE shop_date >= (select initial2 from initial1) AND shop_date <= (select final2 from final1)
)
SELECT * from HelloWorld;
With config table having just 1 row and cross-joining it with your table, your query can be written like below.
WITH config AS (
SELECT DATE_TRUNC(DATE_TRUNC(CURRENT_DATE(), MONTH)+7,ISOWEEK) AS initial,
LAST_DAY(DATE_TRUNC(CURRENT_DATE(), MONTH)+7, ISOWEEK) AS final
),
HelloWorld AS (
SELECT * FROM UNNEST([DATE '2022-06-06']) shop_date, config
WHERE shop_date >= config.initial AND shop_date <= config.final
)
SELECT * FROM HelloWorld;
A few patterns I've used:
If you have many that have the same return type (STRING)
CREATE TEMP FUNCTION config(key STRING)
RETURNS STRING AS (
CASE key
WHEN "timezone" THEN "America/Edmonton"
WHEN "something" THEN "Value"
END
);
Then use config(key) to retrieve the value.
Or,
Create a function for each constant
CREATE TEMP FUNCTION timezone()
RETURNS STRING AS ("America/Edmonton");
Then use timezone() to get the value.
It would execute the function each time, so don't do something expensive in there (like SELECT from another table).
I have a table I have create in BigQuery.
In that table I have some placeholder values in fields.
I then thought I could simply update the placeholder values, so I wrote a small temp function
CREATE TEMP FUNCTION getBoroughFromCoords(longitude FLOAT64, latitude FLOAT64 )
RETURNS STRING
AS ((SELECT CAST(UPPER(tz_loc.borough)as STRING) FROM `bigquery-public-data.new_york_taxi_trips.taxi_zone_geom` tz_loc WHERE (ST_DWithin(tz_loc.zone_geom, ST_GeogPoint(longitude, latitude),0)) )
);
and tested it like this sql select getBoroughFromCoords(-73.95908, 40.705246) and it returned "BROOKLYN" - which was great.
But when I try to update a value (and I am specifially targetting a single row here for testing) using:
UPDATE `project-id.datasetid.collated_data`
SET NEIGHBORHOOD = (select getBoroughFromCoords(LONG, LAT))
WHERE collision_date = "2019-10-27" AND LAT = 40.705246 AND LONG = -73.95908
I get the error "LEFT OUTER JOIN cannot be used without a condition that is an equality of fields from both sides of the join."
What I don't understand is why? I mean, this function is standalone, correct? it just takes in 2 params and returns a string, so why does it suddenly want a join?
You encounter the LEFT OUTER JOIN error since you are not passing values to LONG and LAT on your subquery. To fix this you should SET values to these variables at the start of your query.
DECLARE long_var, lat_var FLOAT64;
SET long_var = -73.95908;
SET lat_var = 40.705246;
To test this out, I just created a table with dummy values just enough to satisfy your WHERE conditions.
Query used:
DECLARE long_var, lat_var FLOAT64;
SET long_var = -73.95908;
SET lat_var = 40.705246;
CREATE TEMP FUNCTION getBoroughFromCoords(longitude FLOAT64, latitude FLOAT64 )
RETURNS STRING
AS ((SELECT CAST(UPPER(tz_loc.borough)as STRING) FROM `bigquery-public-data.new_york_taxi_trips.taxi_zone_geom` tz_loc
WHERE (ST_DWithin(tz_loc.zone_geom, ST_GeogPoint(longitude, latitude),0)))
);
UPDATE `project-id.dataset_id.collated_data`
SET NEIGHBORHOOD = (select getBoroughFromCoords(long_var, lat_var))
WHERE collision_date = "2019-10-27" AND LAT = lat_var AND LONG = long_var
Output:
Query project-id.dataset_id.collated_data table:
I 'm not able to use both Declare statements in my select query when I do the insert into.
How do you use a column itemId to be set in a another column called id?
How do you write a same string to a specific column?
Why can't I call my scalar variable ?
DECLARE #Solutions_id as INT
DECLARE #Solutions_name as nvarchar(50)
INSERT INTO ticket_historical_actions
(
[id]
,[type]
[Solutions _rowId]
,[Solutions_dw_dateCreate]
,[Solutions_dw_dateMod]
,[Solutions_dw_dateDelete]
,[Solutions_sourceId]
,[Solutions_date_create]
,[Solutions_date_mod]
,[Solutions_date_approval]
,[Solutions_itemId]
,[Solutions_solutionTypeName]
,[Solutions_content_plainText]
,[Solutions_userId]
,[Solutions_userId_editor]
,[Solutions_userId_approval]
,[Solutions_userName]
,[Solutions_userName_approval]
,[Solutions_status]
,[Ticket_rowId]
,[Ticket_dw_dateCreate]
,[Ticket_dw_dateMod]
,[Ticket_dw_dateDelete]
,[Ticket_sourceId]
,[Ticket_date_create]
,[Ticket_date_mod]
,[Ticket_date_close]
,[Ticket_date_solve]
,[Ticket_entityId]
,[Ticket_name]
,[Ticket_date]
,[Ticket_status]
,[Ticket_is_deleted]
,[Ticket_content_PlainText]
,[Ticket_type]
,[Ticket_urgency]
,[Ticket_impact]
,[Ticket_priority]
,[Ticket_requestTypeId]
,[Ticket_userId_lastUpdater]
,[Ticket_userId_recipient]
,[Ticket_time_to_resolve]
,[Ticket_time_to_own]
,[Users_rowId]
,[Users_dw_dateCreate]
,[Users_dw_dateMod]
,[Users_dw_dateDelete]
,[Users_sourceId]
,[Users_date_create]
,[Users_date_mod]
,[Users_name]
,[Users_LastName]
,[Users_firstName]
,[Users_phone]
,[Users_mobile]
,[Users_language]
,[Users_profileId]
,[Users_entitieId]
,[Users_titleId]
,[Users_categoryId]
,[Users_managerId]
,[Company_rowId]
, [Company_dw_dateCreate]
,[Company_dw_dateMod]
,[Company_dw_dateDelete]
,[Company_sourceId]
,[Company_date_mod]
,[Company_date_create]
,[Company_completename]
,[Company_name]
,[Company_address]
,[Company_postcode]
,[Company_town]
,[Company_state]
,[Company_country]
,[Company_phonenumber]
,[Company_email]
,[Company_admin_email]
,[Company_admin_name]
)
SELECT
#Solutions_id =[itemId], #Solutions_name = 'Solutions',
ts.[rowId]
,ts.[dw_dateCreate]
,ts.[dw_dateMod]
,ts.[dw_dateDelete]
,ts.[sourceId]
,ts.[date_create]
,ts.[date_mod]
,ts.[date_approval]
,ts.[itemId]
,ts.[solutionTypeName]
,ts.[content_plainText]
,ts.[userId]
,ts.[userId_editor]
,ts.[userId_approval]
,ts.[userName]
,ts.[userName_approval]
,ts.[status]
, tt. [rowId]
,tt.[dw_dateCreate]
,tt.[dw_dateMod]
,tt.[dw_dateDelete]
,tt.[sourceId]
,tt.[date_create]
,tt.[date_mod]
,tt.[date_close]
,tt.[date_solve]
,tt.[entityId]
,tt.[name]
,tt.[date]
,tt.[status]
,tt.[is_deleted]
,tt.[content_PlainText]
,tt.[type]
,tt.[urgency]
,tt.[impact]
,tt.[priority]
,tt.[requestTypeId]
,tt.[userId_lastUpdater]
,tt.[userId_recipient]
,tt.[time_to_resolve]
,tt.[time_to_own]
, tu.[rowId]
,tu.[dw_dateCreate]
,tu.[dw_dateMod]
,tu.[dw_dateDelete]
,tu.[sourceId]
,tu.[date_create]
,tu.[date_mod]
,tu.[name]
,tu.[LastName]
,tu.[firstName]
,tu.[phone]
,tu.[mobile]
,tu.[language]
,tu.[profileId]
,tu.[entitieId]
,tu.[titleId]
,tu.[categoryId]
,tu.[managerId]
, tc. [rowId]
,tc.[dw_dateCreate]
,tc.[dw_dateMod]
,tc.[dw_dateDelete]
,tc.[sourceId]
,tc.[date_mod]
,tc.[date_create]
,tc.[completename]
,tc.[name]
,tc.[address]
,tc.[postcode]
,tc.[town]
,tc.[state]
,tc.[country]
,tc.[phonenumber]
,tc.[email]
,tc.[admin_email]
,tc.[admin_name]
FROM
ticket_ticketSolutions ts
left join ticket_tickets tt
on ts.itemId =tt.sourceId
left join ticket_users tu
on ts.userId = tu.sourceId
left join ticket_company tc
on tu.entitieId = tc.sourceId
Would tell me if possible how to do it?
Thank you very much.
I'm just going to answer, even though I have stated this in the comments of both your questions.
The problem here is you are trying to assign a value to your variables in the same statement you are trying to return a dataset; in T-SQL that in not allowed.
In short, you have something like this:
SELECT #MyVariable = SomeColumn,
AnotherColumn
FROM dbo.YourTable
WHERE ID = 'SomeID';
So you want to return the value of AnotherColumn to the presentation layer, but assign the value of SomeColumn to the variable #MyVariable; you can't do this.
Instead, you have to use 2 statements:
SELECT #MyVariable = SomeColumn --Assigns the value to #MyVariable
FROM dbo.YourTable
WHERE ID = 'SomeID';
SELECT AnotherColumn --Returns the dataset to the presentation layer
FROM dbo.YourTable
WHERE ID = 'SomeID';
I have been pondering about what and how much I can actually put into the substitution variable. can we only have it to replace a single 'entity' but not a string of command? I found if I want to substitute a block of actual SQL, it did not work (in SqlDeveloper).
I wonder how I can get around this...
/*------test--------------*/
CREATE TABLE Test_Persons (
PersonID int,
LastName varchar(255),
FirstName varchar(255)
);
INSERT INTO Test_Persons
(PersonID,LastName,FirstName)
values(1,'LN_1','FN_1');
INSERT INTO Test_Persons
(PersonID,LastName,FirstName)
values(2,'LN_2','FN_2');
INSERT INTO Test_Persons
(PersonID,LastName,FirstName)
values(3,'LN_21','FN_2');
commit;
--------------sub-table, worked!---
set define #;
define first_name_input = 'FN_2';
define last_name_input = 'LN_2';
define tbl_Test_Persons = 'Test_Persons';
define temp_tbl_Test_Persons_FN = 'Test_Persons_FN';
with
#temp_tbl_Test_Persons_FN as
(
select * from #tbl_Test_Persons tp
where tp.FIRSTNAME = '#first_name_input'
)
select * from #temp_tbl_Test_Persons_FN tp
where tp.LASTNAME = '#last_name_input';
set define off;
--------------sub-table, replace the entire with clause, does NOT work - 'Invalid value for DEFINE command.'!---
set define #;
define first_name_input = 'FN_2';
define last_name_input = 'LN_2';
define tbl_Test_Persons = 'Test_Persons';
define temp_tbl_Test_Persons_FN = 'Test_Persons_FN2';
define with_clause = '
with
#temp_tbl_Test_Persons_FN as
(
select * from #tbl_Test_Persons tp
where tp.FIRSTNAME = ''#first_name_input''
)
';
#with_clause
select * from #temp_tbl_Test_Persons_FN tp
where tp.LASTNAME = '#last_name_input';
set define off;
From the documentation (for SQL*Plus but applies to SQL Developer too):
If the value of a defined variable extends over multiple lines (using the SQL*Plus command continuation character), SQL*Plus replaces each continuation character and carriage return with a space.
SQL Developer actually behaves slightly differently, and leaves the continuation character inside the string, which isn't helpful. But is also leaves the line breaks, so you can double-up the continuation character from - to --to stop it causing a problem, as that is then treated a comment; so you can do:
define with_clause = '--
with--
#temp_tbl_Test_Persons_FN as--
(--
select * from #tbl_Test_Persons tp--
where tp.FIRSTNAME = ''#first_name_input''--
)--
';
But there is a further complication, because #with_clause isn't recognised as a command - as you aren't yet in a command substitution doesn't happen. You can get around that, messily, with a subquery:
select * from (
#with_clause
select * from #temp_tbl_Test_Persons_FN tp
where tp.LASTNAME = '#last_name_input'
);
which isn't ideal. But if you do that, the script output window shows the substitution happening and you get a row back:
old:select * from (
#with_clause
select * from #temp_tbl_Test_Persons_FN tp
where tp.LASTNAME = '#last_name_input'
)
new:select * from (
--
with--
Test_Persons_FN2 as--
(--
select * from Test_Persons tp--
where tp.FIRSTNAME = 'FN_2'--
)--
select * from Test_Persons_FN2 tp
where tp.LASTNAME = 'LN_2'
)
PERSONID LASTNAME FIRSTNAME
---------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2 LN_2 FN_2
(Also, interestingly, this still doesn't work in SQL*Plus if you use # as the substitution character - you get "ORA-24333: zero iteration count". It doesn't get that error with & though; I haven't tried any others. And you have to go back to a single concatenation character. But even with those changes it still doesn't substitute the with clause properly. Not sure why yet.)
I have a problem with a function in db2
The function finds a record, and returns a number according to whether the first and second recorded by a user
The query within the function is this
SELECT
CASE
WHEN NUM IN (1,2) THEN 5
ELSE 2.58
END AS VAL
FROM (
select ROW_NUMBER() OVER() AS NUM ,s.POLLIFE
from LQD943DTA.CAQRTRML8 c
INNER JOIN LSMODXTA.SCSRET s ON c.MCCNTR = s.POLLIFE
WHERE s.NOEMP = ( SELECT NOEMP FROM LSMODDTA.LOLLM04 WHERE POLLIFE = '0010111003')
) AS T WHERE POLLIFE = '0010111003'
And works perfect
I create the function with this code
CREATE FUNCTION LIBWEB.BNOWPAPOL(POL CHAR)
RETURNS DECIMAL(7,2)
LANGUAGE SQL
NOT DETERMINISTIC
READS SQL DATA
RETURN (
SELECT
CASE
WHEN NUM IN (1,2) THEN 5
ELSE 2.58
END AS VAL
FROM (
select ROW_NUMBER() OVER() AS NUM ,s.POLLIFE
from LQD943DTA.CAQRTRML8 c
INNER JOIN LSMODXTA.SCSRET s ON c.MCCNTR = s.POLLIFE
WHERE s.NOEMP = ( SELECT NOEMP FROM LSMODDTA.LOLLM04 WHERE POLLIFE = POL)
) AS T WHERE POLLIFE = POL
)
The command runs executed properly
WARNING: 17:55:40 [CREATE - 0 row(s), 0.439 secs] Command processed.
No rows were affected
When I want execute the query a get a error
SELECT LIBWEB.BNOWPAPOL('0010111003') FROM DATAS.DUMMY -- dummy has only one row
I get
[Error Code: -204, SQL State: 42704] [SQL0204] BNOWPAPOL in LIBWEB
type *N not found.
I detect, when I remove the parameter the function works fine!
With this code
CREATE FUNCTION LIBWEB.BNOWPAPOL()
RETURNS DECIMAL(7,2)
LANGUAGE SQL
NOT DETERMINISTIC
READS SQL DATA
RETURN (
SELECT
CASE
WHEN NUM IN (1,2) THEN 5
ELSE 2.58
END AS VAL
FROM (
select ROW_NUMBER() OVER() AS NUM ,s.POLLIFE
from LQD943DTA.CAQRTRML8 c
INNER JOIN LSMODXTA.SCSRET s ON c.MCCNTR = s.POLLIFE
WHERE s.NOEMP = ( SELECT NOEMP FROM LSMODDTA.LOLLM04 WHERE POLLIFE = '0010111003')
) AS T WHERE POLLIFE = '0010111003'
)
Why??
This statement:
SELECT LIBWEB.BNOWPAPOL('0010111003') FROM DATAS.DUMMY
causes this error:
[Error Code: -204, SQL State: 42704] [SQL0204] BNOWPAPOL in LIBWEB
type *N not found.
The parm value passed into the BNOWPAPOL() function is supplied as a quoted string with no definition (no CAST). The SELECT statement assumes that it's a VARCHAR value since different length strings might be given at any time and passes it to the server as a VARCHAR.
The original function definition says:
CREATE FUNCTION LIBWEB.BNOWPAPOL(POL CHAR)
The function signature is generated for a single-byte CHAR. (Function definitions can be overloaded to handle different inputs, and signatures are used to differentiate between function versions.)
Since a VARCHAR was passed from the client and only a CHAR function version was found by the server, the returned error fits. Changing the function definition or CASTing to a matching type can solve this kind of problem. (Note that a CHAR(1) parm could only correctly handle a single-character input if a value is CAST.)