How can i recreate a while function in snowflake example:
WHILE #counter <= #LastRow
BEGIN
SELECT #DateLoad = CONVERT (date, SUNDAY)
FROM [Staging].[Stg_EC_WeeksLoad]
WHERE SEQUENCE = #counter;
EXEC #return_value = [dbo].[pETLFillFact_Demographics_History] #Date = #DateLoad;
SET #counter = #counter + 1;
END
Snowflake supports expressing stored procedures in JavaScript, and JavaScript supports multiple looping constructs including while (condition).
A rough translation of the described logic as a Snowflake procedure would appear as follows:
CREATE OR REPLACE PROCEDURE looping_process()
RETURNS BOOLEAN
LANGUAGE JAVASCRIPT
AS
$$
// Get 'LastRow' dynamically, or set/pass a constant alternatively
var row_count_query = snowflake.createStatement({ sqlText: "SELECT * FROM SOURCE_TABLE" });
var _resultSet = row_count_query.execute();
var LastRow = row_count_query.getRowCount();
var counter = 0;
while (counter <= LastRow) {
// Get the dynamic date value by applying the counter
var seq_query = `SELECT SUNDAY::DATE AS "DateLoad" FROM "Staging"."Stg_EC_WeeksLoad" WHERE SEQUENCE = $counter`;
var seq_stmt = snowflake.createStatement({ sqlText: seq_query });
var seq_result = seq_stmt.execute(); seq_result.next();
var DateLoad = seq_result.getColumnValue("DateLoad");
// Construct and run the nested procedure call with value found above
var sub_call_query = `CALL "dbo"."pETLFillFact_Demographics_History"($DateLoad)`;
var sub_call_stmt = snowflake.createStatement({ sqlText: sub_call_query });
var sub_call_result = sub_call_stmt.execute(); sub_call_stmt.next();
var return_value = sub_call_result.getColumnValue(1);
// Increment for the next loop
counter += 1;
}
// (… add other omitted logic, to return/error-handle/etc. …)
$$
;
Working with Loops:
Snowflake Scripting supports the following types of loops:
FOR
WHILE
REPEAT
LOOP
Query in question could be translated as:
DECLARE
c1 CURSOR FOR select SUNDAY::date AS DateLoad
from Staging.Stg_EC_WeeksLoad order by SEQUENCE;
BEGIN
FOR record IN c1 DO
CALL pETLFillFact_Demographics_History(record.DateLoad);
END FOR;
END;
An example of the WHILE specific syntax (from the Snowflake docs) looks like this:
while (condition) {
// Do something ...
}
Related
So I want to accept postgress function which takes an argument to determine what all features should user have access to.
In Javascript/Typescript I would ideally do something like this
const a = ['featureA', 'featureB', 'featureC'];
function unlockFeatures (abc) {
let unlock = ''
if (abc.includes('featureA')) {
unlock = 'somethingA'
}
else if (abc.includes('featureB')) {
unlock = 'somethingB'
}
else if (abc.includes('featureC')) {
unlock = 'somethingC'
}
return unock
}
unlockFeatures(a)
How can I write equivalent of this in postgres/sql function where I can pass array to function?
Something like this, I guess
CREATE FUNCTION unlockFeatures(abc TEXT[])
RETURNS TEXT
LANGUAGE plpgsql AS $$
DECLARE
unlock TEXT:= '';
BEGIN
IF (SELECT 'featureA' = ANY($1)) THEN
unlock:= 'somethingA';
ELSIF (SELECT 'featureB' = ANY($1)) THEN
unlock:= 'somethingB';
ELSEIF (SELECT 'featureC' = ANY($1)) THEN
unlock:= 'somethingC';
END IF;
RETURN unlock;
END; $$;
Like the Javascript includes() method PostgreSQL has the ANY() function.
The CASE expression checks multiple condition.
At the end SQL function return last query:
CREATE OR REPLACE FUNCTION test(abc text[])
RETURNS text AS
$BODY$
SELECT case
when 'featureA' = any(abc) then 'somethingA'
when 'featureB' = any(abc) then 'somethingB'
when 'featureC' = any(abc) then 'somethingC'
end$BODY$
LANGUAGE sql VOLATILE
COST 100;
SELECT test(
ARRAY['featureA', 'featureB', 'featureC', 'featureD']
);
Say I call a stored procedure like this:
call SP_TEST('CAT','LION');
Now post successful run I get the query id using the command:
select last_query_id();
This returns the query id as 01a07606-0b02-362d-0001-1d6602361072
Say I want this query id to be written to a variable during runtime - such as :
create or replace procedure test_proc111
("SRC" VARCHAR(30), "TGT" VARCHAR(30))
returns varchar
language javascript
execute as owner
as '
{
var TABLE_VALUE = "";
var code1 = "select last_query_id();"
var code1_excecute = snowflake.execute({sqlText: code1});
while(code1_excecute.next()) {
var TABLE_VALUE = code1_excecute.getColumnValue(1);
}
return TABLE_VALUE
}
';
Now I call this stored procedure like this:
call test_proc1111('TEST','TARGET')
But I get this error:
Execution error in stored procedure TEST_PROC1111: Statement NULL not found At Snowflake. execute, line 8 position 31
How can we achieve this use case?
In the stored procedure in the question, you'll only get a query id if it's executed as caller:
create or replace procedure test_proc111
("SRC" VARCHAR(30), "TGT" VARCHAR(30))
returns varchar
language javascript
execute as caller
as '
var TABLE_VALUE;
var code1 = "select last_query_id(-1);"
var code1_excecute = snowflake.execute({sqlText: code1});
while(code1_excecute.next()){
TABLE_VALUE = code1_excecute.getColumnValue(1);
}
return TABLE_VALUE
'
;
This because if it's defined at execute as owner, it will only have a previous query id if there is another query executed within the procedure:
create or replace procedure test_proc111
("SRC" VARCHAR(30), "TGT" VARCHAR(30))
returns varchar
language javascript
execute as owner
as '
var TABLE_VALUE;
var code1 = "select last_query_id(-1);"
var code1_excecute = snowflake.execute({sqlText: code1});
var code1 = "select last_query_id(-1);"
var code1_excecute = snowflake.execute({sqlText: code1});
while(code1_excecute.next()){
TABLE_VALUE = code1_excecute.getColumnValue(1);
}
return TABLE_VALUE
'
;
I have a query that is select from an s3 bucket, but the value I need to select from changes each quarter - indicated below by {}. Is there anyway in snowflake I can write logic to be the most recent quarter
Select $1:date
from
'#lake.lake./s3key/{variable}/data.json.gzip' );
I would want to variable = 2021Q3, and then next quarter 2022Q1 ect
Is this possible? Or will I have to get python involved
I tried to use identifier for stages, but it does not seem to work.
I guess you can use Stored Procedure to achieve this.
create or replace procedure get_stage_data(quarter varchar)
returns string
language javascript
as
$$
var query = "Select $1::date from '#lake.lake./s3key/"+QUARTER+"/data.json.gzip";
var stmt = snowflake.createStatement({sqlText: query});
var res = stmt.execute();
var retVal = '';
while (res.next()) {
retVal += res.getColumnValue(1) + "\n"
}
return retVal;
$$;
Maybe write data to a table then you can analyze later.
I am having a procedure on snowflake which executing the following query:
select
array_size(split($1, ',')) as NO_OF_COL,
split($1, ',') as COLUMNS_ARRAY
from
#mystage/myfile.csv(file_format => 'ONE_COLUMN_FILE_FORMAT')
limit 1;
And the result would be like:
Why I run this query in a procedure:
CREATE OR REPLACE PROCEDURE ADD_TEMPORARY_TABLE(TEMP_TABLE_NAME STRING, FILE_FULL_PATH STRING, ONE_COLUMN_FORMAT_FILE STRING, FILE_FORMAT_NAME STRING)
RETURNS variant
LANGUAGE JAVASCRIPT
EXECUTE AS CALLER
AS
$$
try{
var final_result = [];
var nested_obj = {};
var nbr_rows = 0;
var NO_OF_COL = 0;
var COLUMNS_ARRAY = [];
var get_length_and_columns_array = "select array_size(split($1,',')) as NO_OF_COL, "+
"split($1,',') as COLUMNS_ARRAY from "+FILE_FULL_PATH+" "+
"(file_format=>"+ONE_COLUMN_FORMAT_FILE+") limit 1";
var stmt = snowflake.createStatement({sqlText: get_length_and_columns_array});
var array_result = stmt.execute();
array_result.next();
//return array_result.getColumnValue('COLUMNS_ARRAY');
NO_OF_COL = array_result.getColumnValue('NO_OF_COL');
COLUMNS_ARRAY = array_result.getColumnValue('COLUMNS_ARRAY');
return COLUMNS_ARRAY;
}
...
$$;
It will return an error as the following:
{
"code": 100183,
"message": "Given column name/index does not exist: NO_OF_COL",
"stackTraceTxt": "At ResultSet.getColumnValue, line 16 position 29",
"state": "P0000",
"toString": {}
}
The other issue is if I keep trying, it will return the desired array, but most of the times is returning this error.
The other issue is if I keep trying, it will return the desired array
If it works one time and not another time, my educated guess is that the stored procedure is called from different schemas.
Querying stage
( FILE_FORMAT => '<namespace>.<named_file_format>' )
If referencing a file format in the current namespace for your user session, you can omit the single quotes around the format identifier.
In the standalone query we can see:
select
array_size(split($1, ',')) as NO_OF_COL,
split($1, ',') as COLUMNS_ARRAY
from
#mystage/myfile.csv(file_format => 'ONE_COLUMN_FILE_FORMAT')
>-< >-<
But in the stored procedure body:
"(file_format=>"+ONE_COLUMN_FORMAT_FILE+") limit 1";
--here the text is appended, but without wrapping with ''
=>
"(file_format=>'"+ONE_COLUMN_FORMAT_FILE+"') limit 1";
Suggestion: always provide file format as as string wrapped with ', preferably prefixed with namespace '<schema_name>.<format_name>'.
Here is my stored procedure
ALTER PROCEDURE Delete
#ID nvarchar(64),
#value int = 0 output
AS
BEGIN
IF(EXISTS(SELECT * FROM A where Ap = #ID))
BEGIN
set #value = 1
END
ELSE IF(EXISTS(SELECT * FROM B where Bp = #ID))
BEGIN
set #value = 2
END
ELSE
BEGIN
select *
from Table_D
END
END
RETURN #value
Problem is that when I execute it, this does not return any value
There are multiple ways of returning status information from a stored procedure to an application. Each has its pros and cons; no single technique can definitely be said to be the right one in all circumstances. Even so, I'll start off with:
TL;DR: recommendation
Use RAISERROR if your stored procedure runs into trouble and cannot return the data it normally returns. Use OUTPUT parameters for information the client isn't free to ignore, but which isn't logically part of your result. Use the return value if you have an informational status code that the client is free to ignore. Use additional result sets only if you know what you're doing.
RAISERROR
If your stored procedure encounters an error and can't return any data, you can use RAISERROR to terminate execution and cause an exception to be raised on the client side.
CREATE PROCEDURE [Delete]
#ID nvarchar(64)
AS BEGIN
IF(EXISTS(SELECT * FROM A where Ap = #ID))
BEGIN
RAISERROR('Wrong. Try again.', 11, 1);
RETURN;
END
ELSE IF(EXISTS(SELECT * FROM B where Bp = #ID))
BEGIN
RAISERROR('Wrong in a different way. Try again.', 11, 2);
RETURN;
END
ELSE
BEGIN
select *
from Table_D
END
END
The second parameter (severity) must be set to at least 11 to make the error propagate as an exception, otherwise it's just an informational message. Those can be captured too, but that's out of the scope of this answer. The third parameter (state) can be whatever you like and could be used to pass the code of the error, if you need to localize it, for example. User-generated message always have SQL error code 50000, so that can't be used to distinguish different errors, and parsing the message is brittle.
The C# code to process the result:
try {
using (var reader = command.ExecuteReader()) {
while (reader.Read()) {
...
}
}
} catch (SqlException e) {
Console.WriteLine(
"Database error executing [Delete] (code {0}): {1}", e.State, e.Message
);
}
This is a natural fit for errors because the code to actually process the data stays what it is, and you can handle the exception at the right location (rather than propagating a status code everywhere). But this method is not appropriate if the stored procedure is expected to return a status that is informational and not an error, as you would be catching exceptions all the time even though nothing's wrong.
Output parameter
A stored procedure can set parameter values as well as receive them, by declaring them OUTPUT:
CREATE PROCEDURE [Delete]
#ID nvarchar(64),
#StatusCode INT OUTPUT
AS BEGIN
IF(EXISTS(SELECT * FROM A where Ap = #ID))
BEGIN
SET #StatusCode = 1;
END
ELSE IF(EXISTS(SELECT * FROM B where Bp = #ID))
BEGIN
SET #StatusCode = 2;
END
ELSE
BEGIN
SET #StatusCode = 0;
select *
from Table_D
END
END
From C#, this is captured in a parameter marked as an output parameter:
SqlParameter statusCodeParameter = command.Parameters.Add(
new SqlParameter {
ParameterName = "#StatusCode",
SqlDbType = SqlDbType.Int,
Direction = ParameterDirection.Output
}
);
using (var reader = command.ExecuteReader()) {
int statusCode = (int) statusCodeParameter.Value;
if (statusCode != 0) {
// show alert
return;
}
while (reader.Read()) {
...
}
}
The benefits here are that the client cannot forget to declare the parameter (it must be supplied), you're not restricted to a single INT, and you can use the value of the parameter to decide what you want to do with the resul set. Returning structured data is cumbersome this way (lots of OUTPUT parameters), but you could capture this in a single XML parameter.
Return value
Every stored procedure has a return value, which is a single INT. If you don't explicitly set it using RETURN, it stays at 0.
CREATE PROCEDURE [Delete]
#ID nvarchar(64)
AS BEGIN
IF(EXISTS(SELECT * FROM A where Ap = #ID))
BEGIN
RETURN 1
END
ELSE IF(EXISTS(SELECT * FROM B where Bp = #ID))
BEGIN
RETURN 2
END
ELSE
BEGIN
select *
from Table_D
END
END
From C#, the return value has to be captured in a single special parameter marked as the return value:
SqlParameter returnValueParameter = command.Parameters.Add(
new SqlParameter { Direction = ParameterDirection.ReturnValue }
);
using (var reader = command.ExecuteReader()) {
// this could be empty
while (reader.Read()) {
...
}
}
int returnValue = (int) returnValueParameter.Value;
It's important to note that the return value will not be available until you've processed all other result sets that the stored procedure generates (if any), so if you're using it for a status code that indicates there are no rows, you must still process the empty result set first before you have the status code. You cannot return anything other than an INT. Frameworks/OR mappers often have no support for the return value. Finally, note that the client is not required to do anything with the return value, so you have to carefully document its intended use.
Result set
The stored procedure can simply return what it wants as the result set, just like it's returning the other data. A stored procedure is allowed to return multiple result sets, so even if your status is logically separate from the other data, you can return it as a row.
CREATE PROCEDURE [Delete]
#ID nvarchar(64)
AS BEGIN
DECLARE #StatusCode INT = 0;
IF(EXISTS(SELECT * FROM A where Ap = #ID))
BEGIN
SET #StatusCode = 1;
END
ELSE IF(EXISTS(SELECT * FROM B where Bp = #ID))
BEGIN
SET #StatusCode = 2;
END
SELECT #StatusCode AS StatusCode;
IF #StatusCode = 0
BEGIN
select *
from Table_D
END
END
To process this with C#, we need SqlDataReader.NextResult:
using (var reader = command.ExecuteReader()) {
if (!reader.Read()) throw new MyException("Expected result from stored procedure.");
statusCode = reader.GetInt32(reader.GetOrdinal("StatusCode"));
if (statusCode != 0) {
// show alert
return;
}
reader.NextResult();
while (reader.Read()) {
// use the actual result set
}
}
The main drawback here is that it's not intuitive for a stored procedure to return a variable number of result sets, and very few data frameworks/OR mappers support it, so you'll nearly always end up writing manual code like this. Returning multiple result sets is not really a good fit for returning a single piece of data like a status code, but it might be an alternative to returning structured data in an XML output parameter (especially if there's lots).
The return seems to be out of scope of the procedure. Try:
ALTER PROCEDURE Delete
#ID nvarchar(64),
#value int=0 output
AS
BEGIN
IF(EXISTS(SELECT * FROM A where Ap=#ID))
BEGIN
set #value=1
END
ELSE IF(EXISTS(SELECT * FROM B where Bp=#ID))
BEGIN
set #value=2
END
ELSE
BEGIN
set #value=5
end --end if
RETURN #value
end --end procedure
This is where using tabbing properly makes the code a lot more readable, and these problems more obvious
Don't use the output parameter. Rather, use this:
ALTER PROCEDURE Delete
#ID nvarchar(64)
AS
BEGIN
DECLARE #value int
SET #value = 0
IF(EXISTS(SELECT 1 FROM A where Ap=#ID))
BEGIN
set #value=1
END
ELSE IF(EXISTS(SELECT 1 FROM B where Bp=#ID))
BEGIN
set #value=2
END
ELSE
BEGIN
set #value=5
end
Select #value as Value, * from Table_D
end
Can you try running the SP as the script below?
declare #pID as nvarchar(64)
declare #pValue as int
set #pID = 1 -- Your ID filter
exec Delete #pID, #pValue OUTPUT
select #pValue