Snowflake use variable as a filename while unloading data to S3 - sql
I believe the problem is not specific to unloading data to S3, but how can we use variables in the names.
I open an S3 stage as following
CREATE OR REPLACE STAGE s3_stage
URL = 's3://some-foo-bar-url'
CREDENTIALS = (aws_role='arn:aws:iam::<some-number>:role/<some-foo-bar-role-with-relevant-permissions>')
;
Then I want to use it to unload my table data as
SET DATE_TODAY = TO_DATE(CURRENT_TIMESTAMP(2));
COPY INTO #crm.s3_stage/toProcess/{$TODAY_DATE}/restaurants
FROM (my_restaurants_table)
file_format = (FIELD_OPTIONALLY_ENCLOSED_BY='"')
include_query_id = TRUE
header = TRUE
max_file_size = 2000000
;
I want to $TODAY_DATE in S3 bucket's directory path
What should I change to make it work?
I believe you need to use a Stored Procedure / Snowflake Scripting, the below works
execute immediate
$$
declare
date_today varchar default TO_DATE(CURRENT_TIMESTAMP(2));
copy_into varchar default 'COPY INTO #<stage>/' || date_today || '/filename FROM (select * from <table> limit 10) overwrite = true';
begin
execute immediate :copy_into;
return date_today;
end;
$$;
Related
Is it possible to invoke BigQuery procedures in python client?
Scripting/procedures for BigQuery just came out in beta - is it possible to invoke procedures using the BigQuery python client? I tried: query = """CALL `myproject.dataset.procedure`()....""" job = client.query(query, location="US",) print(job.results()) print(job.ddl_operation_performed) print(job._properties) but that didn't give me the result set from the procedure. Is it possible to get the results? Thank you! Edited - stored procedure I am calling CREATE OR REPLACE PROCEDURE `Project.Dataset.Table`(IN country STRING, IN accessDate DATE, IN accessId, OUT saleExists INT64) BEGIN IF EXISTS (SELECT 1 FROM dataset.table where purchaseCountry = country and purchaseDate=accessDate and customerId = accessId) THEN SET saleExists = (SELECT 1); ELSE INSERT Dataset.MissingSalesTable (purchaseCountry, purchaseDate, customerId) VALUES (country, accessDate, accessId); SET saleExists = (SELECT 0); END IF; END;
If you follow the CALL command with a SELECT statement, you can get the return value of the function as a result set. For example, I created the following stored procedure:
BEGIN
-- Build an array of the top 100 names from the year 2017.
DECLARE
top_names ARRAY<STRING>;
SET
top_names = (
SELECT
ARRAY_AGG(name
ORDER BY
number DESC
LIMIT
100)
FROM
`bigquery-public-data.usa_names.usa_1910_current`
WHERE
year = 2017 );
-- Which names appear as words in Shakespeare's plays?
SET
top_shakespeare_names = (
SELECT
ARRAY_AGG(name)
FROM
UNNEST(top_names) AS name
WHERE
name IN (
SELECT
word
FROM
`bigquery-public-data.samples.shakespeare` ));
END
Running the following query will return the procedure's return as the top-level results set.
DECLARE top_shakespeare_names ARRAY<STRING> DEFAULT NULL;
CALL `my-project.test_dataset.top_names`(top_shakespeare_names);
SELECT top_shakespeare_names;
In Python:
from google.cloud import bigquery
client = bigquery.Client()
query_string = """
DECLARE top_shakespeare_names ARRAY<STRING> DEFAULT NULL;
CALL `swast-scratch.test_dataset.top_names`(top_shakespeare_names);
SELECT top_shakespeare_names;
"""
query_job = client.query(query_string)
rows = list(query_job.result())
print(rows)
Related: If you have SELECT statements within a stored procedure, you can walk the job to fetch the results, even if the SELECT statement isn't the last statement in the procedure.
# TODO(developer): Import the client library.
# from google.cloud import bigquery
# TODO(developer): Construct a BigQuery client object.
# client = bigquery.Client()
# Run a SQL script.
sql_script = """
-- Declare a variable to hold names as an array.
DECLARE top_names ARRAY<STRING>;
-- Build an array of the top 100 names from the year 2017.
SET top_names = (
SELECT ARRAY_AGG(name ORDER BY number DESC LIMIT 100)
FROM `bigquery-public-data.usa_names.usa_1910_2013`
WHERE year = 2000
);
-- Which names appear as words in Shakespeare's plays?
SELECT
name AS shakespeare_name
FROM UNNEST(top_names) AS name
WHERE name IN (
SELECT word
FROM `bigquery-public-data.samples.shakespeare`
);
"""
parent_job = client.query(sql_script)
# Wait for the whole script to finish.
rows_iterable = parent_job.result()
print("Script created {} child jobs.".format(parent_job.num_child_jobs))
# Fetch result rows for the final sub-job in the script.
rows = list(rows_iterable)
print("{} of the top 100 names from year 2000 also appear in Shakespeare's works.".format(len(rows)))
# Fetch jobs created by the SQL script.
child_jobs_iterable = client.list_jobs(parent_job=parent_job)
for child_job in child_jobs_iterable:
child_rows = list(child_job.result())
print("Child job with ID {} produced {} rows.".format(child_job.job_id, len(child_rows)))
It works if you have SELECT inside your procedure, given the procedure being:
create or replace procedure dataset.proc_output() BEGIN
SELECT t FROM UNNEST(['1','2','3']) t;
END;
Code:
from google.cloud import bigquery
client = bigquery.Client()
query = """CALL dataset.proc_output()"""
job = client.query(query, location="US")
for result in job.result():
print result
will output:
Row((u'1',), {u't': 0})
Row((u'2',), {u't': 0})
Row((u'3',), {u't': 0})
However, if there are multiple SELECT inside a procedure, only the last result set can be fetched this way.
Update
See below example:
CREATE OR REPLACE PROCEDURE zyun.exists(IN country STRING, IN accessDate DATE, OUT saleExists INT64)
BEGIN
SET saleExists = (WITH data AS (SELECT "US" purchaseCountry, DATE "2019-1-1" purchaseDate)
SELECT Count(*) FROM data where purchaseCountry = country and purchaseDate=accessDate);
IF saleExists = 0 THEN
INSERT Dataset.MissingSalesTable (purchaseCountry, purchaseDate, customerId) VALUES (country, accessDate, accessId);
END IF;
END;
BEGIN
DECLARE saleExists INT64;
CALL zyun.exists("US", DATE "2019-2-1", saleExists);
SELECT saleExists;
END
BTW, your example is much better served with a single MERGE statement instead of a script.
Running update from SQL Developer with different results as from package
I tried to run an update within an cursor ('cur_palettenkosten') operation in an plsql procedure.
I narrowed down, that the enclosing cursor has data and that the update does not affect any rows (output sql%rowcount)
PROCEDURE p_ref_lschein_rueckstellungen AS
for cur_palettenkosten in (
select land, spediteur_nr, plz_von, plz_bis, preis, gueltig_von, geultig_bis, gzp.behaelter_nr
from spediteur_fahrtkosten sp,gutschrift_zuord_pal gzp
where sp.behaelter_nr = 1
)
LOOP
UPDATE lschein_rueckstellungen
SET preis = cur_palettenkosten.preis
WHERE to_number(sped_nr) = to_number(cur_palettenkosten.spediteur_nr)
AND to_number(lhm_typ) = to_number(cur_palettenkosten.behaelter_nr)
AND to_char(kst) = to_char(cur_palettenkosten.land);
dbms_output.put_line (cur_palettenkosten.spediteur_nr || ' '||cur_palettenkosten.behaelter_nr|| ' '|| cur_palettenkosten.land || sql%rowcount);
END LOOP;
COMMIT;
END p_ref_lschein_rueckstellungen;
Running the script from the editor this way:
BEGIN
p_ref_lschein_rueckstellungen;
END
does have any effect on the table 'lschein_rueckstellungen' which I wanted to update.
Running it from the same editor window like this:
BEGIN
<procedure content copied here>
END
updates the data as desired.
Are the any ideas, what I did wrong?
Could it be that you open a new session per tab? Then of course you have to commit the update!
import array type into hana?
I am importing data into SAP HANA using the CSV files.
When I try to import a column which has an array type then it results in the following error
ARRAY type is not compatible with PARAMETER TYPE
For example
CREATE COLUMN TABLE "SCHEMA"."TABLE"
( 'ID' INT,
'SUBJECTS' INT ARRAY)
The above query creates the table and when I run
INSERT INTO "SCHEMA"."TABLE" VALUES (1,ARRAY(1,2,3))
It inserts successfully into the HANA database.
But when I try
INSERT INTO "SCHEMA"."TABLE" VALUES (1,"{1,2,3}")
It does not work.So how can I import the array values in the CSV file to the column in HANA database.
Array storage types can currently only be created by using the ARRAY() function.
You could construct the INSERT statement in a loop but you still need to construct the ARRAY() call for every record.
Ok, here's the example you asked for.
By now you should understand that there is no simple IMPORT command that would automatically insert arrays into a HANA table.
That leaves you with two options as I see it:
you write a loader program that reads your CSV file and parses
the array data {..., ... , ...} and makes INSERT statements with
ARRAY functions out of it.
or
You load the data in two steps:
2.1 Load the data from the CSV as-is and put the array data into a CLOB column.
2.2. Add the array columns to the table and run a loop that takes the CLOB data, replaces the curly brackets with normal brackets and creates a dynamic SQL statement.
Like so:
create column table array_imp_demo as (
select owner_name , object_type, to_clob( '{'|| string_agg ( object_oid, ', ') || '}' )array_text
from ownership
group by owner_name, object_type);
select top 10 * from array_imp_demo;
/*
OWNER_NAME OBJECT_TYPE ARRAY_TEXT
SYS TABLE {142540, 142631, 142262, 133562, 185300, 142388, 133718, 142872, 133267, 133913, 134330, 143063, 133386, 134042, 142097, 142556, 142688, 142284, 133577, 185357, 142409, 133745, 142902, 133308, 133948, 134359, 143099, 133411, 134135, 142118, 142578, 142762, 142306, 133604, 142429, 133764, 142928, 133322, 133966, 134383, 143120, 133443, 134193, 142151, 142587, 142780, 142327, 133642, 142448, 133805, 142967, 185407, 133334, 133988, 134624, 143134, 133455, 134205, 142173, 142510, 142606, 142798, 142236, 133523, 142359, 133663, 142465, 133825, 142832, 133175, 133866, 134269, 143005, 133354, 134012, 134646, 143148, 133497, 134231, 142195, 142526, 142628, 142816, 142259, 133551, 142382, 133700, 142493, 133855, 142862, 133235, 133904, 134309, 143051, 133373, 134029, 142082, 143306, 133513, 134255, 142216, 142553, 142684, 142281, 133572, 185330, 142394, 133738, 142892, 133299, 133929, 134351, 143080, 133406, 134117, 142115, 142576, 142711, 142303, 133596, 142414, 133756, 142922, 185399, 133319, 133958, 134368, 143115,
SYSTEM TABLE {154821, 146065, 146057, 173960, 174000, 173983, 144132, 173970}
_SYS_STATISTICS TABLE {151968, 146993, 147245, 152026, 147057, 147023, 151822, 147175, 147206, 151275, 151903, 147198, 147241, 151326, 151798, 152010, 147016, 147039, 147068, 151804, 151810, 147002, 147202, 151264, 151850, 147186, 147114, 147228, 151300, 151792, 151992, 147030, 147062, 151840, 147181, 147210, 151289, 151754, 149419, 150274, 147791, 150574, 147992, 149721, 150672, 148132, 149894, 147042, 148434, 149059, 150071, 147518, 148687, 149271, 150221, 150877, 147716, 148913, 150388, 147957, 149675, 150634, 148102, 149829, 148267, 148979, 149976, 147494, 148663, 151107, 149224, 150178, 147667, 148855, 149532, 150306, 147925, 149602, 150598, 148080, 149792, 148179, 149926, 147417, 148547, 149186, 150107, 147568, 148804}
_SYS_EPM TABLE {143391, 143427, 143354, 143385, 143411, 143376, 143398, 143367, 143414, 143344, 175755}
_SYS_REPO TABLE {145086, 151368, 171680, 145095, 152081, 169183, 151443, 149941, 154366, 143985, 145116, 152104, 151496, 169029, 177728, 179047, 145065, 169760, 178767, 151659, 169112, 169258, 153736, 177757, 174923, 145074, 150726, 151697, 169133, 178956, 145083, 169171, 168992, 145092, 177665, 169178, 151378, 169271, 178881, 174911, 154128, 143980, 145101, 152098, 151481, 177720, 152504, 145062, 151570, 169102, 154058, 145071, 170733, 151687, 169130, 145080, 171629, 169166, 178846, 145089, 149588, 151373, 177614, 143976, 145098, 152087, 151458, 149955, 178907, 154386, 145059, 169605, 151529, 169035, 178579, 151176, 179178, 145068, 150709, 151670, 169124, 174905, 177778, 154244, 145077, 170883, 169158, 144072, 152681, 144285, 154415, 144620, 145268, 168884, 144895, 143512, 151428, 168774, 143750, 152337, 168558, 144114, 149559, 152719, 144327, 144674, 145508, 168924, 144939, 143578, 152135, 143793, 152392, 168587, 144151, 152753, 144370, 144720, 145722, 168960, 144990, 143626, 152174, 143832, 152435, 168620, 144188, 152785,
_SYS_TASK TABLE {146851, 146847, 146856, 146231, 146143, 146839, 146854, 146834, 146430, 146464, 146167, 146505, 146205, 146257, 146384, 146313, 146420, 146356, 146454, 146155, 146495, 146193, 146525, 146244, 146368, 146296, 146410, 146346, 146443, 146148, 146484, 146181, 146517, 146234, 146275, 146395, 146325}
_SYS_AFL TABLE {177209, 176741, 176522, 177243, 176777, 176692, 176201, 177294, 176929, 177328, 176967, 177383, 177105, 177015, 176826, 177143, 177056, 176866, 177215, 176748, 176550, 176474, 177249, 176784, 176699, 176218, 146871, 177300, 176935, 177335, 176973, 177389, 177111, 177021, 176835, 177156, 177062, 176883, 185427, 177221, 176755, 176572, 176487, 177261, 176790, 176706, 176398, 177306, 176941, 177341, 176979, 177397, 177119, 177033, 176843, 177162, 177069, 176889, 177228, 176762, 176589, 177267, 176799, 176713, 176416, 177313, 176953, 177348, 176991, 177404, 177127, 177040, 176849, 177173, 177075, 176895, 177195, 176732, 176507, 177234, 176768, 177274, 176805, 176720, 176448, 177285, 176921, 177319, 176959, 177354, 176997, 177375, 177095, 177007, 176818, 177411, 177133, 177047, 176858, 177179, 177081, 176911, 177201, 176738, 176519, 177241, 176775, 176690, 176196, 177280, 176814, 176727, 176464, 177291, 176927, 177326, 176965, 177360, 177003, 177381, 177103, 177013, 176824, 177141, 177053, 176864, 177191, 177091,
_SYS_XB TABLE {146971, 146957}
DEVDUDE TABLE {167121, 165374, 182658, 156616, 173111, 181568, 174901, 183413, 184432, 183498, 183470, 184464, 155821, 173102, 183613, 184495, 155857, 166744, 180454, 184547, 156234, 172096, 166765, 165548, 184649, 183399, 184357, 184577, 183477, 181594, 183537, 181572, 167201, 184685, 185467, 183406, 184422, 184610, 183491, 155842, 172923, 157723, 182636, 167895, 183463, 184454, 183505, 165542, 183606, 184488, 155849, 172749, 157626, 184527, 183449, 166759, 184627, 182827, 184347, 184568, 157619, 172118, 183530, 181556, 167137, 184670, 182642, 184411, 184598, 183484, 155835, 183456, 183593, 181584, 167328, 183421, 184443, 183586, 184474, 155828, 166392, 183620, 184517, 183435, 183442, 183512, 166753, 184557, 156598, 172106, 183523, 166771, 166568, 184660, 182630, 184381, 184587, 183428, 157681, 182649, 167264, 168236}
ADMIN TABLE {158520, 158925, 158982, 158492, 158571, 158583, 158560, 158541, 158744}
*/
Ok, let's just assume that this is the data that you managed to load from your CSV file into a table and that the ARRAY_DATA column is a big CLOB.
Now, the data is in the table, but you need to put it into an ARRAY column.
alter table array_imp_demo add (array_data integer array);
The unconvenient part follows now: creating a new UPDATE command for each record to transform the CLOB data into a proper ARRAY() function call.
Be aware that this update command works correctly, because (OWNER, OBJECT_TYPE) are used as keys in this case.
do begin
declare vsqlmain nvarchar (50) :='UPDATE array_imp_demo set array_data = ARRAY ';
declare vsql nvarchar(5000);
declare cursor c_upd for
select OWNER_NAME, OBJECT_TYPE,
replace (replace (ARRAY_TEXT, '{', '(' ), '}', ')' ) array_data
from array_imp_demo;
for cur_row as c_upd do
vsql = :vsqlmain || cur_row.array_data;
vsql = :vsql || ' WHERE owner_name = ''' || cur_row.owner_name || '''';
vsql = :vsql || ' AND object_type = ''' || cur_row.object_type || '''';
exec :vsql;
end for;
end;
select * from array_imp_demo;
OWNER_NAME OBJECT_TYPE ARRAY_TEXT ARRAY_DATA
SYS TABLE {142540, 142631, 142262,... 142540, 142631, 142262,...
SYSTEM TABLE {154821, 146065, 146057,... 154821, 146065, 146057,...
_SYS_STATISTICS TABLE {151968, 146993, 147245,... 151968, 146993, 147245,...
_SYS_EPM TABLE {143391, 143427, 143354,... 143391, 143427, 143354,...
_SYS_REPO TABLE {145086, 151368, 171680,... 145086, 151368, 171680,...
After that you can drop the CLOB column.
Ok, that's about it from my side.
Have you tried using CTL method to import data. I hope that might help.
DB2 Procedures return result set after for loop
I'm currently on writing a SQL-Procedure on IBM i 5 V7R1. If I want to return a result set after my for loop I can't create the procedure. Here is the code: create or replace procedure test1() DYNAMIC RESULT SETS 1 LANGUAGE SQL SPECIFIC EDVVAEH1.test1 NOT DETERMINISTIC MODIFIES SQL DATA CALLED ON NULL INPUT SET OPTION ALWBLK = *ALLREAD , ALWCPYDTA = *OPTIMIZE , COMMIT = *NONE , DECRESULT = (31, 31, 00) , DFTRDBCOL = *NONE , DYNDFTCOL = *NO , DYNUSRPRF = *USER , SRTSEQ = *HEX proc: BEGIN DECLARE x VARCHAR(255); DECLARE return_cur CURSOR with return FOR select * from sysibm.sysdummy1; loop1: for record as C1 cursor for SELECT TABLE_NAME from qsys2.systables DO SET x = record.table_name; END FOR loop1; open return_cur; END proc As the options define, I want to return a result set after both loops, but it doesn't let me create this procedure. The Error is -104 token: "End of Statement" is not valid, but everything is there :(
It seems I found the root cause of my problem, but I can't fix it. As far as I can see, my i Access has a Problem with the for-statement. I copied the statement to my RDi 9.1 in the Database Development view and ran it there. I also set a different end of statement command and after that I was able to create my procedure. Now I need to wait until someone in my company can provide me a PTF for the client access. I already tried to install all PTFs delivered with the IBM i, but no improvements ... If anyone has an idea, where I can download a PTF for client Access please give me a hint. The IBM site is not possible for me since, I am not registered for out IBM i. Thanks.
How to display the value of a variable used in a PL/SQL block in the log file of a batch file? And Arrays?
Basically I'm using a batch file to run a .sql file on Windows Task Scheduler. The batch file generates a log file that displays all the put_lines. I now want to also see the value assigned to the variable: v_chks_dm, but couldn't figure out a way to do it. Tried the get_line statement but failed... Does anyone know how to do it?
thanks!
This is what's in the batch file:
echo off
echo ****************************>>C:\output.log
sqlplus userid/password#csdpro #V:\CONDITION_TEST.sql>>C:\output.log
Here's the .sql file
declare
v_chks_dm number;
begin
Select /*+parallel (a,4)*/ count(distinct a.src_table) into v_chks_dm
from hcr_dm.hcr_dm_fact a;
dbms_output.put_line('v_chkt_dm value assigned');
-- dbms_output.get_line(v_chks_dm);
if.... then... else.... end if;
end;
One more question... what if the variable is an array? I have something like this, but got an error says ORA-06533: Subscript beyond count. The number of values in the array usually varies from 0 to 10, but could be more. Thanks!
declare
type v_chks_array is varray(10) of varchar2(50);
arrSRCs v_chks_array;
begin
arrSRCs :=v_chks_array();
arrSRCs.EXTEND(10);
Select /*+parallel (a,4)*/ distinct a.src_table BULK collect into arrSRCs
from hcr_dm.hcr_dm_fact a;
dbms_output.put_line(arrSRCs(10));
end;
dbms_output.put_line('v_chkt_dm value = ' || v_chkt_dm);
or, better
dbms_output.put_line('v_chkt_dm value = ' || to_char(v_chkt_dm, '<number format>'));
You can choose appropriate number formats in documentation.