I'm using BigQuery's DynamicSQL to dynamically construct queries so that I can re-use code.
In my use case, I'm trying to build a re-usable SQL script to get meta data of all tables in a project.
Where I'm getting stuck is trying to populate an array with the results of an EXECUTE IMMEDIATE command.
Here is a simple snippet which illustrates my problem:
DECLARE project_name string;
DECLARE schema_query string;
DECLARE schemas ARRAY<string>;
SET project_name = 'my_project';
-- Get all the schemas in a project
set schema_query = format("""
select schema_name from `%s.INFORMATION_SCHEMA.SCHEMATA`""",
project_name
);
-- Add the schemas to an array
-- THIS COMMAND FAILS
set schemas = ARRAY(execute immediate schema_query)
One way to get the desired result, but which is not dynamic, is:
SET schemas = ARRAY(select schema_name from `my_project.INFORMATION_SCHEMA.SCHEMATA`);
So I want to be able to pass project_name as a parameter and then run the dynamic sql query, which returns a single column of type string, and save the results in an array of strings.
Consider below approach
declare project string;
declare schemas array<string>;
set project = 'my_project';
execute immediate 'select array_agg(schema_name) from `' || project || '.INFORMATION_SCHEMA.SCHEMATA`' into schemas;
select * from unnest(schemas);
Related
I'am an SQL rookie and just started my Apprenticeship in the CMD Team of my Company.
It's my first time working with Oracle SQL-dev and I never coded Functions/Procedures in SQL before
So my Sensei gave me a task to solve for myself which is:
To create a stored procedure which generates views for all of my Synonyms in the current Scheme. If executed again it should Replace the current views of those Synonyms.
As second part of the task I should also add a function to DROP all the views i don't have a Synonym for
Yes I already discussed with her if it is useful to create views this way or not
The first thing i found out that it is not possible to create views from a stored procedure the traditional way and that i have to use a workaround method with EXEC() for example
my Sensei gave me a code Snippet to begin with:
FOR KO IN(SELECT * FROM all_synonyms WHERE OWNER = 'CMD_SANDBOX')
She told me i have to fill Variables with the Names of the Synonyms .. so far so good
it makes sense because i have to generate unique names for the views as well as the procedure to know which is the current synonym to create a view of
CREATE OR REPLACE PROCEDURE CREATE_VIEWS AS
DECLARE #viewCommand varchar(1000)
DECLARE #viewName varchar(75)
DECLARE #synonymName varchar(75)
SET #viewCommand = 'CREATE OR REPLACE VIEW' + #viewName + 'AS SELECT * FROM '
+#synonymName
BEGIN
FOR KO IN(SELECT * FROM all_synonyms WHERE OWNER = 'CMD_SANDBOX')
SET #synonymName = <Pointer on Synonym>
SET #viewName = 'v_' + #synonymName
EXEC(#viewCommand)
END LOOP KO;
END CREATE_VIEWS
Long story short...
My questions are:
How do I Point to a Certain Object without using its specific name to fill my #synonymName ?
Is the For Loop header already complete ? I kinda don't get how it works in SQL
How do you pros research this stuff? I feel pretty confident in queries but as it comes to specific things like pointing to objects or similar it is pretty hard to find out.
Your syntax is invalid in Oracle as:
there are multiple DECLAREs but only a single BEGIN/END block;
# is not valid in (unquoted) variable names;
+ is the numeric addition operator and does not concatenate strings. To concatenate strings you want to use the || operator;
assignment in PL/SQL does not require SET and requires := instead of =; and
you want EXECUTE IMMEDIATE and not EXEC.
If you fix all that then there are still issues as VIEW' + #viewName + 'AS should have spaces between the VIEW and AS keywords and the identifier.
The syntax looks more like SQL Server than Oracle.
For Oracle, you want:
CREATE OR REPLACE PROCEDURE CREATE_VIEWS
AS
BEGIN
FOR KO IN(
SELECT synonym_name
FROM all_synonyms
WHERE OWNER = 'CMD_SANDBOX'
)
LOOP
EXECUTE IMMEDIATE 'CREATE OR REPLACE VIEW v_' || ko.synonym_name
|| ' AS SELECT * FROM ' || ko.synonym_name;
END LOOP;
END CREATE_VIEWS;
/
In Oracle's SQL Developer, I can execute a "dynamic" SQL select statement in the Script Output pane with something like:
script
var tabName = 'all_users';
sqlcl.setStmt('select * from ' + tabName);
sqlcl.run();
/
Now, I am wondering if it is possible to execute a dynamic select statement such that its result is displayed in the result grid.
SqlDev is implemented in Java which includes Nashorn scripting engine. Therefore, you can basically execute JDBC in your Nashorn script. Here is one way to do it.
Paste
select REGIONS,
COUNTRIES,
LOCATIONS,
DEPARTMENTS,
JOBS,
EMPLOYEES,
JOB_HISTORY,
TM_USER_INFO,
USER_ROLES,
PAYMENT_PRICE_SHOP,
SHOP_USER from dual
into your worksheet.
Open "Code Outline" panel. Type "arbori", or open the following "querySQL.arbori" file:
include "std.arbori"
prelude: runOnce -> {
var ConnectionResolver = Java.type('oracle.dbtools.db.ConnectionResolver');
var connectionList = ConnectionResolver.getConnectionNames();
var conn = ConnectionResolver.getConnection('IdeConnections%23local_hr');
}
queries: [node) column -> {
var node = tuple.get('node');
var table = target.input.substring(
target.src[node.from].begin,
target.src[node.to-1].end
);
var query = 'select * from ' + table;
var ps = conn.createStatement();
var rs = ps.executeQuery(query);
while( rs.next() )
print(rs.getString(1));
}
It outputs the first column of each table to the standard java output
, so it would require some reverse engineering to get a handle to the SqlDev script output panel and channel the result there.
Dynamic SQL can be displayed in an Oracle SQL Developer grid using a function returning a ref cursor (sort of), a polymorphic table function (18c+), or Oracle Data Cartridge (requires custom PL/SQL packages and types).
Function Returning Ref Cursor
As explained in this answer, the output from a function returning a ref cursor can be displayed in the "Output Variables" window. The example from the answer used static SQL, but it's pretty easy to make it dynamic:
create or replace function refcursor_function return sys_refcursor as
c sys_refcursor;
begin
open c for 'select * from all_objects';
return c;
end;
/
The downside is that getting the results takes a few more clicks than a normal query, and the Output Variables grid is not nearly as powerful as the regular results grid. If you just need a window for viewing and copying and pasting, Output Variables is good enough. But it doesn't allow any advanced GUI features.
Polymorphic Table Function
Since Oracle 18c, you can create a polymorphic table function that accepts input and has variable output. You have to program how to handle the tables and columns, but if you just need simple pass-through logic it's not that difficult. See my answer here for an example of a query that returns every column from a table excluding specific columns. The results are "regular" SQL as far as any program knows, and will work in any grid GUI.
Oracle Data Cartridge
My open source program Method4 can run dynamic SQL in SQL. After you download and install the packages and types, you can write a SQL statement that generates another SQL statement. If you need to use PL/SQL to generate the query, you may need to use a PL/SQL WITH function. Like the polymorphic table function, the results look like normal SQL and will work in any grid.
select * from table(method4.dynamic_query(
q'[
--Query that builds another query.
select replace(
q'!
select * from #TABLE_NAME#
!', '#TABLE_NAME#', table_name) sql_statement
from
(
--Enter your script here that returns a table name to select from.
select 'ALL_USERS' table_name
from dual
)
]'
));
You may want to add some details about what exactly you're trying to do and why; that might help narrow down the possible solutions.
I'm writing a procedure to count rows in every table in my database. It so far looks like this:
create or replace procedure count_database_rows()
dynamic result sets 1
P1: begin atomic
DECLARE stmt CHAR(40);--
FOR v1 AS
c1 CURSOR FOR
SELECT TABLE_SCHEMA, TABLE_NAME FROM sysibm.tables
DO
SET stmt = 'SELECT COUNT(*) FROM '||TABLE_SCHEMA||'.'||TABLE_NAME;--
PREPARE s FROM stmt;--
EXECUTE s;--
END FOR;--
end P1
~
however, when I run it:
db2 -ntd~ -f script.sql > dump.csv
all I'm getting is:
DB20000I The SQL command completed successfully.
how can I print all results instead?
Just for demonstration. I assume, that it's some educational task, and it's Db2 for LUW.
For non-DPF Db2 for LUW systems only
--#SET TERMINATOR #
CREATE OR REPLACE FUNCTION COUNT_DATABASE_ROWS()
RETURNS TABLE (P_TABSCHEMA VARCHAR(128), P_TABNAME VARCHAR(128), P_ROWS BIGINT)
BEGIN
DECLARE L_STMT VARCHAR(256);
DECLARE L_ROWS BIGINT;
FOR V1 AS
SELECT TABSCHEMA, TABNAME
FROM SYSCAT.TABLES
WHERE TYPE IN ('T', 'S')
FETCH FIRST 10 ROWS ONLY
DO
SET L_STMT = 'SET ? = (SELECT COUNT(*) FROM "'||V1.TABSCHEMA||'"."'||V1.TABNAME||'")';
PREPARE S FROM L_STMT;
EXECUTE S INTO L_ROWS;
PIPE(V1.TABSCHEMA, V1.TABNAME, L_ROWS);
END FOR;
RETURN;
END#
SELECT * FROM TABLE(COUNT_DATABASE_ROWS())#
For any Db2 for LUW systems
A little bit tricky for DPF systems, but doable as well. We have to wrap the code which is not allowed in the inlined compound statement into the stored procedure.
--#SET TERMINATOR #
CREATE OR REPLACE PROCEDURE COUNT_DATABASE_ROWS_DPF(OUT P_DOC XML)
READS SQL DATA
BEGIN
DECLARE L_STMT VARCHAR(256);
DECLARE L_ROWS BIGINT;
DECLARE L_NODE XML;
SET P_DOC = XMLELEMENT(NAME "DOC");
FOR V1 AS
SELECT TABSCHEMA, TABNAME
FROM SYSCAT.TABLES
WHERE TYPE IN ('T', 'S')
FETCH FIRST 10 ROWS ONLY
DO
SET L_STMT = 'SET ? = (SELECT COUNT(*) FROM "'||V1.TABSCHEMA||'"."'||V1.TABNAME||'")';
PREPARE S FROM L_STMT;
EXECUTE S INTO L_ROWS;
SET L_NODE = XMLELEMENT
(
NAME "NODE"
, XMLELEMENT(NAME "TABSCHEMA", V1.TABSCHEMA)
, XMLELEMENT(NAME "TABNAME", V1.TABNAME)
, XMLELEMENT(NAME "ROWS", L_ROWS)
);
SET P_DOC = XMLQUERY
(
'transform copy $mydoc := $doc modify do insert $node as last into $mydoc return $mydoc'
passing P_DOC as "doc", L_NODE as "node"
);
END FOR;
END#
CREATE OR REPLACE FUNCTION COUNT_DATABASE_ROWS_DPF()
RETURNS TABLE (P_TABSCHEMA VARCHAR(128), P_TABNAME VARCHAR(128), P_ROWS BIGINT)
BEGIN ATOMIC
DECLARE L_DOC XML;
CALL COUNT_DATABASE_ROWS_DPF(L_DOC);
RETURN
SELECT *
FROM XMLTABLE ('$D/NODE' PASSING L_DOC AS "D" COLUMNS
TYPESCHEMA VARCHAR(128) PATH 'TABSCHEMA'
, TABNAME VARCHAR(128) PATH 'TABNAME'
, LENGTH BIGINT PATH 'ROWS'
);
END#
-- Usage. Either CALL or SELECT:
CALL COUNT_DATABASE_ROWS_DPF(?)#
SELECT * FROM TABLE(COUNT_DATABASE_ROWS_DPF())#
If your Db2-server runs on Linux/Unix/Windows then you can use the DBMS_OUT.PUT_LINE function to send diagnostic output from SQL routines to the console. The idea is that in your routine, you assign to a variable some text (example, the table name and its count), then call DBMS_OUTPUT.PUT_LINE(...) to cause that text to appear on the console. The disadvantage of this approach is that the output will only appear once the routine has completed. This is often not what you want, sometimes you want to see the row-counts as they become available, so consider instead alternative approaches, as shown below.
To see DBMS_OUTPUT.PUT_LINE output with the Db2 CLP (or db2cmd.exe) you first need to use set serveroutput on before calling the procedure.
But for simple stuff like this, a stored procedure might be unsuitable, because you can use the CLP to do the work in two steps after connecting to the database. This is often more convenient for scripting purposes. The idea is that you make a file to generate the queries, which when you run with CLP creates a second file, and you execute the second file to get the desired results.
Example
Create file gen_counts.sql containing the query that generates the real queries, for example gen_counts.sql might contain
select 'select count(*) from '||rtrim(tabschema)||'.'||rtrim(tabname)||' with ur;'
from syscat.tables;
Then you can do these steps:
db2 connect to $database
db2 -txf gen_counts.sql > count_queries.sql
db2 -tvf count_queries.sql > count_results.txt
Note that the output file (in this case count_results.txt) is readable via another shell session while the script continues to run. You can also pipe the output to concurrent jobs if required.
However, experienced DBAs might avoid row-counting all tables in this manner, and might choose instead to ensure that the runstats are always up-to-date for all tables, and accept recent estimates of row counts, which are visible in SYSCAT.TABLES.CARD once runstats are completed. If the stats are up to date, the CARD count is often good enough for many purposes. If exact counts are required, they are often valid only for a specific timestamp if the database is live.
I'm creating an update statement that generate SHA256 for table columns based on table's name
1st Step: I created a procedure that get the table columns, concatenate it all in one columns, then format to a desired format.
-- Procedure code : Extract table's columns list, concatenate it and format it
Create procedure SHA_PREP (in inp1 nvarchar(20))
as
begin
SELECT concat(concat('hash_sha256(',STRING_AGG(A, ', ')),')') AS Names
FROM (
SELECT concat('to_varbinary(IFNULL("',concat(COLUMN_NAME,'",''0''))')) as A
FROM SYS.TABLE_COLUMNS
WHERE SCHEMA_NAME = 'SCHEMA_NAME' AND TABLE_NAME = :inp1
AND COLUMN_NAME not in ('SHA')
ORDER BY POSITION
);
end;
/* Result of this procedures :
hash_sha256(
to_varbinary("ID"),to_varbinary(IFNULL("COL1",'0')),to_varbinary(IFNULL("COL2",'0')) )
*/
-- Update Statement needed
UPDATE "SCHEMA_NAME"."TABLE_NAME"
SET "SHA" = CALL "SCHEMA_NAME"."SHA_PREP"('SCHEMA_NAME')
WHERE "ID" = 99 -- a random filter
The solution by #SonOfHarpy technically works but has several issues, namely:
unnecessary use of temporary tables
overly complicated string assignment approach
use of fixed system table schema (SYS.TABLE_COLUMNS) instead of PUBLIC synonym
wrong data type and variable name for the input parameter
An improved version of the code looks like this:
create procedure SHA_PREP (in TABLE_NAME nvarchar(256))
as
begin
declare SQL_STR nvarchar(5000);
SELECT
'UPDATE "SCHEMA_NAME"."TABLE_NAME" SET "SHA"= hash_sha256(' || STRING_AGG(A, ', ') || ')'
into SQL_STR
FROM (
SELECT
'TO_VARBINARY(IFNULL("'|| "COLUMN_NAME" ||'",''0''))' as A
FROM TABLE_COLUMNS
WHERE
"SCHEMA_NAME" = 'SCHEMA_NAME'
AND "TABLE_NAME" = :TABLE_NAME
AND "COLUMN_NAME" != 'SHA'
ORDER BY POSITION
);
-- select :sql_str from dummy; -- this is for debugging output only
EXECUTE IMMEDIATE (:SQL_STR);
end;
By changing the CONCAT functions to the shorter || (double-pipe) operator, the code becomes a lot easier to read as the formerly nested function calls are now simple chained concatenations.
By using SELECT ... INTO variable the whole nonsense with the temporary table can be avoided, again, making the code easier to understand and less prone to problems.
The input parameter name now correctly reflects its meaning and mirrors the HANA dictionary data type for TABLE_NAME (NVARCHAR(256)).
The procedure now consists of two commands (SELECT and EXECUTE IMMEDIATE) that each performs an essential task of the procedure:
Building a valid SQL update command string.
Executing the SQL command.
I removed the useless line-comments but left a debugging statement as a comment in the code, so that the SQL string can be reviewed without having to execute the command.
For that to work, obviously, the EXECUTE... line needs to be commented out and the debugging line has to be uncommented.
What's more worrying than the construction of the solution is its purpose.
It looks as if the SHA column should be used as a kind of shorthand row-data fingerprint. The UPDATE approach certainly handles this as an after-thought activity but leaves the "finger-printing" for the time when the update gets executed.
Also, it takes an essential part of the table design (that the SHA column should contain the fingerprint) away from the table definition.
An alternative to this could be a GENERATED COLUMN:
create table test (aaa int, bbb int);
alter table test add (sha varbinary (256) generated always as
hash_sha256(to_varbinary(IFNULL("AAA",'0'))
, to_varbinary(IFNULL("BBB",'0'))
)
);
insert into test (aaa, bbb) values (12, 32);
select * from test;
/*
AAA BBB SHA
12 32 B6602F58690CA41488E97CD28153671356747C951C55541B6C8D8B8493EB7143
*/
With this, the "generator" approach could be used for table definition/modification time, but all the actual data handling would be automatically done by HANA, whenever values get changed in the table.
Also, no separate calls to the procedure will ever be necessary as the fingerprints will always be current.
I find a solution that suits my need, but maybe there's other easier or more suitable approchaes :
I added the update statement to my procedure, and inserted all the generated query into a temporary table column, the excuted it using EXECUTE IMMEDIATE
Create procedure SHA_PREP (in inp1 nvarchar(20))
as
begin
/* ********************************************************** */
DECLARE SQL_STR VARCHAR(5000);
-- Create a temporary table to store a query in
create local temporary table #temp1 (QUERY varchar(5000));
-- Insert the desirable query into the QUERY column (Temp Table)
insert into #temp1(QUERY)
SELECT concat('UPDATE "SCHEMA_NAME"."TABLE_NAME" SET "SHA" =' ,concat(concat('hash_sha256(',STRING_AGG(A, ', ')),')'))
FROM (
SELECT concat('to_varbinary(IFNULL("',concat(COLUMN_NAME,'",''0''))')) as A
FROM SYS.TABLE_COLUMNS
WHERE SCHEMA_NAME = 'SCHEMA_NAME' AND TABLE_NAME = :inp1
AND COLUMN_NAME not in ('SHA')
ORDER BY POSITION
);
end;
/* QUERY : UPDATE "SCHEMA_NAME"."TABLE_NAME" SET "SHA" =
hash_sha256(to_varbinary("ID"),to_varbinary(IFNULL("COL1",'0')),to_varbinary(IFNULL("COL2",'0'))) */
SELECT QUERY into SQL_STR FROM "SCHEMA_NAME".#temp1;
--Excuting the query
EXECUTE IMMEDIATE (:SQL_STR);
-- Dropping the temporary table
DROP TABLE "SCHEMA_NAME".#temp1;
/* ********************************************************** */
end;
Any other solution or improvement are well welcomed
Thank you
i want to use the result of a query as an input in another query.
What might make it difficult: The variable is the schema in the database.
CREATE or replace VARIABLE myschema varchar(15) ;
set myschema = (select owner from syscat.tables where tabname = 'xyz');
select count(name) as result from myschema.USR02 where USTYP = 'A';
DROP VARIABLE myschema;
This is my last try, after i failed using declare.
But i get an error, because "myschema" is used as a string, and of course there is no schema with name "myschema". The result of the first query is not used.
If I just run the first two lines, i get the schemaname as result. Do i have to mark the variable in a special way? The goal is just the result of the query in line 3 by using the dynamic value of "myschema".
Unfortunately, you have to use dynamic SQL (forming a custom SQL query through string manipulation) if you want to deal with table, schema, or column names dynamically:
This is the basic idea:
execute immediate 'select * from ' || myschema || '.USR02';
However, you can't just run a bare select in dynamic SQL; you have to put the result in something. And the whole thing must be in a compound SQL block. So the full example would look something like this (simplified query for space).
This query assumes that a table called "result" exists to store the result you are returning.
begin
declare myschema varchar(100) default '';
set myschema = (select owner from syscat.tables where tabname = 'xyz');
execute immediate 'insert into result select count(*) from ' || myschema || '.USR02';
end
select * from result;
Note that within the block, you can simply declare a variable (as shown in my example). So you don't have to declare a global variable for this purpose, unless you want it to persist beyond this one statement.