How to use query created in stored procedure in other procedure - sql

sql procedure creates temporary table which should used in other sql procedure.
I tried
CREATE or replace FUNCTION f_createquery()
RETURNS TABLE ( kuupaev date
) AS $f_createquery$
-- actually this is big and time consuming select statement which should evaluated only once:
select current_date as kuupaev
$f_createquery$ LANGUAGE sql STABLE;
CREATE or replace FUNCTION f_usequery()
RETURNS TABLE ( kuupaev date
) AS $f_usequery$
-- big query tehing is used several times in query:
select kuupaev from tehing
union all
select kuupaev+1 from tehing
union all
select kuupaev+2 from tehing
$f_usequery$ LANGUAGE sql STABLE;
with tehing as (
select * from f_createquery() _
)
select * from f_usequery() _
but got error
ERROR: relation "tehing" does not exist
tehing contains temporary data created by stored procedure and it does not exist in database. How to allow other stored procedure to use it ?
How to fix it for Postgres 9.1+ ?
Is there simething like
external table (kuupaev date)
which allows to define external tables?
In real applicaton select statement in f_createquery is big and time consuming and should evaluated only once.
Replacing select * from tehing to dynamic sql in f_usequery() probably works but this prevents procedure compiling at run time. Is there better solution or can tehing passed in better way to other stored procedure, e.q. like parameter ?
Or should f_createquery create temporary table with fixed name tehing?

If f_createquery() produces the temporary table tehing then it should not return anything. Your function should look like:
CREATE FUNCTION f_createquery() RETURNS void AS $f_createquery$
CREATE TEMPORARY TABLE tehing AS
SELECT current_date AS kuupaev; -- complex query here
$f_createquery$ LANGUAGE sql STABLE;
You can then use the tehing table in the current session like any other table:
CREATE FUNCTION f_usequery() RETURNS TABLE (kuupaev date) AS $f_usequery$
SELECT kuupaev FROM tehing
UNION ALL
SELECT kuupaev+1 FROM tehing
UNION ALL
SELECT kuupaev+2 FROM tehing;
$f_usequery$ LANGUAGE sql STABLE;
Note that the temporary table is dropped at the end of the session.
You can also integrate the 2 functions so you can call f_usequery() without worrying about calling another function first:
CREATE FUNCTION f_usequery() RETURNS TABLE (kuupaev date) AS $f_usequery$
BEGIN
PERFORM 1 FROM information_schema.tables WHERE table_name = 'tehing';
IF NOT FOUND THEN -- temp table tehing does not exist
CREATE TEMPORARY TABLE tehing AS
SELECT current_date AS kuupaev; -- etc, etc
END IF;
RETURN QUERY
SELECT kuupaev FROM tehing
UNION ALL
SELECT kuupaev+1 FROM tehing
UNION ALL
SELECT kuupaev+2 FROM tehing;
END; $f_usequery$ LANGUAGE plpgsql STABLE;
Note that this is now a plpgsql function so the syntax is slightly different.
The construct
with tehing as (
select * from f_createquery() _
)
select * from f_usequery() _
won't work, because you re-declare tehing as the result of the CTE. Instead, f_usequery() works with the tehing temporary table and you can select from it or do further analysis with the result from f_usequery():
SELECT f_createquery(); -- this creates the tehing temporary table
SELECT * FROM f_usequery(); -- this operates on the tehing table and returns some results
SELECT *
FROM tableX
JOIN f_usequery() USING (kuupaev)
WHERE kuupaev < '2015-09-19';

Simple case: CTE
If you can do everything in a single query, you typically don't need a temp table nor a function at all. A CTE does the job:
WITH tehing AS (
SELECT current_date AS kuupaev -- expensive query here
)
SELECT kuupaev FROM tehing
UNION ALL
SELECT kuupaev+1 FROM tehing
UNION ALL
SELECT kuupaev+2 FROM tehing;
If you need a temp table
A temporary table only makes sense if you actually have to run multiple queries using the result of the expensive query. Or if you need to create an index on the table or something.
Just try to create the temp table:
CREATE TEMP TABLE tehing AS
SELECT current_date AS kuupaev; -- expensive query here
If the table already exists you get an error, and that's fine as well.
Then run your queries using tehing.
If you really need a function that avoids this error (which I doubt):
CREATE FUNCTION f_create_tbl()
RETURNS text AS
$func$
BEGIN
IF to_regclass('pg_temp.tehing') IS NULL THEN -- temp table does not exist
CREATE TEMP TABLE tehing AS
SELECT current_date AS kuupaev; -- expensive query here
RETURN 'Temp table "tehing" created.';
ELSE
RETURN 'Temp table "tehing" already exists';
END IF;
END
$func$ LANGUAGE plpgsql;
Call:
SELECT f_create_tbl();
The function cannot be declared STABLE, it's a VOLATILE function (default).
A test like this would be subtly inaccurate:
PERFORM 1 FROM information_schema.tables WHERE table_name = 'tehing';
It would find any table named "tehing" in the search path. But you are only interested if a temporary table of that name exists. The temporary schema is first in the search path by default:
How does the search_path influence identifier resolution and the "current schema"
Related:
How to check if a table exists in a given schema

Related

Dynamic query that uses CTE gets "syntax error at end of input"

I have a table that looks like this:
CREATE TABLE label (
hid UUID PRIMARY KEY DEFAULT UUID_GENERATE_V4(),
name TEXT NOT NULL UNIQUE
);
I want to create a function that takes a list of names and inserts multiple rows into the table, ignoring duplicate names, and returns an array of the IDs generated for the rows it inserted.
This works:
CREATE OR REPLACE FUNCTION insert_label(nms TEXT[])
RETURNS UUID[]
AS $$
DECLARE
ids UUID[];
BEGIN
CREATE TEMP TABLE tmp_names(name TEXT);
INSERT INTO tmp_names SELECT UNNEST(nms);
WITH new_names AS (
INSERT INTO label(name)
SELECT tn.name
FROM tmp_names tn
WHERE NOT EXISTS(SELECT 1 FROM label h WHERE h.name = tn.name)
RETURNING hid
)
SELECT ARRAY_AGG(hid) INTO ids
FROM new_names;
DROP TABLE tmp_names;
RETURN ids;
END;
$$ LANGUAGE PLPGSQL;
I have many tables with the exact same columns as the label table, so I would like to have a function that can insert into any of them. I'd like to create a dynamic query to do that. I tried that, but this does not work:
CREATE OR REPLACE FUNCTION insert_label(h_tbl REGCLASS, nms TEXT[])
RETURNS UUID[]
AS $$
DECLARE
ids UUID[];
query_str TEXT;
BEGIN
CREATE TEMP TABLE tmp_names(name TEXT);
INSERT INTO tmp_names SELECT UNNEST(nms);
query_str := FORMAT('WITH new_names AS ( INSERT INTO %1$I(name) SELECT tn.name FROM tmp_names tn WHERE NOT EXISTS(SELECT 1 FROM %1$I h WHERE h.name = tn.name) RETURNING hid)', h_tbl);
EXECUTE query_str;
SELECT ARRAY_AGG(hid) INTO ids FROM new_names;
DROP TABLE tmp_names;
RETURN ids;
END;
$$ LANGUAGE PLPGSQL;
This is the output I get when I run that function:
psql=# select insert_label('label', array['how', 'now', 'brown', 'cow']);
ERROR: syntax error at end of input
LINE 1: ...SELECT 1 FROM label h WHERE h.name = tn.name) RETURNING hid)
^
QUERY: WITH new_names AS ( INSERT INTO label(name) SELECT tn.name FROM tmp_names tn WHERE NOT EXISTS(SELECT 1 FROM label h WHERE h.name = tn.name) RETURNING hid)
CONTEXT: PL/pgSQL function insert_label(regclass,text[]) line 19 at EXECUTE
The query generated by the dynamic SQL looks like it should be exactly the same as the query from static SQL.
I got the function to work by changing the return value from an array of UUIDs to a table of UUIDs and not using CTE:
CREATE OR REPLACE FUNCTION insert_label(h_tbl REGCLASS, nms TEXT[])
RETURNS TABLE (hid UUID)
AS $$
DECLARE
query_str TEXT;
BEGIN
CREATE TEMP TABLE tmp_names(name TEXT);
INSERT INTO tmp_names SELECT UNNEST(nms);
query_str := FORMAT('INSERT INTO %1$I(name) SELECT tn.name FROM tmp_names tn WHERE NOT EXISTS(SELECT 1 FROM %1$I h WHERE h.name = tn.name) RETURNING hid', h_tbl);
RETURN QUERY EXECUTE query_str;
DROP TABLE tmp_names;
RETURN;
END;
$$ LANGUAGE PLPGSQL;
I don't know if one way is better than the other, returning an array of UUIDs or a table of UUIDs, but at least I got it to work one of those ways. Plus, possibly not using a CTE is more efficient, so it may be better to stick with the version that returns a table of UUIDs.
What I would like to know is why the dynamic query did not work when using a CTE. The query it produced looked like it should have worked.
If anyone can let me know what I did wrong, I would appreciate it.
... why the dynamic query did not work when using a CTE. The query it produced looked like it should have worked.
No, it was only the CTE without (required) outer query. (You had SELECT ARRAY_AGG(hid) INTO ids FROM new_names in the static version.)
There are more problems, but just use this query instead:
INSERT INTO label(name)
SELECT unnest(nms)
ON CONFLICT DO NOTHING
RETURNING hid;
label.name is defined UNIQUE NOT NULL, so this simple UPSERT can replace your function insert_label() completely.
It's much simpler and faster. It also defends against possible duplicates from within your input array that you didn't cover, yet. And it's safe under concurrent write load - as opposed to your original, which might run into race conditions. Related:
How to use RETURNING with ON CONFLICT in PostgreSQL?
I would just use the simple query and replace the table name.
But if you still want a dynamic function:
CREATE OR REPLACE FUNCTION insert_label(_tbl regclass, _nms text[])
RETURNS TABLE (hid uuid)
LANGUAGE plpgsql AS
$func$
BEGIN
RETURN QUERY EXECUTE format(
$$
INSERT INTO %s(name)
SELECT unnest($1)
ON CONFLICT DO NOTHING
RETURNING hid
$$, _tbl)
USING _nms;
END
$func$;
If you don't need an array as result, stick with the set (RETURNS TABLE ...). Simpler.
Pass values (_nms) to EXECUTE in a USING clause.
The tablename (_tbl) is type regclass, so the format specifier %I for format() would be wrong. Use %s instead. See:
Table name as a PostgreSQL function parameter

How to use one sql parameter to represent input array

Is there a way to write sql for Oracle, MS SQL:
Select * from table where id in(:arr)
Select * from table where id in(#arr)
With one param in sql 'arr' to represent an array of items?
I found examples that explode arr to #arr0,.., #arrn and feed array as n+1 separate parameters, not array, like this
Select * from table where id in(:arr0, :arr1, :arr2)
Select * from table where id in(#arr0, #arr1, #arr2)
Not what i want.
These will cause change in sql query and this creates new execution plans based on number of parameter.
I ask for .net, c# and Oracle and MS SQL.
Thanks for constructive ideas!
/ip/
I believe Table Value Parameter is good option for this case. Have a look at a sample code below in SQL Server.
-- Your table
CREATE TABLE SampleTable
(
ID INT
)
INSERT INTO SampleTable VALUES
(1010),
(2010),
(3010),
(4010),
(5010),
(6010),
(7010),
(8030)
GO
-- Create a TABLE type in SQL database which you can fill from front-end code
CREATE TYPE ParameterTableType AS TABLE
(
ParameterID INT
--, some other columns
)
GO
-- Create a stored proc using table type defined above
CREATE PROCEDURE ParameterArrayProcedure
(
#ParameterTable AS ParameterTableType READONLY
)
AS
BEGIN
SELECT
S.*
FROM SampleTable S
INNER JOIN #ParameterTable P ON S.ID = P.ParameterID
END
GO
-- Populated table type variable
DECLARE #ParameterTable AS ParameterTableType
INSERT INTO #ParameterTable (ParameterID) VALUES (1010), (4010), (7010)
EXECUTE ParameterArrayProcedure #ParameterTable
DROP PROCEDURE ParameterArrayProcedure
DROP TYPE ParameterTableType
DROP TABLE SampleTable
GO
Apart from Table Value Parameter, you can also use Json or XML values as SQL parameter but yes, it will definitely change your execution plan accordingly.
In addition to a Table Valued Parameter as Steve mentioned, there are a couple of other techniques available. For example you can parse a delimited string
Example
Declare #arr varchar(50) = '10,20,35'
Select A.*
From YourTable A
Join string_split(#arr,',') B on A.ID=value
Or even
Select A.*
From YourTable A
Where ID in ( select value from string_split(#arr,',') )
Oracle
In other languages (i.e. Java) you can pass an SQL collection as a bind parameter and directly use it in an SQL statement.
However, C# does not support passing SQL collections and only supports passing OracleCollectionType.PLSQLAssociativeArray (documentation link) which is a PL/SQL only data-type and cannot be used (directly) in SQL statements.
To pass an array, you would need to pass a PLSQLAssociativeArray to a PLSQL stored procedure and use that to convert it to an SQL collection that you can use in an SQL statement. An example of a procedure to convert from a PL/SQL associative array to an SQL collection is:
CREATE TYPE IntList AS TABLE OF INTEGER
/
CREATE PACKAGE tools IS
TYPE IntMap IS TABLE OF INTEGER INDEX BY BINARY_INTEGER;
FUNCTION IntMapToList(
i_map IntMap
) RETURN IntList;
END;
/
CREATE PACKAGE BODY tools IS
FUNCTION IntMapToList(
i_map IntMap
) RETURN IntList
IS
o_list IntList := IntList();
i BINARY_INTEGER;
BEGIN
IF i_map IS NOT NULL THEN
i := o_list.FIRST;
WHILE i IS NOT NULL LOOP
o_list.EXTEND;
o_list( o_list.COUNT ) := i_map( i );
i := i_map.NEXT( i );
END LOOP;
END IF;
RETURN o_list;
END;
END;
/

Temporary table postgresql function

I can't find a clear explanation of the syntax to create (and use) tables just for the inside calculations of a function. Could anyone give me a syntax exemple please ?
From what I've found, I have tried this (with and without # before temp_table) :
CREATE FUNCTION test.myfunction()
RETURNS SETOF test.out_table
AS $$
DECLARE #temp_table TABLE
(
id int,
value text
)
BEGIN
INSERT INTO #temp_table
SELECT id, value
FROM test.another_table;
INSERT INTO test.out_table
SELECT id, value
FROM #temp_table;
RETURN END
$$ LANGUAGE SQL;
I get :
ERROR: syntax error at or near "DECLARE"
LINE 5: DECLARE #temp_table TABLE
-
I also tried the CREATE TABLE approach suggested here, this way :
CREATE FUNCTION test.myfunction()
RETURNS SETOF test.out_table
AS $$
CREATE TABLE temp_table AS
SELECT id, value
FROM test.another_table;
INSERT INTO test.out_table
SELECT id, value
FROM temp_table;
$$ LANGUAGE SQL;
And I get this :
ERROR: relation "temp_table " does not exist
LINE 11: FROM temp_table
(Obviously, I'm aware the temp_table is not necessary for what I'm doing in the code above, but that's not the point :) => I want to understand the syntax to get it to work)
The appropriate syntax for creating a temp table is
create temp table...
but you have to be sure to drop the temp table before existing out of the function. Also, I'd suggest this syntax instead:
CREATE TEMP TABLE IF NOT EXISTS temp_table AS
SELECT id, value
FROM test.another_table;
Thus your function will be like this:
CREATE FUNCTION test.myfunction()
RETURNS SETOF test.out_table
AS $$
CREATE TEMP TABLE IF NOT EXISTS temp_table AS
SELECT id, value
FROM test.another_table;
INSERT INTO test.out_table
SELECT id, value
FROM temp_table;
DROP TABLE temp_table;
$$ LANGUAGE SQL;
But if I can be so kind, I'd like to rewrite this function so it is more correct:
CREATE FUNCTION test.myfunction()
RETURNS TABLE (id int, value varchar) -- change your datatype as needed
AS $$
BEGIN;
CREATE TEMP TABLE IF NOT EXISTS temp_table AS
SELECT id, value
FROM test.another_table;
INSERT INTO test.out_table
SELECT id, value
FROM temp_table;
DROP TABLE temp_table;
RETURN QUERY
SELECT id, value
from temp_table;
END;
$$ LANGUAGE plpgsql;
Untested; let me know if this fails.

Iterate on tables

I need to perform the same operation on several tables. Right now the query looks like this:
create view foobar
as
select this, that
from here
where this=1
union all
select this,that
from there
where this=1
..... .....
and so on, for several tables. All the results are union-ed.
Is there a way to, instead of writing this very long query where it's easy to get something wrong, to write something like
for table in here, there, upthere
do
select this, that from $table where this=1
and then take the union of them all.
I have the query working right now and it's going to take a while, but this is just curiosity and I didn't know how to search for this on Google!
Create a VIEW from UNION of all tables you need and then SELECT from this VIEW.
Additionally you can create a VIEW like this:
CREATE OR REPLACE VIEW table_set AS
SELECT 'table1' as table_name, field1, field2 ...
FROM table1
UNION ALL
SELECT 'table2' as table_name, field1, field2 ...
FROM table2
UNION ALL
SELECT 'table3' as table_name, field1, field2 ...
FROM table3
UNION ALL
...
and SELECT from this VIEW like:
SELECT field1, field3
FROM table_set
WHERE table_name IN ('table2','table4')
AND field5 = 'abc'
A view like #Igor desrcibes is one option.
If performance is of the essence and you don't have a lot of write operations, you could use that view to (re-)create a materialized view.
Or, for shorter syntax and dynamic selection of source tables, you could utilize a plpgsql function like this one:
CREATE OR REPLACE FUNCTION f_select_from_tbls(tables text[], cond int)
RETURNS TABLE (this int, that text) AS
$BODY$
DECLARE
tbl text; -- automatically avoids SQLi
BEGIN
FOREACH tbl IN ARRAY tables LOOP
RETURN QUERY EXECUTE format('
SELECT this, that
FROM %I
WHERE this = $1', tbl)
USING cond;
END LOOP;
END
$BODY$
LANGUAGE plpgsql;
Call:
SELECT * FROM f_select_from_tbls('{t1,t2,nonStandard tablename}', 4);
I assume a number of tables sharing columns of the same name and type (this int, that text) (You didn't define the data types in you question.) The function takes an array of text as tables names and an integer value as condition.
More about Looping Through Arrays in the manual.
More about Returning From a Function in the manual.
With dynamic SQL you have to be wary of SQL injection. That threat is neutralized here in two ways:
Table names in the array tables are injected with the function format() which properly quotes any non-standard table name
The value in parameter cond is passed in as is via USING, thereby making SQLi impossible.
Variant with VARIADIC parameter
The purpose is to make the function call simpler. You can hand in a list of text variables instead of building an array from them (array is built internally from the parameters). That's all.
CREATE OR REPLACE FUNCTION f_select_from_tbls2(cond int, VARIADIC tables text[])
RETURNS TABLE (this int, that text) AS
$BODY$
DECLARE
tbl text; -- automatically avoids SQLi
BEGIN
FOREACH tbl IN ARRAY tables LOOP
RETURN QUERY EXECUTE format('
SELECT this, that
FROM %I
WHERE this = $1', tbl)
USING cond;
END LOOP;
END
$BODY$
LANGUAGE plpgsql;
I placed tables last in this case to have an open end for the VARIADIC parameter.
Call:
SELECT * FROM f_select_from_tbls2(4, 't1','t2','iLLegal name')

SQL script to UNION a large number of tables

I have to do a union of a large number of disjoint daily tables from 2012-12-17 to 2012-10-30 in this example. The code for this gets ugly here is the snippet:
CREATE table map
with (appendonly=true, compresstype = quicklz)
AS
SELECT * FROM final_map_12_17
UNION ALL
SELECT * FROM final_map_12_16
UNION ALL
SELECT * FROM final_map_12_15
UNION ALL
SELECT * FROM final_map_12_14
UNION ALL
....
SELECT * FROM final_map_10_30;
Can I do this type of thing with a sequence or PL/PGSQL function instead of writing out each individual select by hand?
You can loop over date range in plpgsql function like this:
create or replace function add_map(date_from date, date_to date)
returns void language plpgsql as $$
declare
day date;
begin
for day in
select generate_series(date_from, date_to, '1 day')
loop
execute 'insert into map select * from final_map_'||
to_char(extract(month from day), '09')|| '_' ||
to_char(extract(day from day), '09');
end loop;
end; $$;
Calling the function:
-- create table map (....);
select add_map('2012-11-30', '2012-12-02');
is equivalent to:
insert into map select * from final_map_11_30;
insert into map select * from final_map_12_01;
insert into map select * from final_map_12_02;
There isn't a SQL function that would do this.
I would recommend that you put the list of tables in Excel. Then put in a formula such as:
="select * from "&a1&" union all"
Copy this formula down. Voila! You almost have the view defniition.
Copy the column with these statements into the SQL command tool. Add the create view at the top. Remove the union all at the end. And voila. You can easily create the view.
Have a think about redefining your list of tables as a partitioned table, with a single master table and multiple child tables. http://www.postgresql.org/docs/9.2/static/ddl-partitioning.html
Alternatively, maintain a view to union all the tables together, and when you add a new table to the schema add it to the view also.