My application uses multiple schemas to partition tenants across the database to improve performance. I am trying to create a plpgsql function that will give me an arbitrary result set based on the union of all application schemas given a table. Here is what I have so far (inspired by this blog post):
CREATE OR REPLACE FUNCTION app_union(tbl text) RETURNS SETOF RECORD AS $$
DECLARE
schema RECORD;
sql TEXT := '';
BEGIN
FOR schema IN EXECUTE 'SELECT distinct schema FROM tenants' LOOP
sql := sql || format('SELECT * FROM %I.%I %s UNION ALL ', schema.schema, tbl);
END LOOP;
RETURN QUERY EXECUTE left(sql, -11);
END
$$ LANGUAGE plpgsql;
This works great, but has to be called with a row type definition at the end:
select * from app_union('my_table') t(id uuid, name text, ...);
So, how can I call my function without providing a row type?
I know that I can introspect my tables using information_schema.columns, but I'm not sure how to dynamically generate the type declaration without a lot of case statements (columns doesn't report the definition sql the way that e.g., pg_indexes does).
Even if I could dynamically generate the row declaration, it seems I would have to append it to my former function call as dynamic sql anyway, which sort of chicken/eggs the problem of returning a result set of an arbitrary type from a function.
Instead of providing the table as a string, you could provide it as type anyelement to specify the actual type of the returning data, then infer the table's name using pg_typeof. You can also use string_agg rather than a loop to build your sql:
CREATE OR REPLACE FUNCTION app_union(tbl anyelement)
RETURNS setof anyelement AS $$
BEGIN
return query execute string_agg(
distinct format('select * from %I.%I', schema, pg_typeof(tbl)::text),
' union all '
) from tenants;
END
$$ LANGUAGE plpgsql;
select * from app_union(null::my_table);
Simplified example
Related
How can I write a stored procedure that contains a dynamically built SQL statement that returns a result set? Here is my sample code:
CREATE OR REPLACE FUNCTION reporting.report_get_countries_new (
starts_with varchar,
ends_with varchar
)
RETURNS TABLE (
country_id integer,
country_name varchar
) AS
$body$
DECLARE
starts_with ALIAS FOR $1;
ends_with ALIAS FOR $2;
sql VARCHAR;
BEGIN
sql = 'SELECT * FROM lookups.countries WHERE lookups.countries.country_name >= ' || starts_with ;
IF ends_with IS NOT NULL THEN
sql = sql || ' AND lookups.countries.country_name <= ' || ends_with ;
END IF;
RETURN QUERY EXECUTE sql;
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100 ROWS 1000;
This code returns an error:
ERROR: syntax error at or near "RETURN"
LINE 1: RETURN QUERY SELECT * FROM omnipay_lookups.countries WHERE o...
^
QUERY: RETURN QUERY SELECT * FROM omnipay_lookups.countries WHERE omnipay_lookups.countries.country_name >= r
CONTEXT: PL/pgSQL function "report_get_countries_new" line 14 at EXECUTE statement
I have tried other ways instead of this:
RETURN QUERY EXECUTE sql;
Way 1:
RETURN EXECUTE sql;
Way 2:
sql = 'RETURN QUERY SELECT * FROM....
/*later*/
EXECUTE sql;
In all cases without success.
Ultimately I want to write a stored procedure that contains a dynamic sql statement and that returns the result set from the dynamic sql statement.
There is room for improvements:
CREATE OR REPLACE FUNCTION report_get_countries_new (starts_with text
, ends_with text = NULL)
RETURNS SETOF lookups.countries AS
$func$
DECLARE
sql text := 'SELECT * FROM lookups.countries WHERE country_name >= $1';
BEGIN
IF ends_with IS NOT NULL THEN
sql := sql || ' AND country_name <= $2';
END IF;
RETURN QUERY EXECUTE sql
USING starts_with, ends_with;
END
$func$ LANGUAGE plpgsql;
-- the rest is default settings
Major points
PostgreSQL 8.4 introduced the USING clause for EXECUTE, which is useful for several reasons. Recap in the manual:
The command string can use parameter values, which are referenced in
the command as $1, $2, etc. These symbols refer to values supplied in
the USING clause. This method is often preferable to inserting data
values into the command string as text: it avoids run-time overhead of
converting the values to text and back, and it is much less prone to
SQL-injection attacks since there is no need for quoting or escaping.
IOW, it is safer and faster than building a query string with text representation of parameters, even when sanitized with quote_literal().
Note that $1, $2 in the query string refer to the supplied values in the USING clause, not to the function parameters.
While you return SELECT * FROM lookups.countries, you can simplify the RETURN declaration like demonstrated:
RETURNS SETOF lookups.countries
In PostgreSQL there is a composite type defined for every table automatically. Use it. The effect is that the function depends on the type and you get an error message if you try to alter the table. Drop & recreate the function in such a case.
This may or may not be desirable - generally it is! You want to be made aware of side effects if you alter tables. The way you have it, your function would break silently and raise an exception on it's next call.
If you provide an explicit default for the second parameter in the declaration like demonstrated, you can (but don't have to) simplify the call in case you don't want to set an upper bound with ends_with.
SELECT * FROM report_get_countries_new('Zaire');
instead of:
SELECT * FROM report_get_countries_new('Zaire', NULL);
Be aware of function overloading in this context.
Don't quote the language name 'plpgsql' even if that's tolerated (for now). It's an identifier.
You can assign a variable at declaration time. Saves an extra step.
Parameters are named in the header. Drop the nonsensical lines:
starts_with ALIAS FOR $1;
ends_with ALIAS FOR $2;
Use quote_literal() to avoid SQL injection (!!!) and fix your quoting problem:
CREATE OR REPLACE FUNCTION report_get_countries_new (
starts_with varchar,
ends_with varchar
)
RETURNS TABLE (
country_id integer,
country_name varchar
) AS
$body$
DECLARE
starts_with ALIAS FOR $1;
ends_with ALIAS FOR $2;
sql VARCHAR;
BEGIN
sql := 'SELECT * FROM lookups.countries WHERE lookups.countries.country_name ' || quote_literal(starts_with) ;
IF ends_with IS NOT NULL THEN
sql := sql || ' AND lookups.countries.country_name <= ' || quote_literal(ends_with) ;
END IF;
RETURN QUERY EXECUTE sql;
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100 ROWS 1000;
This is tested in version 9.1, works fine.
I have a plpgsql function which act as a wrapper around several other functions, all of which are set returning functions of different data types. Through this function, I am trying to get a generic set returning function which can return set of any data type. The data type of record is decided based on the input table name and column name.
Here is the code snippet:
create function cs_get(tablename text, colname text) returns setof record as $$
declare
type_name text;
ctype text;
loadfunc text;
result record;
begin
select data_type into type_name from information_schema.columns where table_name = tablename and column_name = colname;
if type_name is null then
raise exception 'Table % doesnot exist or does not have attribute % ',tablename, colname;
end if;
ctype := cs_get_ctype(type_name);
loadfunc := 'select * from cs_get_'||ctype||'('''||tablename||''','''||colname||''')';
for result in execute loadfunc loop
return next result;
end loop;
return;
end; $$ language plpgsql;
Suppose the column is of type integer (corresponding c type is int64), loadfunc would be
select * from cs_get_int64(tablename, colname)
cs_get_int64 is a set-returning function defined in a C library which returns values of type int64.
However, this gives an error saying
ERROR: a column definition list is required for functions returning "record" at character 15
The easiest way to achieve this is to return a setof text for every data type. But that of course is not a clean way to do this. I have tried replacing loadFunc with
select cs_get_int64::integer from cs_get_int64(tablename, colname)
which is required to use records. But, this did not help.
My question now is:
Is it possible to create such a generic set returning function? If yes, how?
The answer is yes. But it's not trivial.
As long as you return anonymous records (returns setof record) a column definition list is required for the returned record to work with it in SQL. Pretty much what the error message says.
There is a (limited) way around this with polymorphic types:
CREATE OR REPLACE FUNCTION cs_get(_tbl_type anyelement, _colname text)
RETURNS SETOF anyelement AS
...
Detailed explanation int this related answer (climax in the last chapter!):
Refactor a PL/pgSQL function to return the output of various SELECT queries
I want to return a row from a table with this function (I don't know the name of the table, it's random)
CREATE FUNCTION foo( text ) RETURNS setof record AS $$
DECLARE
table_name ALIAS FOR $1;
BEGIN
SELECT * from table_name ;
END
$$ LANGUAGE plpgsql;
and then I want to do this:
select col1,col2 from foo('bar');
Any ideas?
SQL demands to know the return type at call time. And functions require you to define a return type as well. What you are after is not trivial.
If you don't know the return type at call time, you are basically out of luck. You cannot solve the problem with a single function call.
If you know the type at call time, there is an option with polymorphic types and dynamic SQL with EXECUTE:
CREATE OR REPLACE FUNCTION f_data_of_table(_tbl_type anyelement)
RETURNS SETOF anyelement AS
$func$
BEGIN
RETURN QUERY EXECUTE
'SELECT * FROM ' || pg_typeof(_tbl_type);
END
$func$ LANGUAGE plpgsql;
Call:
SELECT * FROM f_data_of_table(NULL::my_table_name);
Details in this related answer:
Refactor a PL/pgSQL function to return the output of various SELECT queries
Be wary of SQL injection:
Table name as a PostgreSQL function parameter
Only makes sense if you do more than just SELECT * FROM tbl, or you'd simply use the SQL command.
Aside:
Do not use ALIAS to attach names to parameter values. That's outdated and discouraged. Use named parameters instead.
I have a function that looks like this:
CREATE OR REPLACE FUNCTION mffcu.test_ty_hey()
RETURNS setof record
LANGUAGE plpgsql
AS $function$
Declare
cname1 text;
sql2 text;
Begin
for cname1 in
select array_to_string(useme, ', ') from (
select array_agg(column_name) as useme
from(
select column_name::text
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'crosstab_183'
and ordinal_position != 1
) as fin
) as fine
loop
sql2 := 'select distinct array['|| cname1 ||'] from mffcu.crosstab_183';
execute sql2;
end loop;
END;
$function$
I call the function with this:
select mffcu.test_ty_hey()
How do I return the results of the sql2 query without creating a table/temporary table?
While #Pavel is right, of course, your very convoluted function could be untangled to:
CREATE OR REPLACE FUNCTION mffcu.test_ty_hey()
RETURNS SETOF text[] LANGUAGE plpgsql
AS $func$
DECLARE
cname1 text;
BEGIN
FOR cname1 IN
SELECT column_name::text
FROM information_schema.columns
WHERE table_name = 'crosstab_183'
AND table_schema = 'mffcu'
AND ordinal_position <> 1
LOOP
RETURN QUERY
EXECUTE format('SELECT DISTINCT ARRAY[%I::text]
FROM mffcu.crosstab_183', cname1);
END LOOP;
END
$func$
format() requires PostgreSQL 9.1 or later. In 9.0 you can substitute with:
EXECUTE 'SELECT DISTINCT ARRAY['|| quote_ident(cname1) ||'::text]
FROM mffcu.crosstab_183';
Call:
select * FROM mffcu.test_ty_hey();
By casting each column to text we arrive at a consistent data type that can be used to declare the RETURN type. This compromise has to be made to return various data types from one function. Every data type can be cast to text, so that's the obvious common ground.
BTW, I have trouble imagining what the ARRAY wrapper around every single value should be good for. I suppose you could just drop that.
PostgreSQL functions should to have fixed result type before execution. You cannot specify type late in execution. There is only two workarounds - using temp tables or using cursors.
PLpgSQL language is not good for too generic routines - it good for implementation strict and clean business rules. And bad for generic crosstab calculations or generic auditing or similar generic task. It works, but code is slower and usually not well maintainable.
but reply for your query, you can use a output cursors
example http://okbob.blogspot.cz/2008/08/using-cursors-for-generating-cross.html
I am using PostgreSQL 8.4 and I want to create a function that returns a query with many rows.
The following function does not work:
create function get_names(varchar) returns setof record AS $$
declare
tname alias for $1;
res setof record;
begin
select * into res from mytable where name = tname;
return res;
end;
$$ LANGUAGE plpgsql;
The type record only allows single row.
How to return an entire query? I want to use functions as query templates.
CREATE OR REPLACE FUNCTION get_names(_tname varchar)
RETURNS TABLE (col_a integer, col_b text) AS
$func$
BEGIN
RETURN QUERY
SELECT t.col_a, t.col_b -- must match RETURNS TABLE
FROM mytable t
WHERE t.name = _tname;
END
$func$ LANGUAGE plpgsql;
Call like this:
SELECT * FROM get_names('name')
Major points:
Use RETURNS TABLE, so you don't have to provide a list of column names with every call.
Use RETURN QUERY, much simpler.
Table-qualify column names to avoid naming conflicts with identically named OUT parameters (including columns declared with RETURNS TABLE).
Use a named variable instead of ALIAS. Simpler, doing the same, and it's the preferred way.
A simple function like this could also be written in LANGUAGE sql:
CREATE OR REPLACE FUNCTION get_names(_tname varchar)
RETURNS TABLE (col_a integer, col_b text) AS
$func$
SELECT t.col_a, t.col_b --, more columns - must match RETURNS above
FROM mytable t
WHERE t.name = $1;
$func$ LANGUAGE sql;