How to iterate over a record when the columns are dynamic - sql

I have this function in postgres which takes PVH_COLS_DYNA that contains the columns that are going in to the query:
CREATE OR REPLACE FUNCTION DRYNAMIC_DATA_F(PVH_COLS_DYNA VARCHAR) RETURNS numeric AS $$
DECLARE
VV_QUERY_DINAMIC VARCHAR;
VV_ROW_RECORD record;
BEGIN
VV_QUERY_DINAMIC:=' SELECT '|| PVH_COLS_DYNA ||' FROM as_detalle_carga WHERE fk_id_carga_cartera = 1234 ;';
FOR VV_ROW_RECORD IN EXECUTE VV_QUERY_DINAMIC LOOP
raise notice ' data % ', VV_ROW_RECORD.???????;
END LOOP;
return 1;
END;
$$ LANGUAGE plpgsql;
How can I get the data from the record variable VV_ROW_RECORD, since the columns are dynamic?
VV_ROW_RECORD.1
VV_ROW_RECORD.?1
VV_ROW_RECORD.[1]
VV_ROW_RECORD.?????

You cannot reference columns like array items, columns have to be referenced by name.
The dynamic part is not getting the row in your example, but referencing each column.
CREATE OR REPLACE FUNCTION dynamic_data_f(pvh_cols_dyna text)
RETURNS numeric AS
$func$
DECLARE
_row as_detalle_carga%ROWTYPE;
_col text;
_data text;
BEGIN
SELECT *
INTO _row
FROM as_detalle_carga
WHERE fk_id_carga_cartera = 1234;
FOREACH _col IN ARRAY string_to_array(pvh_cols_dyna, ',')
LOOP
EXECUTE format('SELECT ($1).%I::text', trim(_col))
USING _row
INTO _data;
RAISE NOTICE 'data: % ', _data;
END LOOP;
RETURN 1;
END
$func$ LANGUAGE plpgsql;
%I is an argument to format(), properly escaping identifiers as needed.
$1 in the query string for EXECUTE is a parameter filled in by the USING clause (not to be confused with function parameters!).
Related answers (with more explanation):
Postgres pl/pgsql ERROR: column "column_name" does not exist
Iterating over integer[] in PL/pgSQL
How to use EXECUTE FORMAT ... USING in postgres function

You cannot iterate record columns directly. You have to convert it first into something iterable, like json or hstore.
FOR vv_row_record IN EXECUTE vv_query_dynamic LOOP
FOR vv_row_record_pairs IN SELECT * FROM json_each(row_to_json(vv_row_record)) LOOP
RAISE NOTICE ' field "%" in json is % ',
vv_row_record_pairs.key,
vv_row_record_pairs.value;
END LOOP;
-- OR
FOR vv_row_record_pairs IN SELECT * FROM each(hstore(vv_row_record)) LOOP
RAISE NOTICE ' field "%" in text representation is % ',
vv_row_record_pairs.key,
vv_row_record_pairs.value;
END LOOP;
END LOOP;

Related

Using local variable in psql cli or declare and append in procedure (Postgresql) [duplicate]

Here's what I would like to do:
\set values foo,bar,baz
DO $$
DECLARE
value TEXT;
values TEXT[] := string_to_array(:'values', ',');
BEGIN
FOREACH value IN ARRAY values LOOP
raise notice 'v: %', value;
END LOOP;
END $$ LANGUAGE plpgsql;
Which results in the following error:
ERROR: syntax error at or near ":"
SELECT string_to_array(:'values', ',') INTO values...
^
Here's the solution I have currently, but it feels hacky:
\set values foo,bar,baz
PREPARE get_values AS SELECT string_to_array(:'values', ',');
DO $$
DECLARE
value TEXT;
values TEXT[];
BEGIN
EXECUTE 'EXECUTE get_values' INTO values;
FOREACH value IN ARRAY values LOOP
raise notice 'v: %', value;
END LOOP;
END $$ LANGUAGE plpgsql;
Answer
DO expects a string literal with plpgsql code. Symbols are not substituted inside strings in psql.
You could concatenate the whole string into a psql variable and then execute it.
How to concatenate psql variables?
Pretty multi-line format is not possible, because (per documentation):
But in any case, the arguments of a meta-command cannot continue
beyond the end of the line.
Simple example:
test=# \set value foo
test=# \set do 'BEGIN\n RAISE NOTICE ''v: %'', ' :'value' ';\nEND'
test=# DO :'do';
NOTICE: v: foo
Replace line breaks with \n (or remove them if you don't care for pretty format). Based on this adapted code:
DO
'
DECLARE
_val text;
_vals text[] := string_to_array(>>values<<, '','');
BEGIN
FOREACH _val IN ARRAY _vals
LOOP
RAISE NOTICE ''v: %'', _val;
END LOOP;
END
'
It looks like this:
test=# \set do 'DECLARE\n _val text;\n _vals text[] := string_to_array(' :'values' ', '','');\nBEGIN\n FOREACH _val IN ARRAY _vals\n LOOP\n RAISE NOTICE ''v: %'', _val;\n END LOOP;\nEND'
test=# DO :'do';
NOTICE: v: foo
NOTICE: v: bar
NOTICE: v: baz
DO
I added bold emphasis to the variable to make it easier to spot.
Related answer by #Pavel (ab)using a server session variable:
Referring to session variables (\set var='value') from PL/PGSQL
Alternative solutions
Prepared statement
Your current solution doesn't look that bad. I would simplify:
PREPARE get_values AS SELECT * FROM regexp_split_to_table(:'values', ',');
DO
$do$
DECLARE
_val text;
BEGIN
FOR _val IN EXECUTE
'EXECUTE get_values'
LOOP
RAISE NOTICE 'v: %', _val;
END LOOP;
END
$do$;
Temporary table
Similar solution with a temporary table:
CREATE TEMP TABLE tmp AS SELECT * FROM regexp_split_to_table(:'values', ',') v;
DO
$do$
DECLARE
_val text;
BEGIN
FOR _val IN
TABLE tmp
LOOP
RAISE NOTICE 'v: %', _val;
END LOOP;
END
$do$;
Was able to take advantage of this solution:
Passing argument to a psql procedural script
Where I set the variable as such and retrieve it with current_setting()
\set values foo,bar,baz
SET vars.values TO :'values';
DO $$
DECLARE
value TEXT;
values TEXT[] := string_to_array(current_setting('vars.values'), ',');
BEGIN
FOREACH value IN ARRAY values LOOP
RAISE NOTICE 'v: %', value;
END LOOP;
END $$ LANGUAGE plpgsql

How to insert rows to table in a loop

I have the following plpgsql function in PostgreSQL:
CREATE OR REPLACE FUNCTION func1()
RETURNS SETOF type_a AS
$BODY$
declare
param text;
sqls varchar;
row type_a;
begin
code.....
sqls='select * from func3(' || param || ') ';
for row in execute sqls LOOP
return next row;
END LOOP;
end if;
return;
end
$BODY$
LANGUAGE plpgsql VOLATILE
I want to add an insert statment into the loop, so that the loop will work as it is now but also all rows will be saved in a table.
for row in execute sqls LOOP
INSERT INTO TABLE new_tab(id, name)
return next row;
the thing is that I don't know how to do that... the insert statment normaly has syntax of:
INSERT INTO new_tab(id, name)
SELECT x.id, x.name
FROM y
but this syntax doesn't fit here. There is no query to select rows from.... the rows are in the loop.
Basic insert with values looks like this:
INSERT INTO table_name (column1,column2,column3,...)
VALUES (value1,value2,value3,...);
Based on the additional comments you need to use cursor instead of execute sqls.
No need for a loop, you can use insert .. select ... returning in dynamic SQL just as well:
create or replace function func1()
returns table (id integer, name text)
as
$$
declare
param text;
begin
param := ... ;
return query execute
'insert into new_tab (id, name)
select id, name
from func3($1)
returning *'
using param;
end;
$$
language plpgsql;
Note that I used a parameter placeholder and the USING clause instead of concatenating the parameter into the query - much more robust.

How can I refer to a variable in postgresql dynamic SQL?

I'm trying to write a PostgreSQL function for table upserts that can be used for any table. My starting point is taken from a concrete function for a specific table type:
CREATE TABLE doodad(id BIGINT PRIMARY KEY, data JSON);
CREATE OR REPLACE FUNCTION upsert_doodad(d doodad) RETURNS VOID AS
$BODY$
BEGIN
LOOP
UPDATE doodad
SET id = (d).id, data = (d).data
WHERE id = (d).id;
IF found THEN
RETURN;
END IF;
-- does not exist, or was just deleted.
BEGIN
INSERT INTO doodad SELECT d.*;
RETURN;
EXCEPTION when UNIQUE_VIOLATION THEN
-- do nothing, and loop to try the update again
END;
END LOOP;
END;
$BODY$
LANGUAGE plpgsql;
The dynamic SQL version of this for any table that I've come up with is here:
SQL Fiddle
CREATE OR REPLACE FUNCTION upsert(target ANYELEMENT) RETURNS VOID AS
$$
DECLARE
attr_name NAME;
col TEXT;
selectors TEXT[];
setters TEXT[];
update_stmt TEXT;
insert_stmt TEXT;
BEGIN
FOR attr_name IN SELECT a.attname
FROM pg_index i
JOIN pg_attribute a ON a.attrelid = i.indrelid
AND a.attnum = ANY(i.indkey)
WHERE i.indrelid = format_type(pg_typeof(target), NULL)::regclass
AND i.indisprimary
LOOP
selectors := array_append(selectors, format('%1$s = target.%1$s', attr_name));
END LOOP;
FOR col IN SELECT json_object_keys(row_to_json(target))
LOOP
setters := array_append(setters, format('%1$s = (target).%1$s', col));
END LOOP;
update_stmt := format(
'UPDATE %s SET %s WHERE %s',
pg_typeof(target),
array_to_string(setters, ', '),
array_to_string(selectors, ' AND ')
);
insert_stmt := format('INSERT INTO %s SELECT (target).*', pg_typeof(target));
LOOP
EXECUTE update_stmt;
IF found THEN
RETURN;
END IF;
BEGIN
EXECUTE insert_stmt;
RETURN;
EXCEPTION when UNIQUE_VIOLATION THEN
-- do nothing
END;
END LOOP;
END;
$$
LANGUAGE plpgsql;
When I attempt to use this function, I get an error:
SELECT * FROM upsert(ROW(1,'{}')::doodad);
ERROR: column "target" does not exist: SELECT * FROM upsert(ROW(1,'{}')::doodad)
I tried changing the upsert statement to use placeholders, but I can't figure out how to invoke it using the record:
EXECUTE update_stmt USING target;
ERROR: there is no parameter $2: SELECT * FROM upsert(ROW(1,'{}')::doodad)
EXECUTE update_stmt USING target.*;
ERROR: query "SELECT target.*" returned 2 columns: SELECT * FROM upsert(ROW(1,'{}')::doodad)
I feel really close to a solution, but I can't figure out the syntax issues.
Short answer: you can't.
Variable substitution does not happen in the command string given to EXECUTE or one of its variants. If you need to insert a varying value into such a command, do so as part of constructing the string value, or use USING, as illustrated in Section 40.5.4. 1
Longer answer:
SQL statements and expressions within a PL/pgSQL function can refer to variables and parameters of the function. Behind the scenes, PL/pgSQL substitutes query parameters for such references. 2
This was the first important piece to the puzzle: PL/pgSQL does magic transformations on function parameters that turn them into variable substitutions.
The second was that fields of variable substitutions can referenced:
Parameters to a function can be composite types (complete table rows). In that case, the corresponding identifier $n will be a row variable, and fields can be selected from it, for example $1.user_id. 3
This excerpt confused me, because it referred to function parameters, but knowing that function parameters are implemented as variable substitutions under the hood, it seemed that I should be able to use the same syntax in EXECUTE.
These two facts unlocked the solution: use the ROW variable in the USING clause, and dereference its fields in the dynamic SQL. The results (SQL Fiddle):
CREATE OR REPLACE FUNCTION upsert(v_target ANYELEMENT)
RETURNS SETOF ANYELEMENT AS
$$
DECLARE
v_target_name TEXT;
v_attr_name NAME;
v_selectors TEXT[];
v_colname TEXT;
v_setters TEXT[];
v_update_stmt TEXT;
v_insert_stmt TEXT;
v_temp RECORD;
BEGIN
v_target_name := format_type(pg_typeof(v_target), NULL);
FOR v_attr_name IN SELECT a.attname
FROM pg_index i
JOIN pg_attribute a ON a.attrelid = i.indrelid
AND a.attnum = ANY(i.indkey)
WHERE i.indrelid = v_target_name::regclass
AND i.indisprimary
LOOP
v_selectors := array_append(v_selectors, format('t.%1$I = $1.%1$I', v_attr_name));
END LOOP;
FOR v_colname IN SELECT json_object_keys(row_to_json(v_target))
LOOP
v_setters := array_append(v_setters, format('%1$I = $1.%1$I', v_colname));
END LOOP;
v_update_stmt := format(
'UPDATE %I t SET %s WHERE %s RETURNING t.*',
v_target_name,
array_to_string(v_setters, ','),
array_to_string(v_selectors, ' AND ')
);
v_insert_stmt = format('INSERT INTO %I SELECT $1.*', v_target_name);
LOOP
EXECUTE v_update_stmt INTO v_temp USING v_target;
IF v_temp IS NOT NULL THEN
EXIT;
END IF;
BEGIN
EXECUTE v_insert_stmt USING v_target;
EXIT;
EXCEPTION when UNIQUE_VIOLATION THEN
-- do nothing
END;
END LOOP;
RETURN QUERY SELECT v_target.*;
END;
$$
LANGUAGE plpgsql;
For writeable CTE fans, this is trivially convertible to CTE form:
v_cte_stmt = format(
'WITH up as (%s) %s WHERE NOT EXISTS (SELECT 1 from up t WHERE %s)',
v_update_stmt,
v_insert_stmt,
array_to_string(v_selectors, ' AND '));
LOOP
BEGIN
EXECUTE v_cte_stmt USING v_target;
EXIT;
EXCEPTION when UNIQUE_VIOLATION THEN
-- do nothing
END;
END LOOP;
RETURN QUERY SELECT v_target.*;
NB: I have done zero performance testing on this solution, and I am relying on the analysis of others for its correctness. For now it appears to run correctly on PostgreSQL 9.3 in my development environment. YMMV.

Count rows affected by DELETE

I use this code to verify the DELETE sentence, but I am sure you know a better way:
CREATE OR REPLACE FUNCTION my_schema.sp_delete_row_table(table_name character varying
, id_column character varying
, id_value integer)
RETURNS integer AS
$BODY$
DECLARE
BEFORE_ROWS integer;
AFTER_ROWS integer;
BEGIN
EXECUTE 'SELECT count(*) FROM ' || TABLE_NAME INTO BEFORE_ROWS;
EXECUTE 'DELETE FROM ' || TABLE_NAME || ' WHERE ' || ID_COLUMN || ' = ' || (ID_VALUE)::varchar;
EXECUTE 'SELECT count(*) FROM ' || TABLE_NAME INTO AFTER_ROWS;
IF BEFORE_ROWS - AFTER_ROWS = 1 THEN
RETURN 1;
ELSE
RETURN 2;
END IF;
EXCEPTION WHEN OTHERS THEN
RETURN 0;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
How to improve this code? I need it to work in Postgres 8.4, 9.1 and 9.2.
Actually, you cannot use FOUND with EXECUTE. The manual:
Note in particular that EXECUTE changes the output of GET DIAGNOSTICS,
but does not change FOUND.
There are a couple of other things that might be improved. First of all, your original is open to SQL injection. I suggest:
CREATE OR REPLACE FUNCTION my_schema.sp_delete_row_table(table_name regclass
, id_column text
, id_value int
, OUT del_ct int) AS
$func$
BEGIN
EXECUTE format ('DELETE FROM %s WHERE %I = $1', table_name, id_column);
USING id_value; -- assuming integer columns
GET DIAGNOSTICS del_ct = ROW_COUNT; -- directly assign OUT parameter
EXCEPTION WHEN OTHERS THEN
del_ct := 0;
END
$func$ LANGUAGE plpgsql;
format() requires Postgres 9.1 or later. You can replace it with string concatenation, but be sure to use escape the column name properly with quote_ident()!
The rest works for 8.4 as well.
Closely related answers:
Dynamic SQL (EXECUTE) as condition for IF statement
Table name as a PostgreSQL function parameter
Look into the variables called found and row_count:
http://www.postgresql.org/docs/current/static/plpgsql-statements.html#PLPGSQL-STATEMENTS-DIAGNOSTICS
found is true if any rows were affected. row_count gives you the number of affected rows.
IF FOUND THEN
GET DIAGNOSTICS integer_var = ROW_COUNT;
END IF;

plpgsql function issue

I have the following plpgsql procedure;
DECLARE
_r record;
point varchar[] := '{}';
i int := 0;
BEGIN
FOR _r IN EXECUTE ' SELECT a.'|| quote_ident(column) || ' AS point,
FROM ' || quote_ident (table) ||' AS a'
LOOP
point[i] = _r;
i = i+1;
END LOOP;
RETURN 'OK';
END;
Which its main objective is to traverse a table and store each value of the row in an array. I am still new to plpgsql. Can anyone point out is the error as it is giving me the following error;
This is the complete syntax (note that I renamed the parameter column to col_name as column is reserved word. The same goes for table)
create or replace function foo(col_name text, table_name text)
returns text
as
$body$
DECLARE
_r record;
point character varying[] := '{}';
i int := 0;
BEGIN
FOR _r IN EXECUTE 'SELECT a.'|| quote_ident(col_name) || ' AS pt, FROM ' || quote_ident (table_name) ||' AS a'
loop
point[i] = _r;
i = i+1;
END LOOP;
RETURN 'OK';
END;
$body$
language plpgsql;
Although to be honest: I fail so see what you are trying to achieve here.
#a_horse fixes most of the crippling problems with your failed attempt.
However, nobody should use this. The following step-by-step instructions should lead to a sane implementation with modern PostgreSQL.
Phase 1: Remove errors and mischief
Remove the comma after the SELECT list to fix the syntax error.
You start your array with 0, while the default is to start with 1. Only do this if you need to do it. Leads to unexpected results if you operate with array_upper() et al. Start with 1 instead.
Change RETURN type to varchar[] to return the assembled array and make this demo useful.
What we have so far:
CREATE OR REPLACE FUNCTION foo(tbl varchar, col varchar)
RETURNS varchar[] LANGUAGE plpgsql AS
$BODY$
DECLARE
_r record;
points varchar[] := '{}';
i int := 0;
BEGIN
FOR _r IN
EXECUTE 'SELECT a.'|| quote_ident(col) || ' AS pt
FROM ' || quote_ident (tbl) ||' AS a'
LOOP
i = i + 1; -- reversed order to make array start with 1
points[i] = _r;
END LOOP;
RETURN points;
END;
$BODY$;
Phase 2: Remove cruft, make it useful
Use text instead of character varying / varchar for simplicity. Either works, though.
You are selecting a single column, but use a variable of type record. This way a whole record is being coerced to text, which includes surrounding parenthesis. Hardly makes any sense. Use a text variable instead. Works for any column if you explicitly cast to text (::text). Any type can be cast to text.
There is no point in initializing the variable point. It can start as NULL here.
Table and column aliases inside EXECUTE are of no use in this case. Dynamically executed SQL has its own scope!.
No semicolon (;) needed after final END in a plpgsql function.
It's simpler to just append each value to the array with || .
Almost sane:
CREATE OR REPLACE FUNCTION foo1(tbl text, col text)
RETURNS text[] LANGUAGE plpgsql AS
$func$
DECLARE
point text;
points text[];
BEGIN
FOR point IN
EXECUTE 'SELECT '|| quote_ident(col) || '::text FROM ' || quote_ident(tbl)
LOOP
points = points || point;
END LOOP;
RETURN points;
END
$func$;
Phase 3: Make it shine in modern PL/pgSQL
If you pass a table name as text, you create an ambiguous situation. You can prevent SQLi just fine with format() or quote_ident(), but this will fail with tables outside your search_path.
Then you need to add schema-qualification, which creates an ambiguous value. 'x.y' could stand for the table name "x.y" or the schema-qualified table name "x"."y". You can't pass "x"."y" since that will be escaped into """x"".""y""". You'd need to either use an additional parameter for the schema name or one parameter of type regclass regclass is automatically quoted as need when coerced to text and is the elegant solution here.
The new format() is simpler than multiple (or even a single) quote_ident() call.
You did not specify any order. SELECT returns rows in arbitrary order without ORDER BY. This may seem stable, since the result is generally reproducible as long as the underlying table doesn't change. But that's 100% unreliable. You probably want to add some kind of ORDER BY.
Finally, you don't need to loop at all. Use a plain SELECT with an Array constructor.
Use an OUT parameter to further simplify the code
Proper solution:
CREATE OR REPLACE FUNCTION f_arr(tbl regclass, col text, OUT arr text[])
LANGUAGE plpgsql AS
$func$
BEGIN
EXECUTE format('SELECT ARRAY(SELECT %I::text FROM %s ORDER BY 1)', col, tbl)
INTO arr;
END
$func$;
Call:
SELECT f_arr('myschema.mytbl', 'mycol');