Check if table is empty in runtime - sql

I am trying to write a script which drops some obsolete tables in Postgres database. I want to be sure the tables are empty before dropping them. I also want the script could be kept in our migration scripts where it is safe to run even after these tables are actually dropped.
There is my script:
CREATE OR REPLACE FUNCTION __execute(TEXT) RETURNS VOID AS $$
BEGIN EXECUTE $1; END;
$$ LANGUAGE plpgsql STRICT;
CREATE OR REPLACE FUNCTION __table_exists(TEXT, TEXT) RETURNS bool as $$
SELECT exists(SELECT 1 FROM information_schema.tables WHERE (table_schema, table_name, table_type) = ($1, $2, 'BASE TABLE'));
$$ language sql STRICT;
CREATE OR REPLACE FUNCTION __table_is_empty(TEXT) RETURNS bool as $$
SELECT not exists(SELECT 1 FROM $1 );
$$ language sql STRICT;
-- Start migration here
SELECT __execute($$
DROP TABLE oldtable1;
$$)
WHERE __table_exists('public', 'oldtable1')
AND __table_is_empty('oldtable1');
-- drop auxilary functions here
And finally I got:
ERROR: syntax error at or near "$1"
LINE 11: SELECT not exists(SELECT 1 FROM $1 );
Is there any other way?

You must use EXECUTE if you want to pass a table name as parameter in a Postgres function.
Example:
CREATE OR REPLACE FUNCTION __table_is_empty(param character varying)
RETURNS bool
AS $$
DECLARE
v int;
BEGIN
EXECUTE 'select 1 WHERE EXISTS( SELECT 1 FROM ' || quote_ident(param) || ' ) '
INTO v;
IF v THEN return false; ELSE return true; END IF;
END;
$$ LANGUAGE plpgsql;
/
Demo: http://sqlfiddle.com/#!12/09cb0/1

No, no, no. For many reasons.
#kordirko already pointed out the immediate cause for the error message: In plain SQL, variables can only be used for values not for key words or identifiers. You can fix that with dynamic SQL, but that still doesn't make your code right.
You are applying programming paradigms from other programming languages. With PL/pgSQL, it is extremely inefficient to split your code into multiple separate tiny sub-functions. The overhead is huge in comparison.
Your actual call is also a time bomb. Expressions in the WHERE clause are executed in any order, so this may or may not raise an exception for non-existing table names:
WHERE __table_exists('public', 'oldtable1')
AND __table_is_empty('oldtable1');
... which will roll back your whole transaction.
Finally, you are completely open to race conditions. Like #Frank already commented, a table can be in use by concurrent transactions, in which case open locks may stall your attempt to drop the table. Could also lead to deadlocks (which the system resolves by rolling back all but one competing transactions). Take out an exclusive lock yourself, before you check whether the table is (still) empty.
Proper function
This is safe for concurrent use. It takes an array of table names (and optionally a schema name) and only drops existing, empty tables that are not locked in any way:
CREATE OR REPLACE FUNCTION f_drop_tables(_tbls text[] = '{}'
, _schema text = 'public'
, OUT drop_ct int) AS
$func$
DECLARE
_tbl text; -- loop var
_empty bool; -- for empty check
BEGIN
drop_ct := 0; -- init!
FOR _tbl IN
SELECT quote_ident(table_schema) || '.'
|| quote_ident(table_name) -- qualified & escaped table name
FROM information_schema.tables
WHERE table_schema = _schema
AND table_type = 'BASE TABLE'
AND table_name = ANY(_tbls)
LOOP
EXECUTE 'SELECT NOT EXISTS (SELECT 1 FROM ' || _tbl || ')'
INTO _empty; -- check first, only lock if empty
IF _empty THEN
EXECUTE 'LOCK TABLE ' || _tbl; -- now table is ripe for the plucking
EXECUTE 'SELECT NOT EXISTS (SELECT 1 FROM ' || _tbl || ')'
INTO _empty; -- recheck after lock
IF _empty THEN
EXECUTE 'DROP TABLE ' || _tbl; -- go in for the kill
drop_ct := drop_ct + 1; -- count tables actually dropped
END IF;
END IF;
END LOOP;
END
$func$ LANGUAGE plpgsql STRICT;
Call:
SELECT f_drop_tables('{foo1,foo2,foo3,foo4}');
To call with a different schema than the default 'public':
SELECT f_drop_tables('{foo1,foo2,foo3,foo4}', 'my_schema');
Major points
Reports the number of tables actually dropped. (Adapt to report info of your choice.)
Using the information schema like in your original. Seems the right choice here, but be aware of subtle limitations:
How to check if a table exists in a given schema
For use under heavy concurrent load (with long transactions), consider the NOWAIT option for the LOCK command and possibly catch exceptions from it.
Per documentation on "Table-level Locks":
ACCESS EXCLUSIVE
Conflicts with locks of all modes (ACCESS SHARE, ROW SHARE, ROW EXCLUSIVE,
SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE,
and ACCESS EXCLUSIVE). This mode guarantees that the holder
is the only transaction accessing the table in any way.
Acquired by the ALTER TABLE, DROP TABLE, TRUNCATE, REINDEX, CLUSTER, and VACUUM FULL commands. This is also the
default lock mode for LOCK TABLE statements that do not specify a mode explicitly.
Bold emphasis mine.

I work on DB2 so I can't say for sure if this will work on a Postgres database, but try this:
select case when count(*) > 0 then True else False end from $1
...in place of:
SELECT not exists(SELECT 1 FROM $1 )
If Postgres does not have a CASE / END expression capability, I'd be shocked if it didn't have some kind of similar IF/THEN/ELSE like expression ability to use as a substitute.

Related

PostgreSQL Function returning result set from dynamic tables names

In my database, I have the standard app tables and backup tables. Eg. for a table "employee", I have a table called "bak_employee". The bak_employee table is a backup of the employee table. I use it to restore the employee table between tests.
I'd figure I can use these "bak_" tables to see the changes that have occurred during the test like this:
SELECT * FROM employee EXCEPT SELECT * FROM bak_employee
This will show me the inserted and updated records. I'll ignore the deleted records for now.
Now, what I would like to do is go through all the tables in my database to see if there's any changes in any of the tables. I was thinking of doing this as a function so it's easy to call over and over. This is what I have so far:
CREATE OR REPLACE FUNCTION public.show_diff()
RETURNS SETOF diff_tables AS
$BODY$
DECLARE
app_tables text;
BEGIN
FOR app_tables IN
SELECT table_name
FROM information_schema.tables
WHERE table_catalog = 'myDatabase'
AND table_schema = 'public'
AND table_name not like 'bak_%' -- exclude existing backup tables
LOOP
-- somehow loop through tables to see what's changed something like:
EXECUTE 'SELECT * FROM ' || app_tables || ' EXCEPT SELECT * FROM bak_' || app_tables;
END LOOP;
RETURN;
END;
$BODY$
LANGUAGE plpgsql;
But obviously this isn't going to return me any useful information. Any help would be appreciated.
You cannot return various well-known row types from the same function in the same call. A cheap fix is to cast each row type to text, so we have a common return type.
CREATE OR REPLACE FUNCTION public.show_diff()
RETURNS SETOF text AS -- text!!
$func$
DECLARE
app_table text;
BEGIN
FOR app_table IN
SELECT table_name
FROM information_schema.tables
WHERE table_catalog = 'myDatabase'
AND table_schema = 'public'
AND table_name NOT LIKE 'bak_%' -- exclude existing backup tables
LOOP
RETURN NEXT ' ';
RETURN NEXT '=== ' || app_table || ' ===';
RETURN QUERY EXECUTE format(
'SELECT x::text FROM (TABLE %I EXCEPT ALL TABLE %I) x'
, app_table, 'bak_' || app_table);
END LOOP;
RETURN;
END
$func$ LANGUAGE plpgsql;
Call:
SELECT * FROM public.show_diff();
I had the test suggested by #a_horse at first, but after your comment I realized that there is no need for this. EXCEPT considers NULL values to be equal and shows all differences.
While being at it, I improved and simplified your solution some more. Use EXCEPT ALL: cheaper and does not run the risk of folding complete duplicates.
Using EXCEPT clause in PostgreSQL
TABLE is just syntactical sugar.
Is there a shortcut for SELECT * FROM in psql?
However, if you have an index on a unique (combination of) column(s), a JOIN like I suggested before should be faster: finding the only possible duplicate via index should be substantially cheaper.
Crucial element is the cast the row type to text (x::text).
You can even make the function work for any table - but never more than one at a time: With a polymorphic parameter type:
Refactor a PL/pgSQL function to return the output of various SELECT queries

Elegant way of handling PostgreSQL exceptions?

In PostgreSQL, I would like to create a safe-wrapping mechanism which returns empty result if an exception occurs. Consider the following:
SELECT * FROM myschema.mytable;
I could do the safe-wrapping in the client application:
try {
result = execute_query('SELECT value FROM myschema.mytable').fetchall();
}
catch(pg_exception) {
result = []
}
But could I do such a thing in SQL directly? I would like to make the following code work, but it seems like it should by put into DO $$ ... $$ block and here I'm getting lost.
BEGIN
SELECT * FROM myschema.mytable;
EXCEPTION WHEN others THEN
SELECT unnest(ARRAY[]::TEXT[])
END
Exception handling in PL/pgSQL
PL/pgSQL code is always wrapped into a BEGIN ... END block. That can be inside the body of a DO statement or a function. Blocks can be nested inside - but they cannot exist outside, don't confuse it with plain SQL.
Each block can optionally contain an EXCEPTION clause for handling exceptions, but functions that need to trap exceptions are more expensive, so it's best to avoid exceptions a priori. Postgres needs to prepare for the possibility of rolling back to a point in the transaction before the exception happened, similar to an SQL SAVEPOINT. The manual:
A block containing an EXCEPTION clause is significantly more
expensive to enter and exit than a block without one. Therefore, don't
use EXCEPTION without need.
Example:
Is SELECT or INSERT in a function prone to race conditions?
How to avoid an exception in the example
A DO statement can't return anything. Create a function that takes table and schema name as parameters and returns whatever you want:
CREATE OR REPLACE FUNCTION f_tbl_value(_tbl text, _schema text = 'public')
RETURNS TABLE (value text)
LANGUAGE plpgsql AS
$func$
DECLARE
_t regclass := to_regclass(_schema || '.' || _tbl);
BEGIN
IF _t IS NULL THEN
value := ''; RETURN NEXT; -- return single empty string
ELSE
RETURN QUERY EXECUTE
'SELECT value FROM ' || _t; -- return set of values
END IF;
END
$func$;
Call:
SELECT * FROM f_tbl_value('my_table');
Or:
SELECT * FROM f_tbl_value('my_table', 'my_schema');
Assuming you want a set of rows with a single text column or an empty string if the table does not exist.
Also assuming that a column value exists if the given table exists. You could test for that, too, but you didn't ask for that.
Both input parameters are only case sensitive if double-quoted. Just like identifiers are handled in SQL statements.
The schema name defaults to 'public' in my example. Adapt to your needs. You could even ignore the schema completely and default to the current search_path.
to_regclass() is new in Postgres 9.4. For older versions substitute:
IF EXISTS (
SELECT FROM information_schema.tables
WHERE table_schema = _schema
AND table_name = _tbl
) THEN ...
This is actually more accurate, because it tests exactly what you need. More options and detailed explanation:
Table name as a PostgreSQL function parameter
Always defend against SQL injection when working with dynamic SQL! The cast to regclass does the trick here. More details:
How to check if a table exists in a given schema
If you are selecting only one column then the COALESCE() function should be able to do the trick for you
SELECT COALESCE( value, '{}'::text[] ) FROM myschema.mytable
If you require more rows you may require to create a function with types.

Strange 'Unknown table' when performing plpgsql script

I have one same table in several schemas from PostgreSQL database server. I need
execute one query like below:
CREATE OR REPLACE FUNCTION git_search() RETURNS SETOF git_log AS $$
DECLARE
sch name;
BEGIN
FOREACH sch IN
select schema_name from information_schema.schemata where schema_name not in ('pg_toast','pg_temp_1','pg_toast_temp_1','pg_catalog','information_schema')
LOOP
qry := 'select count(*) from'|| quote_ident(sch) || '.git_log gl where gl.author_contributor_id = 17';
RETURN QUERY qry;
END LOOP;
RETURN;
END;
$$ LANGUAGE plpgsql;
select git_search();
but I have the error:
ERROR: "git_log" type not exists
SQL state: 42704
The git_log table is unknown in the first line of script. (clause CREATE)
Anybody can help me?
There are more than 100 schemas where I need perform the query that is adjusted for this situation. What is the best way to do this? Where I can create the function for this purpose?
The table name would serve just fine as composite type name, because a composite type of the same name (schema-qualified) is created with every table automatically.
The immediate cause of the error: none of your tables (actually the associated composite type of the same name) named git_log can be found in the current search_path, so the type name cannot be resolved.
Since you are operating with many schemas and many instances of tables called git_log, you need to be unambiguous and schema-qualify the table name. Just pick any one of your tables in one of the schemas, they all share the same layout:
But the rest of your function isn't going to work either. It's not a "plpgsql script", but a function definition. Try this:
CREATE OR REPLACE FUNCTION git_search()
RETURNS SETOF one_schema.git_log AS
$func$
DECLARE
sch text;
BEGIN
FOR sch IN
SELECT schema_name
FROM information_schema.schemata
WHERE schema_name NOT LIKE 'pg_%'
AND schema_name <> 'information_schema'
ORDER BY schema_name
LOOP
RETURN QUERY EXECUTE format(
'SELECT count(*)
FROM %I.git_log
WHERE author_contributor_id = 17', sch);
END LOOP;
END
$func$ LANGUAGE plpgsql;
Call:
SELECT * FROM git_search();
Major points
FOREACH is for looping though arrays, you need a FOR loop.
You need dynamic SQL. Search for examples with more explanation.
Call the function with SELECT * FROM.
Related answer (one of many):
Passing column names dynamically for a record variable in PostgreSQL

Update multiple columns that start with a specific string

I am trying to update a bunch of columns in a DB for testing purposes of a feature. I have a table that is built with hibernate so all of the columns that are created for an embedded entity begin with the same name. I.e. contact_info_address_street1, contact_info_address_street2, etc.
I am trying to figure out if there is a way to do something to the affect of:
UPDATE table SET contact_info_address_* = null;
If not, I know I can do it the long way, just looking for a way to help myself out in the future if I need to do this all over again for a different set of columns.
You need dynamic SQL for this. So you must defend against possible SQL injection.
Basic query
The basic query to generate the DML command needed can look like this:
SELECT format('UPDATE tbl SET (%s) = (%s)'
,string_agg (quote_ident(attname), ', ')
,string_agg ('NULL', ', ')
)
FROM pg_attribute
WHERE attrelid = 'tbl'::regclass
AND NOT attisdropped
AND attnum > 0
AND attname ~~ 'foo_%';
Returns:
UPDATE tbl SET (foo_a, foo_b, foo_c) = (NULL, NULL, NULL);
Make use of the "column-list syntax" of UPDATE to shorten the code and simplify the task.
I query the system catalogs instead of information schema because the latter, while being standardized and guaranteed to be portable across major versions, is also notoriously slow and sometimes unwieldy. There are pros and cons, see:
Get column names and data types of a query, table or view
quote_ident() for the column names prevents SQL-injection - also necessary for identifiers.
string_agg() requires 9.0+.
Full automation with PL/pgSQL function
CREATE OR REPLACE FUNCTION f_update_cols(_tbl regclass, _col_pattern text
, OUT row_ct int, OUT col_ct bigint)
LANGUAGE plpgsql AS
$func$
DECLARE
_sql text;
BEGIN
SELECT INTO _sql, col_ct
format('UPDATE tbl SET (%s) = (%s)'
, string_agg (quote_ident(attname), ', ')
, string_agg ('NULL', ', ')
)
, count(*)
FROM pg_attribute
WHERE attrelid = _tbl
AND NOT attisdropped -- no dropped columns
AND attnum > 0 -- no system columns
AND attname LIKE _col_pattern; -- only columns matching pattern
-- RAISE NOTICE '%', _sql; -- output SQL for debugging
EXECUTE _sql;
GET DIAGNOSTICS row_ct = ROW_COUNT;
END
$func$;
COMMENT ON FUNCTION f_update_cols(regclass, text)
IS 'Updates all columns of table _tbl ($1)
that match _col_pattern ($2) in a LIKE expression.
Returns the count of columns (col_ct) and rows (row_ct) affected.';
Call:
SELECT * FROM f_update_cols('myschema.tbl', 'foo%');
To make the function more practical, it returns information as described in the comment. More about obtaining the result status in plpgsql in the manual.
I use the variable _sql to hold the query string, so I can collect the number of columns found (col_ct) in the same query.
The object identifier type regclass is the most efficient way to automatically avoid SQL injection (and sanitize non-standard names) for the table name, too. You can use schema-qualified table names to avoid ambiguities. I would advise to do so if you (can) have multiple schemas in your db! See:
Table name as a PostgreSQL function parameter
db<>fiddle here
Old sqlfiddle
There's no handy shortcut sorry. If you have to do this kind of thing a lot, you could create a function to dynamically execute sql and achieve your goal.
CREATE OR REPLACE FUNCTION reset_cols() RETURNS boolean AS $$ BEGIN
EXECUTE (select 'UPDATE table SET '
|| array_to_string(array(
select column_name::text
from information_schema.columns
where table_name = 'table'
and column_name::text like 'contact_info_address_%'
),' = NULL,')
|| ' = NULL');
RETURN true;
END; $$ LANGUAGE plpgsql;
-- run the function
SELECT reset_cols();
It's not very nice though. A better function would be one that accepts the tablename and column prefix as args. Which I'll leave as an exercise for the readers :)

Update multiple columns in a trigger function in plpgsql

Given the following schema:
create table account_type_a (
id SERIAL UNIQUE PRIMARY KEY,
some_column VARCHAR
);
create table account_type_b (
id SERIAL UNIQUE PRIMARY KEY,
some_other_column VARCHAR
);
create view account_type_a view AS select * from account_type_a;
create view account_type_b view AS select * from account_type_b;
I try to create a generic trigger function in plpgsql, which enables updating the view:
create trigger trUpdate instead of UPDATE on account_view_type_a
for each row execute procedure updateAccount();
create trigger trUpdate instead of UPDATE on account_view_type_a
for each row execute procedure updateAccount();
An unsuccessful effort of mine was:
create function updateAccount() returns trigger as $$
declare
target_table varchar := substring(TG_TABLE_NAME from '(.+)_view');
cols varchar;
begin
execute 'select string_agg(column_name,$1) from information_schema.columns
where table_name = $2' using ',', target_table into cols;
execute 'update ' || target_table || ' set (' || cols || ') = select ($1).*
where id = ($1).id' using NEW;
return NULL;
end;
$$ language plpgsql;
The problem is the update statement. I am unable to come up with a syntax that would work here. I have successfully implemented this in PL/Perl, but would be interested in a plpgsql-only solution.
Any ideas?
Update
As #Erwin Brandstetter suggested, here is the code for my PL/Perl solution. I incoporated some of his suggestions.
create function f_tr_up() returns trigger as $$
use strict;
use warnings;
my $target_table = quote_ident($_TD->{'table_name'}) =~ s/^([\w]+)_view$/$1/r;
my $NEW = $_TD->{'new'};
my $cols = join(',', map { quote_ident($_) } keys $NEW);
my $vals = join(',', map { quote_literal($_) } values $NEW);
my $query = sprintf(
"update %s set (%s) = (%s) where id = %d",
$target_table,
$cols,
$vals,
$NEW->{'id'});
spi_exec_query($query);
return;
$$ language plperl;
While #Gary's answer is technically correct, it fails to mention that PostgreSQL does support this form:
UPDATE tbl
SET (col1, col2, ...) = (expression1, expression2, ..)
Read the manual on UPDATE.
It's still tricky to get this done with dynamic SQL. I'll assume a simple case where views consist of the same columns as their underlying tables.
CREATE VIEW tbl_view AS SELECT * FROM tbl;
Problems
The special record NEW is not visible inside EXECUTE. I pass NEW as a single parameter with the USING clause of EXECUTE.
As discussed, UPDATE with list-form needs individual values. I use a subselect to split the record into individual columns:
UPDATE ...
FROM (SELECT ($1).*) x
(Parenthesis around $1 are not optional.) This allows me to simply use two column lists built with string_agg() from the catalog table: one with and one without table qualification.
It's not possible to assign a row value as a whole to individual columns. The manual:
According to the standard, the source value for a parenthesized
sub-list of target column names can be any row-valued expression
yielding the correct number of columns. PostgreSQL only allows the
source value to be a row constructor or a sub-SELECT.
INSERT is implemented simpler. If the structure of view and table are identical we can omit the column definition list. (Can be improved, see below.)
Solution
I made a couple of updates to your approach to make it shine.
Trigger function for UPDATE:
CREATE OR REPLACE FUNCTION f_trg_up()
RETURNS TRIGGER
LANGUAGE plpgsql AS
$func$
DECLARE
_tbl regclass := quote_ident(TG_TABLE_SCHEMA) || '.'
|| quote_ident(substring(TG_TABLE_NAME from '(.+)_view$'));
_cols text;
_vals text;
BEGIN
SELECT INTO _cols, _vals
string_agg(quote_ident(attname), ', ')
, string_agg('x.' || quote_ident(attname), ', ')
FROM pg_attribute
WHERE attrelid = _tbl
AND NOT attisdropped -- no dropped (dead) columns
AND attnum > 0; -- no system columns
EXECUTE format('
UPDATE %s
SET (%s) = (%s)
FROM (SELECT ($1).*) x', _tbl, _cols, _vals)
USING NEW;
RETURN NEW; -- Don't return NULL unless you knwo what you're doing
END
$func$;
Trigger function for INSERT:
CREATE OR REPLACE FUNCTION f_trg_ins()
RETURNS TRIGGER
LANGUAGE plpgsql AS
$func$
DECLARE
_tbl regclass := quote_ident(TG_TABLE_SCHEMA) || '.'
|| quote_ident(substring(TG_TABLE_NAME FROM '(.+)_view$'));
BEGIN
EXECUTE format('INSERT INTO %s SELECT ($1).*', _tbl)
USING NEW;
RETURN NEW; -- Don't return NULL unless you know what you're doing
END
$func$;
Triggers:
CREATE TRIGGER trg_instead_up
INSTEAD OF UPDATE ON a_view
FOR EACH ROW EXECUTE FUNCTION f_trg_up();
CREATE TRIGGER trg_instead_ins
INSTEAD OF INSERT ON a_view
FOR EACH ROW EXECUTE FUNCTION f_trg_ins();
Before Postgres 11 the syntax (oddly) was EXECUTE PROCEDURE instead of EXECUTE FUNCTION - which also still works.
db<>fiddle here - demonstrating INSERT and UPDATE
Old sqlfiddle
Major points
Include the schema name to make the table reference unambiguous. There can be multiple table of the same name in one database with multiple schemas!
Query pg_catalog.pg_attribute instead of information_schema.columns. Less portable, but much faster and allows to use the table-OID.
How to check if a table exists in a given schema
Table names are NOT safe against SQLi when concatenated as strings for dynamic SQL. Escape with quote_ident() or format() or with an object-identifer type. This includes the special trigger function variables TG_TABLE_SCHEMA and TG_TABLE_NAME!
Cast to the object identifier type regclass to assert the table name is valid and get the OID for the catalog look-up.
Optionally use format() to build the dynamic query string safely.
No need for dynamic SQL for the first query on the catalog tables. Faster, simpler.
Use RETURN NEW instead of RETURN NULL in these trigger functions unless you know what you are doing. (NULL would cancel the INSERT for the current row.)
This simple version assumes that every table (and view) has a unique column named id. A more sophisticated version might use the primary key dynamically.
The function for UPDATE allows the columns of view and table to be in any order, as long as the set is the same.
The function for INSERT expects the columns of view and table to be in identical order. If you want to allow arbitrary order, add a column definition list to the INSERT command, just like with UPDATE.
Updated version also covers changes to the id column by using OLD additionally.
Postgresql doesn't support updating multiple columns using the set (col1,col2) = select val1,val2 syntax.
To achieve the same in postgresql you'd use
update target_table
set col1 = d.val1,
col2 = d.val2
from source_table d
where d.id = target_table.id
This is going to make the dynamic query a bit more complex to build as you'll need to iterate the column name list you're using into individual fields. I'd suggest you use array_agg instead of string_agg as an array is easier to process than splitting the string again.
Postgresql UPDATE syntax
documentation on array_agg function