Set empty strings ('') to NULL in the whole database - sql

Question

In my database are many text columns where values are empty strings (''). The empty strings need to be set to NULL. I do not know the exact schemas, tables and columns in this database or rather I want to write a general solution which can be reused.
How would I write a query / function to find all text columns in all tables in all schemas and update all columns with empty strings ('') to NULL?

Answer 1

The most efficient way to achieve this:
Run a single UPDATE per table.
Only update nullable columns (not defined NOT NULL) with any actual empty string.
Only update rows with any actual empty string.
Leave other values unchanged.
This related answer has a plpgsql function that builds and runs the UPDATE command using system catalog pg_attribute automatically and safely for any given table:
Replace empty strings with null values
Using the function f_empty2null() from this answer, you can loop through selected tables like this:
DO
$do$
DECLARE
_tbl regclass;
BEGIN
FOR _tbl IN
SELECT c.oid::regclass
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r' -- only regular tables
AND n.nspname NOT LIKE 'pg_%' -- exclude system schemas
LOOP
RAISE NOTICE $$PERFORM f_empty2null('%');$$, _tbl;
-- PERFORM f_empty2null(_tbl); -- uncomment to prime the bomb
END LOOP;
END
$do$;
Careful! This updates all empty strings in all columns of all user tables in the DB. Be sure that's what you want or it might nuke your database.
You need UPDATE privileges on all selected tables, of course.
As a child safety device I commented the payload.
You may have noted that I use the system catalogs directly, not the information schema (which would work, too). About this:
How to check if a table exists in a given schema
Query to return output column names and data types of a query, table or view
For repeated use
Here is an integrated solution for repeated use. Without safety devices:
CREATE OR REPLACE FUNCTION f_all_empty2null(OUT _tables int, OUT _rows int) AS
$func$
DECLARE
_typ CONSTANT regtype[] := '{text, bpchar, varchar, \"char\"}';
_sql text;
_row_ct int;
BEGIN
_tables := 0; _rows := 0;
FOR _sql IN
SELECT format('UPDATE %s SET %s WHERE %s'
, t.tbl
, string_agg(format($$%1$s = NULLIF(%1$s, '')$$, t.col), ', ')
, string_agg(t.col || $$ = ''$$, ' OR '))
FROM (
SELECT c.oid::regclass AS tbl, quote_ident(attname) AS col
FROM pg_namespace n
JOIN pg_class c ON c.relnamespace = n.oid
JOIN pg_attribute a ON a.attrelid = c.oid
WHERE n.nspname NOT LIKE 'pg_%' -- exclude system schemas
AND c.relkind = 'r' -- only regular tables
AND a.attnum >= 1 -- exclude tableoid & friends
AND NOT a.attisdropped -- exclude dropped columns
AND NOT a.attnotnull -- exclude columns defined NOT NULL!
AND a.atttypid = ANY(_typ) -- only character types
ORDER BY a.attnum
) t
GROUP BY t.tbl
LOOP
EXECUTE _sql;
GET DIAGNOSTICS _row_ct = ROW_COUNT; -- report nr. of affected rows
_tables := _tables + 1;
_rows := _rows + _row_ct;
END LOOP;
END
$func$ LANGUAGE plpgsql;
Call:
SELECT * FROM pg_temp.f_all_empty2null();
Returns:
_tables | _rows
---------+---------
23 | 123456
Note how I escaped both table and columns names properly!
c.oid::regclass AS tbl, quote_ident(attname) AS col
Consider:
Table name as a PostgreSQL function parameter
Careful! Same warning as above.
Also consider the basic explanation in the answer I linked above:
Replace empty strings with null values

Answer 2

The simplest way to go about it is manually, table by table. For each table, do something like this:
START TRANSACTION;
UPDATE tablename SET
stringfield1 = NULLIF(stringfield1, ''),
stringfield2 = NULLIF(stringfield2, '');
<do some selects to make sure everything looks right>
COMMIT;
That will rewrite every row in the table, but it will only make one pass over the table. It might or might not be impractical for you.
You might instead want to do it field by field using a WHERE clause, to reduce the number of updates, like this.
START TRANSACTION;
UPDATE tablename SET stringfield1 = NULL WHERE stringfield1 = '';
<do some selects to make sure everything looks right>
COMMIT;
That will only rewrite the rows that need to be rewritten, but will require multiple passes over each table.

Answer 3

i think the code below is a generalised one. and you can use it any time anywhere:
Declare #Query varchar(1000)
declare #AllDatabaseTables table(id int,Table_Name varchar(50))
Insert into #AllDatabaseTables select Table_Name from information_schema.tables
declare #Table_Name varchar(50)
declare #i int=1
While #i<=(Select Count(*) from #AllDatabaseTables)
BEGIN
Select #Table_Name=Table_Name from #AllDatabaseTables Where id=#i
Declare #ColumnTable table(id int,ColumnName varchar(100))
Insert into #ColumnTable Select COLUMN_NAME from information_schema.columns Where Table_Name=#Table_Name and DATA_TYPE='varchar' --if the datatype is varchar type
Declare #ColumnName varchar(50)
Declare #k int=1
While #k<=(Select count(*) from #ColumnTable)
BEGIN
Select #ColumnName=ColumnName from #ColumnTable where id=#k
Set #Query='Update '+#Table_Name+' Set '+#ColumnName+'=NULL where '+#ColumnName+'='''' '
Exec(Query)
Set #k=#k+1
END
Set #i=#i+1
END