I have a Database called Knowledge in postgres. It has multiple schemas and every schema has same number of tables, table has same columns as well.
Now I want to create a new schema called Aggregate, table called aggregate.table1 and put values from schema1.table1 and schema2.table1 in it.
I need to add another column in Aggregate.table1 which holds the value representing schema.
If any value in schema1.table1 is updated then aggregate.table1 should get the updated values.
Question,
Is it possible in Postgresql? if so please help me with this.
I need this aggregated table for further processing
You can try writing an anonymous code block to iterate over all schemas and tables, so that you can import your data into the aggregate schema. The following block search for all tables contained in the schemas s1 and s2, creates a corresponding table in the schema s_agg and finally copies its records.
DO $$
DECLARE row record;
BEGIN
FOR row IN SELECT * FROM pg_tables WHERE schemaname IN ('s1','s2') LOOP
EXECUTE 'CREATE TABLE IF NOT EXISTS s_agg.'||quote_ident(row.tablename)||
' AS TABLE ' || quote_ident(row.schemaname)||'.'|| quote_ident(row.tablename) ||
' WITH NO DATA;';
EXECUTE 'INSERT INTO s_agg.' || quote_ident(row.tablename)
|| ' SELECT * FROM '||quote_ident(row.schemaname)||'.'||quote_ident(row.tablename);
END LOOP;
END;
$$;
Demo
CREATE SCHEMA s1;
CREATE SCHEMA s2;
CREATE SCHEMA s_agg;
CREATE TABLE s1.t1 (id int);
INSERT INTO s1.t1 VALUES (1);
CREATE TABLE s2.t1 (id int);
INSERT INTO s2.t1 VALUES (42);
DO $$
DECLARE row record;
BEGIN
FOR row IN SELECT * FROM pg_tables WHERE schemaname IN ('s1','s2') LOOP
EXECUTE 'CREATE TABLE IF NOT EXISTS s_agg.'||quote_ident(row.tablename)||
' AS TABLE ' || quote_ident(row.schemaname)||'.'|| quote_ident(row.tablename) ||
' WITH NO DATA;';
EXECUTE 'INSERT INTO s_agg.' || quote_ident(row.tablename)
|| ' SELECT * FROM '||quote_ident(row.schemaname)||'.'||quote_ident(row.tablename);
END LOOP;
END;
$$;
-- contains values of t1 from s1 and s2
SELECT * FROM s_agg.t1;
id
----
1
42
Note: This code works with the assumption that the aggregate schema is either empty or it has empty tables, otherwise data will be duplicated. If you run this periodically and the size of your tables isn't too large, you can add a DROP TABLE before the CREATE TABLE statement. To make it work on every commit on all tables of all schemas you have to take a look at TRIGGERS or even logical replication.
Related
I have a trigger whose purpose is to fire whenever there is a DELETE on a particular table and insert the deleted data into another table in json format.
The trigger works fine if I am specifying each column explicitly. Is there any way to access the entire table row?
This is my code.
TRIGGER1
AFTER DELETE
ON QUESTION
FOR EACH ROW
DECLARE
json_doc CLOB;
BEGIN
select json_arrayagg (
json_object ('code' VALUE :old.id,
'name' VALUE :old.text,
'description' VALUE :old.text) returning clob
) into json_doc
from dual;
PROCEDURE1(json_doc);
END;
This works fine. However, what I want is something like this. Instead of explicity specifying each column, I want to convert the entire :OLD data
TRIGGER1
AFTER DELETE
ON QUESTION
FOR EACH ROW
DECLARE
json_doc CLOB;
BEGIN
select json_arrayagg (
json_object (:old) returning clob
) into json_doc
from dual;
PROCEDURE1(json_doc);
END;
Any suggestion please.
The short and correct answer is you can't. We have a few tables in our application where we do this and the developer is responsible for updating the trigger when they add a column: this is enforced with code reviews and is probably the cleanest solution for this scenario.
The long answer is you can get close, but I wouldn't do this in production for several reasons:
Triggers are terrible for performance
Triggers are terrible for code clarity
This requires reading the row again using flashback query so
You aren't getting the values of this row from inside your current transaction: if you update the row in your transaction and then delete it the JSON will show what the values were BEFORE your update
There is a performance penalty for reading from UNDO
There is potential that UNDO won't be available and your trigger will fail
Your user needs permission to execute flashback queries
Your database needs to meet all the perquisites to support flashback queries
Deleting a lot of rows will cause the ROWID collection to get large and consume PGA
There are probably more reasons, but in the interest of "can it be done" here you go...
DROP TABLE t1;
DROP TABLE t2;
DROP TRIGGER t1_ad;
CREATE TABLE t1 (
id NUMBER,
name VARCHAR2(100),
description VARCHAR2(100)
);
CREATE TABLE t2 (
dt TIMESTAMP(9),
json_data CLOB
);
INSERT INTO t1 VALUES (1, 'A','aaaa');
INSERT INTO t1 VALUES (2, 'B','bbbb');
INSERT INTO t1 VALUES (3, 'C','cccc');
INSERT INTO t1 VALUES (4, 'D','dddd');
CREATE OR REPLACE TRIGGER t1_ad
FOR DELETE ON t1
COMPOUND TRIGGER
TYPE t_rowid_tab IS TABLE OF ROWID;
v_rowid_tab t_rowid_tab := t_rowid_tab();
AFTER EACH ROW IS
BEGIN
v_rowid_tab.extend;
v_rowid_tab(v_rowid_tab.last) := :old.rowid;
END AFTER EACH ROW;
AFTER STATEMENT IS
v_scn v$database.current_scn := dbms_flashback.get_system_change_number;
v_json_data CLOB;
v_sql CLOB;
BEGIN
FOR i IN 1 .. v_rowid_tab.count
LOOP
SELECT 'SELECT json_arrayagg(json_object(' ||
listagg('''' || lower(t.column_name) || ''' VALUE ' ||
lower(t.column_name),
', ') within GROUP(ORDER BY t.column_id) || ') RETURNING CLOB) FROM t1 AS OF SCN :scn WHERE rowid = :r'
INTO v_sql
FROM user_tab_columns t
WHERE t.table_name = 'T1';
EXECUTE IMMEDIATE v_sql
INTO v_json_data
USING v_scn, v_rowid_tab(i);
INSERT INTO t2
VALUES
(current_timestamp,
v_json_data);
END LOOP;
END AFTER STATEMENT;
END t1_ad;
/
UPDATE t1
SET NAME = 'zzzz' -- not captured
WHERE id = 2;
DELETE FROM t1 WHERE id < 3;
SELECT *
FROM t2;
-- 13-NOV-20 01.08.15.955426000 PM [{"id":1,"name":"A","description":"aaaa"}]
-- 13-NOV-20 01.08.15.969755000 PM [{"id":2,"name":"B","description":"bbbb"}]
I have a database with the column "endpointid" in a lot of tables. I am looking for a search function that would find every table containing a specific endpointid in order to write a query to delete that endpoint. I have tried a delete function to delete it from all tables but that is not working properly since a specific endpointid might not be in all tables. I know the following query gives all tables with the column name:
select table_name from all_tab_columns where lower(column_name) like lower('%endpointid%');
How can I extend that query to search for a specific record of endpointid?
Here is an example to delete rows with a specific endpointid value:
CREATE TABLE mytest (
endpointid NUMBER
);
INSERT INTO mytest VALUES ( 1 );
INSERT INTO mytest VALUES ( 2 );
DECLARE
ep NUMBER := 2;
BEGIN
FOR t_rec IN (
SELECT
table_name
FROM
all_tab_columns
WHERE
lower(column_name) LIKE lower('%endpointid%')
) LOOP
EXECUTE IMMEDIATE 'delete from '
|| t_rec.table_name
|| ' where endpointid = :1'
USING ep;
END LOOP;
END;
Note that if these tables have foreign key relationships, this may fail, as it does not take into account the ordering of the table references. If that is needed, then you would need to structure your metatada query to find those relationships.
I would be interested to drop all tables in a Redshift schema. Even though this solution works
DROP SCHEMA public CASCADE;
CREATE SCHEMA public;
is NOT good for me since that it drops SCHEMA permissions as well.
A solution like
DO $$ DECLARE
r RECORD;
BEGIN
-- if the schema you operate on is not "current", you will want to
-- replace current_schema() in query with 'schematodeletetablesfrom'
-- *and* update the generate 'DROP...' accordingly.
FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = current_schema()) LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(r.tablename) || ' CASCADE';
END LOOP;
END $$;
as reported in this thread How can I drop all the tables in a PostgreSQL database?
would be ideal. Unfortunately it doesn't work on Redshift (apparently there is no support for for loops).
Is there any other solution to achieve it?
Run this SQL and copy+paste the result on your SQL client.
If you want to do it programmatically you need to built little bit code around it.
SELECT 'DROP TABLE IF EXISTS ' || tablename || ' CASCADE;'
FROM pg_tables
WHERE schemaname = '<your_schema>'
I solved it through a procedure that deletes all records. Using this technique to truncate fails but deleting it works fine for my intents and purposes.
create or replace procedure sp_truncate_dwh() as $$
DECLARE
tables RECORD;
BEGIN
FOR tables in SELECT tablename
FROM pg_tables
WHERE schemaname = 'dwh'
order by tablename
LOOP
EXECUTE 'delete from dwh.' || quote_ident(tables.tablename) ;
END LOOP;
RETURN;
END;
$$ LANGUAGE plpgsql;
--call sp_truncate_dwh()
In addition to demircioglu's answer, I had to add Commit after every drop statement to drop all tables in my schema. SELECT 'DROP TABLE IF EXISTS ' || tablename || ' CASCADE; COMMIT;' FROM pg_tables WHERE schemaname = '<your_schema>'
P.S.: I do not have required reputation to add this note as a comment and had to add as an answer.
Using Python and pyscopg2 locally on my PC I came up with this script to delete all tables in schema:
import psycopg2
schema = "schema_to_be_deleted"
try:
conn = psycopg2.connect("dbname='{}' port='{}' host='{}' user='{}' password='{}'".format("DB_NAME", "DB_PORT", "DB_HOST", "DB_USER", "DB_PWD"))
cursor = conn.cursor()
cursor.execute("SELECT tablename FROM pg_tables WHERE schemaname = '%s'" % schema)
rows = cursor.fetchall()
for row in rows:
cursor.execute("DROP TABLE {}.{}".format(schema, row[0]))
cursor.close()
conn.commit()
except psycopg2.DatabaseError as error:
logger.error(error)
finally:
if conn is not None:
conn.close()
Replace correctly values for DB_NAME, DB_PORT, DB_HOST, DB_USER and DB_PWD to connect to the Redshift DB
The following recipe differs from other answers in the regard that it generates one SQL statement for all tables we're going to delete.
SELECT
'DROP TABLE ' ||
LISTAGG("table", ', ') ||
';'
FROM
svv_table_info
WHERE
"table" LIKE 'staging_%';
Example result:
DROP TABLE staging_077815128468462e9de8ca6fec22f284, staging_abc, staging_123;
As in other answers, you will need to copy the generated SQL and execute it separately.
References
|| operator concatenates strings
LISTAGG function concatenates every table name into a string with a separator
The table svv_table_info is used because LISTAGG doesn't want to work with pg_tables for me. Complaint:
One or more of the used functions must be applied on at least one user created tables. Examples of user table only functions are LISTAGG, MEDIAN, PERCENTILE_CONT, etc
UPD. I just now noticed that SVV_TABLE_INFO page says:
The SVV_TABLE_INFO view doesn't return any information for empty tables.
...which means empty tables will not be in the list returned by this query. I usually delete transient tables to save disk space, so this does not bother me much; but in general this factor should be considered.
I have a main table that has two columns with table names and id's. And I have those tables with table names in my DB.
For example, I find particular table name, selecting id. And then I want to populate table with that name with data. And I want to do that in one query. How I can do that?
The goal: to populate with data all tables at once, that has the names that similar with values in table name column from main table.
That is how I'm getting the list of tables. I should probably loop through it.
select tbl from asp_tbl where asp in (
select id from (
SELECT * FROM DIMENSION WHERE EXTERNALKEY LIKE 'W16%')
);
And then I will try to merge the data from other tables inside the table that needs to be populated:
MERGE INTO tbl d
USING
(SELECT ? nums, ? names from data_table) s
ON(d.product = s.product and d.ga = s.ga and d.metric_id = s.metric_id)
WHEN MATCHED THEN UPDATE SET d.names = s.names
WHEN NOT MATCHED THEN INSERT (nums, names)values(s.nums,s.names);
Did I provide enough info?
As I understand you need some stored procedure witch may fulfil a table with some test data. If so you may write something like:
create procedure fulfil_test_data (p_table_name varchar2) is
begin
for x IN (select tbl from asp_tbl where asp in (
SELECT table_id FROM DIMENSION WHERE EXTERNALKEY LIKE p_table_name )) loop
execute immediate 'insert into '|| x.tbl ||' (nums, names)
select level , chr(ascci(''A'') + mod(level,26)) from dual connect by level < 1001';
end loop;
end;
/
And call it
begin
fulfil_test_data('W16%');
end;
/
Goal: Use a given subquery's results (a single column with many rows of names) to act as the outer select's selection field.
Currently, my subquery is the following:
SELECT column_name
FROM information_schema.columns
WHERE table_name = 'test_table' AND column_name not in ('colRemove');
What I am doing in this subquery is grabbing all the column names from a table (i.e. test_table) and outputting all except for the column name specified (i.e. colRemove). As stated in the "goal", I want to use this subquery as such:
SELECT (*enter subquery from above here*)
FROM actual_table
WHERE (*enter specific conditions*)
I am working on a Netezza SQL server that is version 7.0.4.4. Ideally, I would like to make the entire query executable in one line, but for now, a working solution would be much appreciated. Thanks!
Note: I do not believe that the SQL extensions has been installed (i.e. arrays), but I will need to double check this.
A year too late, here's the best I can come up with but, as you already noticed, it requires a stored procedure to do the dynamic SQL. The stored proc creates a view with the all the columns from the source table minus the one you want to exclude.
-- Create test data.
CREATE TABLE test (firstcol INTEGER, secondcol INTEGER, thirdcol INTEGER);
INSERT INTO test (firstcol, secondcol, thirdcol) VALUES (1, 2, 3);
INSERT INTO test (firstcol, secondcol, thirdcol) VALUES (4, 5, 6);
-- Install stored procedure.
CREATE OR REPLACE PROCEDURE CreateLimitedView (varchar(ANY), varchar(ANY)) RETURNS BOOLEAN
LANGUAGE NZPLSQL AS
BEGIN_PROC
DECLARE
tableName ALIAS FOR $1;
columnToExclude ALIAS FOR $2;
colRec RECORD;
cols VARCHAR(2000); -- Adjust as needed.
isfirstcol BOOLEAN;
BEGIN
isfirstcol := true;
FOR colRec IN EXECUTE
'SELECT ATTNAME AS NAME FROM _V_RELATION_COLUMN
WHERE
NAME=UPPER('||quote_literal(tableName)||')
AND ATTNAME <> UPPER('||quote_literal(columnToExclude)||')
ORDER BY ATTNUM'
LOOP
IF isfirstcol THEN
cols := colRec.NAME;
ELSE
cols := cols || ', ' || colRec.NAME;
END IF;
isfirstcol := false;
END LOOP;
-- Should really check if 'LimitedView' already exists as a view, table or synonym.
EXECUTE IMMEDIATE 'CREATE OR REPLACE VIEW LimitedView AS SELECT ' || cols || ' FROM ' || quote_ident(tableName);
RETURN true;
END;
END_PROC
;
-- Run the stored proc to create the view.
CALL CreateLimitedView('test', 'secondcol');
-- Select results from the view.
SELECT * FROM limitedView WHERE firstcol = 4;
FIRSTCOL | THIRDCOL
----------+----------
4 | 6
You could have the stored proc return a resultset directly but then you wouldn't be able to filter results with a WHERE clause.