Partitioned tables cannot have ROW triggers. when adding a for each row trigger that calls a fn which creates a partition before inserting (postgres) - sql

I am new to Database, so seeking some advise or help.
I have a table that is partitioned by list host, as shown below.
CREATE TABLE public. services
(
id integer,
service_name character varying(128),
host character varying(128),
) PARTITION BY LIST (host)
And to insert in the table, I have created a function that will check if the partitioned table is present if not, let's create one before inserting and trying to add a trigger to the function.
CREATE OR REPLACE FUNCTION service_function()
RETURNS TRIGGER AS $$
DECLARE
host TEXT;
partition_name TEXT;
BEGIN
partition_name := 'services_' || host;
IF NOT EXISTS
(SELECT 1
FROM information_schema.tables
WHERE table_name= partation_name)
THEN
RAISE NOTICE 'A partition has been created %', partition_name;
EXECUTE format(E'CREATE TABLE %I PARTITION OF services for values in (%L)'), partition_name, host;
END IF;
EXECUTE format('INSERT INTO %I (id, service_name, host) VALUES ($1, $2, $3)', partition_name) using NEW.id, NEW.service_name, NEW.host;
RETURN NULL;
Now when I am trying to add a trigger,
CREATE TRIGGER insert_service_trigger
BEFORE INSERT on services
FOR EACH ROW EXECUTE PROCEDURE service_function();
END
following error is thrown: ERROR: "services" is a partitioned table
DETAIL: Partitioned tables cannot have ROW triggers.
any suggestions or solution for it?
what I am trying to achieve: services table will be in 100G+ and always host will be there in where clause for all the SELECT queries, so I thought of creating a partition using host. is this is rite approach ??

You must be using PostgreSQL v10, which is the only release that ever sported that error message. Triggers on partitioned tables were introduced in v11.
But you won't be able to achieve what you want in any PostgreSQL version. By the time your trigger has started processing, it is too late too change the table definition. You'd get the following error:
ERROR: cannot CREATE TABLE .. PARTITION OF "services" because it is being used by active queries in this session
There is no way to achieve what you want in PostgreSQL. For a thorough discussion of this and possible workarounds, read my article.

Related

PostgreSQL FOUND for CREATE TABLE statements

I am creating a function that will create a new table and insert informations about that table into other tables.
To create that table I am using the
CREATE TABLE IF NOT EXISTS
statement. Sadly it does not update the FOUND special variable in PostgreSQL nor can i find any other variable that would be updated.
Is there any way in PL/PGSQL to know whether that statement created a table or not?
The target of it is to not to double the informations in the other tables.
You may use CREATE TABLE AS in combination with ON_ERROR_ROLLBACK:
BEGIN;
-- Do inital stuff
\set ON_ERROR_ROLLBACK on
CREATE TABLE my_table AS
SELECT id, name FROM (VALUES (1, 'Bob'), (2, 'Mary')) v(id, name);
\set ON_ERROR_ROLLBACK off
-- Do remaining stuff
END;
To put it bluntly, with \set ON_ERROR_ROLLBACK on postgres will create a savepoint before each statement and automatically rollback to this savepoint or releasing it depending on the success of that statement.
The code above will execute initial and remaining stuff even if the table creation fails.
No, there are not any information if this command created table or not. The found variable is updated after query execution - not after DDL command. There is guaranteed so after this command, the table will be or this command fails to an exception.

JDBC - Get Sequence CURRVAL after NEXTVAL was used by a Trigger

I use Oracle DB V10.2.0.1.0 for my project, along with Java as the server. I am trying to add data into a few tables only through the code, but it requires using the same sequence value.
I have a sequence which represents the T_GROUP table's ID named GROUP_SEQ.
(Increment by: 1, Min_Value: 1, Max_Value: 999999999999999999999999, Cache Size: 20, Cycle: No, Order: No).
Said GROUP_SEQ is incremented by a trigger once I enter a new group into the database:
CREATE OR REPLACE TRIGGER GROUP_TRIGGER2
BEFORE INSERT ON T_GROUP for each row
begin
SELECT GROUP_SEQ.nextval
INTO :new.ID
from dual;
END;
In my code, I performed addGroup() function in my code which successfully adds a new group along with the right GROUP_SEQ value, however when I try to get the currval it fails, because I did not use nextval on it's own, and I get this exception:
ORA-08002: sequence GROUP_SEQ.currval is not yet defined in this session
Even though I did define it in the trigger. Happens the same if I run the same commands through the SQLplus cmd.
Thanks in advance!
Solved it!
The problem was indeed the connection I was using - there was a place where I used getConnection again and thus had a different session.
Thanks #krokodilko
If you have data to insert into multiple tables then write a stored procedure to do the DML for all the tables in one go:
CREATE OR REPLACE PROCEDURE add_group(
in_column_a IN T_GROUP.COLUMN_A%TYPE,
in_column_b IN T_GROUP.COLUMN_B%TYPE,
out_id OUT T_GROUP.ID%TYPE
)
AS
BEGIN
INSERT INTO T_GROUP (
id,
column_a,
column_b
) VALUES (
GROUP_SEQ.NEXTVAL,
in_column_a,
in_column_b
)
RETURNING id INTO out_id;
INSERT INTO other_table (
id
) VALUES (
out_id
);
END;
/
In almost all cases, you do not need to use triggers.

Generic Postgres 9.5 trigger to convert an UPDATE into modified INSERT

Is it possible to create a generic (not table-specific) trigger in Postgres 9.5 that would perform on instead of update that converts the update into an insert?
Basically what I want to do is (pseudocode):
sql
instead of UPDATE on TG_TABLE_NAME INSERT on TG_TABLE_NAME
I know I can create a very table-specific trigger that maps each value into an insert statement. What I'm trying to do is get away from creating this trigger on every single table.
It is a bit of an oddball idea (nothing personal), but how about this:
CREATE FUNCTION not_update_but_insert() RETURNS trigger AS $$
BEGIN
INSERT INTO TG_TABLE_NAME -- Do an INSERT...
SELECT NEW.*; -- ... using the values from the row to be updated
RETURN NULL; -- Fail the UPDATE
END;
$$ LANGUAGE plpgsql;
Obviously this would not work for any table that has a PRIMARY KEY or other UNIQUE constraints. You do have to CREATE TRIGGER x BEFORE UPDATE for every table this would apply to, so analyze your table structure before creating the trigger.
There is obviously a work-around - at least for the PKs based on a sequence - by examining the information_schema for "safe" columns in TG_TABLE_NAME and then assembling their names into strings to splice into the INSERT statement (column list of main statement and select list). Just leave the columns with sequences or appropriate default values out. This, however, does not address UNIQUE constraints that have no obvious replacement (like a user name or an email address).

Oracle equivalent for SQL Server INSERTED and DELETED tables

I am in the process of migrating a SQL Server database to Oracle, where I have to convert SQL Server procedure which uses special tables called INSERTED and DELETED in SQL Server.
As per my understanding these tables hold copies the data of last inserted/deleted records.
(find the msdn article here: http://msdn.microsoft.com/en-us/library/ms191300.aspx)
Are there any similar tables in Oracle to achieve this..? Please advise.
UPDATE:
Thanks for your answers and comments ,I think I need to explain the situation some more. Here is the full story to understand the real scenario;
Data base contains tables and shadow tables (shadow has an additional column).
When a table is updated same changes should be recorded in relevant shadow table with some additional data.
For this purpose they are having triggers for each table (these triggers copy data to relevant shadow table).
The above mentioned procedure generates these triggers dynamically for each and every table.
Now the real problem is I don't have the knowledge about the columns as triggers are dynamically generated for each table.
Basically I can’t get value like: NEW.col_1 or: OLD.col_1 as APC mentioned. Can I.?
Or else I have to write all those triggers manually using prefixes: NEW and: OLD rather than trying to generate them dynamically.
I am using Oracle 11g
Oracle triggers use pseudo-records rather than special tables. That is, instead of tables we can access the values of individual columns.
We distinguish pseudo-records in the affected table from records in (other) tables by using the prefixes :NEW and :OLD . Oracle allows us to declare our own names for these, but there is really no good reason for abandoning the standard.
Which column values can we access?
Action :OLD :NEW
------ ---- ----
INSERTING n/a Inserted value
UPDATING Superseded value Amended value
DELETING Deleted value n/a
You will see that :OLD is the same as the MSSQL table DELETED and :NEW is the same as table INSERTED
So, to trigger a business rule check when a certain column is updated:
create or replace trigger t23_bus_check_trg
before update on t23
for each row
begin
if :NEW.col_1 != :OLD.col_1 then
check_this(:NEW.col_1 , :OLD.col_1);
end if;
end t23_bus_check_trg;
There's a whole chapter on records in the PL/SQL Reference. Find out more.
There are many differences between Sql Server triggers and Oracle triggers. In Oracle, you can declare statement level or row level triggers. Sql Server only has statement level. In Oracle, you can declare before triggers or after triggers. Sql Server only has after triggers.
If you're going to be working with Oracle, although later versions have the compound trigger, get used to working with row level triggers. There you have the pseudo row designation of :old and :new, kinda like Deleted and Inserted except it's just the one row of data. It's like being in a cursor loop, something you can do in Sql Server, but cursor perform so poorly in Sql Server, developers go to great lengths to avoid them. They are commonly used in Oracle.
The general rule of thumb is this: if you need to examine the data and possibly alter it before it goes to the table, use a "before" trigger. If you want to perform an audit or logging procedure, use an "after" trigger.
The page I linked to above gives a lot of technical details, but it is absolutely atrocious at giving usable examples. For that, just google "oracle trigger tutorial" and you should get lots of handy, easy-to-learn-from examples.
Thanks for the answers and comments. here is the complete solution to my problem.If some one meet the exact problem this will help.
create or replace PROCEDURE CreateTrackingTriggers
(
-- take the target table and shadow user as agruments
v_TableName IN NVARCHAR2 DEFAULT NULL,
v_ShadowUser IN NVARCHAR2 DEFAULT 'SHADOW_USER'
)
AUTHID CURRENT_USER -- grant permission to create triggers
AS
v_TriggerName NVARCHAR2(500);
v_ColList NVARCHAR2(2000);
v_ColList_shadow NVARCHAR2(2000);
v_SQLCommand VARCHAR2(4000);
v_ColName NVARCHAR2(500);
v_ColSize NUMBER(10,0);
v_Prefix NVARCHAR2(500);
v_count NUMBER(1,0);
BEGIN
DECLARE
-- define a cursor to get the columns of the target table. order by COLUMN_ID is important
CURSOR Cols
IS SELECT COLUMN_NAME , CHAR_COL_DECL_LENGTH FROM USER_TAB_COLS
WHERE TABLE_NAME = upper(v_TableName) order by COLUMN_ID;
-- define a cursor to get the columns of the target shadow table order by COLUMN_ID is important
CURSOR Shadow_Cols
IS SELECT COLUMN_NAME , CHAR_COL_DECL_LENGTH FROM ALL_TAB_COLS
WHERE TABLE_NAME = upper(v_TableName) and upper(owner)=upper(v_ShadowUser) order by COLUMN_ID;
BEGIN
-- generate the trigger name for target table
v_TriggerName := 'TRG_' || upper(v_TableName) || '_Track' ;
-- check v_count , determine whether shdow table exist if not handle it
select count(*) into v_count from all_tables where table_name = upper(v_TableName) and owner = upper(v_ShadowUser);
-- iterate the cursor. generating column names prefixing ':new.'
OPEN Cols;
FETCH Cols INTO v_ColName,v_ColSize;
WHILE Cols%FOUND
LOOP
BEGIN
IF v_ColList IS NULL THEN
v_ColList := ':new.'||v_ColName ;
ELSE
v_ColList := v_ColList || ',' || ':new.'||v_ColName;
END IF;
FETCH Cols INTO v_ColName,v_ColSize;
END;
END LOOP;
CLOSE Cols;
-- iterate the cursor. get the shadow table columns
OPEN Shadow_Cols;
FETCH Shadow_Cols INTO v_ColName,v_ColSize;
WHILE Shadow_Cols%FOUND
LOOP
BEGIN
IF v_ColList_shadow IS NULL THEN
v_ColList_shadow := v_ColName;
ELSE
v_ColList_shadow := v_ColList_shadow || ',' || v_ColName;
END IF;
FETCH Shadow_Cols INTO v_ColName,v_ColSize;
END;
END LOOP;
CLOSE Shadow_Cols;
-- create trigger command. This will generate the trigger that dupilicates target table's data into shdow table
v_SQLCommand := 'CREATE or REPLACE TRIGGER '||v_TriggerName||'
AFTER INSERT OR UPDATE OR DELETE ON '||upper(v_TableName)||'
REFERENCING OLD AS old NEW AS new
FOR EACH ROW
DECLARE
ErrorCode NUMBER(19,0);
BEGIN
-- v_ColList_shadow : shdow table column list
-- v_ColList : target table column list with :new prefixed
INSERT INTO '|| v_ShadowUser ||'.'||upper(v_TableName)||'('||v_ColList_shadow||') values ('||v_ColList||');
EXCEPTION
WHEN OTHERS THEN ErrorCode := SQLCODE;
END;';
EXECUTE IMMEDIATE v_SQLCommand;
END;
END;

What would be the right steps for horizontal partitioning in Postgresql?

We have an E-commerce portal with a Postgresql 9.1 database. One very important table has at the moment 32 million records. If we want to deliver all items this table would grow to 320 million records, mostly dates. Which would be to heavy.
So we are thinking about horizontal partitioning / sharding. We can divide items in this table into 12 pieces horizontal (1 per month). What would be the best steps and technics to do so? Would horizontal partitioning within the database be good enough or do we have to start thinking about sharding?
While 320 million is not small, it's not really huge either.
It largely depends on the queries you run on the table. If you always include the partition key in your queries then "regular" partitioning would probably work.
An example for this can be found in the PostgreSQL wiki:
http://wiki.postgresql.org/wiki/Month_based_partitioning
The manual also explains some of the caveats of partitioning:
http://www.postgresql.org/docs/current/interactive/ddl-partitioning.html
If you are thinking about sharding, you might read how Instagram (which is powered by PostgreSQL) has implemented that:
http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
If you have mostly read-queries, another option might be to use streaming replication to setup multiple servers and distribute the read queries by connecting to the hot-standby for read access and connecting to the master for write access. I think pg-pool II can do that (somewhat) automatically. That can be combined with partitioning to further reduce the query runtime.
If you are adventurous and don't have very immediate needs to do so, you might also consider Postgres-XC which promises to support transparent horizontal scaling:
http://postgres-xc.sourceforge.net/
There is no final release yet, but it looks like this isn't taking too long
Here is my sample code for partitioning:
t_master is a view to be select/insert/update/delete in your application
t_1 and t_2 is the underlying tables actually storing the data.
create or replace view t_master(id, col1)
as
select id, col1 from t_1
union all
select id, col1 from t_2
CREATE TABLE t_1
(
id bigint PRIMARY KEY,
col1 text
);
CREATE TABLE t_2
(
id bigint PRIMARY KEY,
col1 text
);
CREATE OR REPLACE FUNCTION t_insert_partition_function()
returns TRIGGER AS $$
begin
raise notice '%s', 'hello';
execute 'insert into t_'
|| ( mod(NEW.id, 2)+ 1 )
|| ' values ( $1, $2 )' USING NEW.id, NEW.col1 ;
RETURN NULL;
end;
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION t_update_partition_function()
returns TRIGGER AS $$
begin
raise notice '%s', 'hello';
execute 'update t_'
|| ( mod(NEW.id, 2)+ 1 )
|| ' set id = $1, col1 = $2 where id = $1'
USING NEW.id, NEW.col1 ;
RETURN NULL;
end;
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION t_delete_partition_function()
returns TRIGGER AS $$
begin
raise notice '%s', 'hello';
execute 'delete from t_'
|| ( mod(OLD.id, 2)+ 1 )
|| ' where id = $1'
USING OLD.id;
RETURN NULL;
end;
$$
LANGUAGE plpgsql;
CREATE TRIGGER t_insert_partition_trigger instead of INSERT
ON t_master FOR each row
execute procedure t_insert_partition_function();
CREATE TRIGGER t_update_partition_trigger instead of update
ON t_master FOR each row
execute procedure t_update_partition_function();
CREATE TRIGGER t_delete_partition_trigger instead of delete
ON t_master FOR each row
execute procedure t_delete_partition_function();
If you don't mind upgrading to PostgreSQL 9.4, then you could use the pg_shard extension, which lets you transparently shard a PostgreSQL table across many machines. Every shard is stored as a regular PostgreSQL table on another PostgreSQL server and replicated to other servers. It uses hash-partitioning to decide which shard(s) to use for a given query. pg_shard would work well if your queries have a natural partition dimension (e.g., customer ID).
More info: https://github.com/citusdata/pg_shard