Table Structure:
create table example_test (a_id integer, b_id integer, c_id integer, flag integer);
Unique Constraint:
Alter table example_test
add constraint u_key unique(a_id, b_id, c_id);
My code:
with a_ins_upd as (
Insert into example (a_id, b_id, c_id, flag)
select x.a_id, x.b_id, x.c_id, x.flag
from <input_tableType> x
on conflict on constraint u_key
do update
set
a_id = excluded.a_id,
b_id = excluded.b_id,
c_id = excluded.c_id,
flag = excluded.flag
where flag = 0
)
Operations on Data:
I want to ignore the records with flag=1, and do the Upsert on the other records.
Basically I think you want a filtered unique key.
Instead of:
alter table example_test
add constraint u_key unique(a_id, b_id, c_id);
You could do:
create unique index example_idx on example_test(a_id, b_id, c_id) where flag = 0;
You can then use a regular insert ... on conflict clause (without the where clause in on conflict).
ok, Exclusion Constraint is not supported by on conflict clause. Makes sense, it can update multiple records. Only way is to handle programmatically.
I need to ensure that the values in a column from a table are unique as part of a larger process.
I'm aware of the UNIQUE constraint, but I'm wondering if there is a better way to do the check.
I'm running the queries using psycopg2 so adding that tag on the off chance there's something in there that can help with this.
If the column is unique I can add a constraint. If the column is not unique adding the constraint will return an error.
If there is already a constraint of the same name a useful error is returned. in this case would prefer to just check for the existing constraint.
If the column is the primary key, the unique constraint can be added without error but in this case it would be preferable to just recognize that the column must be unique based on the primary key.
Code examples of this below.
DROP TABLE IF EXISTS unique_test;
CREATE TABLE unique_test (
pkey INT PRIMARY KEY,
unique_yes CHAR(1),
unique_no CHAR(1)
);
INSERT INTO unique_test (pkey, unique_yes, unique_no)
VALUES(1, 'a', 'a'),
(2, 'b', 'a');
CREATE UNIQUE INDEX CONCURRENTLY u_test_1 ON unique_test (unique_yes);
ALTER TABLE unique_test
ADD CONSTRAINT unique_target_1
UNIQUE USING INDEX u_test_1;
-- the above runs no problem
-- check what happens when column is not unique
CREATE UNIQUE INDEX CONCURRENTLY u_test_2 ON unique_test (unique_no);
ALTER TABLE unique_test
ADD CONSTRAINT unique_target_2
UNIQUE USING INDEX u_test_2;
-- returns:
-- SQL Error [23505]: ERROR: could not create unique index "u_test_2"
-- Detail: Key (unique_no)=(a) is duplicated.
CREATE UNIQUE INDEX CONCURRENTLY u_test_1 ON unique_test (unique_yes);
ALTER TABLE unique_test
ADD CONSTRAINT unique_target_1
UNIQUE USING INDEX u_test_1;
-- returns
-- SQL Error [42P07]: ERROR: relation "unique_target_1" already exists
-- test what happens if adding constrint to primary key column
CREATE UNIQUE INDEX CONCURRENTLY u_test_pkey ON unique_test (pkey);
ALTER TABLE unique_test
ADD CONSTRAINT unique_target_pkey
UNIQUE USING INDEX u_test_pkey;
-- this runs no problem but is inefficient.
If all you want to do is verify that values are unique, then use a query:
select unique_no, count(*)
from unique_test
group by unique_no
having count(*) > 1;
If it needs to be boolean output:
select not exists (
select unique_no, count(*)
from unique_test
group by unique_no
having count(*) > 1
);
If you just want a flag, you can use:
select count(*) <> count(distinct uniq_no) as duplicate_flag
from unique_test;
DELETE FROM
zoo x
USING zoo y
WHERE
x.animal_id < y.animal_id
AND x.animal = y.animal;
I think this is simpler, https://kb.objectrocket.com/postgresql/delete-duplicate-rows-in-postgresql-762 for reference
I want to check if the same two attribute values exist in two different tables. If the combination from table_a is not existing in table_b it should be inserted into the select statement table. Right now I have the following query, which is working:
CREATE TABLE table_a (
attr_a integer,
attr_b text,
uuid character varying(200),
CONSTRAINT table_a_pkey PRIMARY KEY (uuid)
);
CREATE TABLE table_b (
attr_a integer,
attr_b text,
uuid character varying(200),
CONSTRAINT table_b_pkey PRIMARY KEY (uuid)
);
SELECT * FROM table_a
WHERE (table_a.attr_a::text || table_a.attr_b::text) != ALL(SELECT (table_b.attr_a::text || table_b.attr_a::text) FROM table_b)
However, the execution time is pretty long. So I would like to ask if there is a faster solution to check for that.
Your where clause uses a manipulation of attr_a (casting it to text and concatinating with attr_b), so the index can't be used. Instead of this concatination, why not try a straight-forward exists operator?
SELECT *
FROM table_a a
WHERE NOT EXISTS (SELECT *
FROM table_b b
WHERE a.attr_a = b.attr_a AND
b.attr_b = b.attr_b)
Let's say I have the following PostgreSQL table:
id | key
---+--------
1 | 'a.b.c'
I need to prevent inserting records with a key that is a prefix of another key. For example, I should be able to insert:
'a.b.b'
But the following keys should not be accepted:
'a.b'
'a.b.c'
'a.b.c.d'
Is there a way to achieve this - either by a constraint or by a locking mechanism (check the existance before inserting)?
This solution is based on PostgreSQL user-defined operators and exclusion constraints (base syntax, more details).
NOTE: more testing shows this solution does not work (yet). See bottom.
Create a function has_common_prefix(text,text) which will calculate logically what you need. Mark the function as IMMUTABLE.
CREATE OR REPLACE FUNCTION
has_common_prefix(text,text)
RETURNS boolean
IMMUTABLE STRICT
LANGUAGE SQL AS $$
SELECT position ($1 in $2) = 1 OR position ($2 in $1) = 1
$$;
Create an operator for the index
CREATE OPERATOR <~> (
PROCEDURE = has_common_prefix,
LEFTARG = text,
RIGHTARG = text,
COMMUTATOR = <~>
);
Create exclusion constraint
CREATE TABLE keys ( key text );
ALTER TABLE keys
ADD CONSTRAINT keys_cannot_have_common_prefix
EXCLUDE ( key WITH <~> );
However, the last point produces this error:
ERROR: operator <~>(text,text) is not a member of operator family "text_ops"
DETAIL: The exclusion operator must be related to the index operator class for the constraint.
This is because to create an index PostgreSQL needs logical operators to be bound with physical indexing methods, via entities calles "operator classes". So we need to provide that logic:
CREATE OR REPLACE FUNCTION keycmp(text,text)
RETURNS integer IMMUTABLE STRICT
LANGUAGE SQL AS $$
SELECT CASE
WHEN $1 = $2 OR position ($1 in $2) = 1 OR position ($2 in $1) = 1 THEN 0
WHEN $1 < $2 THEN -1
ELSE 1
END
$$;
CREATE OPERATOR CLASS key_ops FOR TYPE text USING btree AS
OPERATOR 3 <~> (text, text),
FUNCTION 1 keycmp (text, text)
;
ALTER TABLE keys
ADD CONSTRAINT keys_cannot_have_common_prefix
EXCLUDE ( key key_ops WITH <~> );
Now, it works:
INSERT INTO keys SELECT 'ara';
INSERT 0 1
INSERT INTO keys SELECT 'arka';
INSERT 0 1
INSERT INTO keys SELECT 'barka';
INSERT 0 1
INSERT INTO keys SELECT 'arak';
psql:test.sql:44: ERROR: conflicting key value violates exclusion constraint "keys_cannot_have_common_prefix"
DETAIL: Key (key)=(arak) conflicts with existing key (key)=(ara).
INSERT INTO keys SELECT 'bark';
psql:test.sql:45: ERROR: conflicting key value violates exclusion constraint "keys_cannot_have_common_prefix"
DETAIL: Key (key)=(bark) conflicts with existing key (key)=(barka).
NOTE: more testing shows this solution does not work yet: The last INSERT should fail.
INSERT INTO keys SELECT 'a';
INSERT 0 1
INSERT INTO keys SELECT 'ac';
ERROR: conflicting key value violates exclusion constraint "keys_cannot_have_common_prefix"
DETAIL: Key (key)=(ac) conflicts with existing key (key)=(a).
INSERT INTO keys SELECT 'ab';
INSERT 0 1
You can use ltree module to achieve this, it will let you to create hierarchical tree-like structures. Also will help you to prevent from reinventing the wheel, creating complicated regular expressions and so on. You just need to have postgresql-contrib package installed. Take a look:
--Enabling extension
CREATE EXTENSION ltree;
--Creating our test table with a pre-loaded data
CREATE TABLE test_keys AS
SELECT
1 AS id,
'a.b.c'::ltree AS key_path;
--Now we'll do the trick with a before trigger
CREATE FUNCTION validate_key_path() RETURNS trigger AS $$
BEGIN
--This query will do our validation.
--It'll search if a key already exists in 'both' directions
--LIMIT 1 because one match is enough for our validation :)
PERFORM * FROM test_keys WHERE key_path #> NEW.key_path OR key_path <# NEW.key_path LIMIT 1;
--If found a match then raise a error
IF FOUND THEN
RAISE 'Duplicate key detected: %', NEW.key_path USING ERRCODE = 'unique_violation';
END IF;
--Great! Our new row is able to be inserted
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER test_keys_validator BEFORE INSERT OR UPDATE ON test_keys
FOR EACH ROW EXECUTE PROCEDURE validate_key_path();
--Creating a index to speed up our validation...
CREATE INDEX idx_test_keys_key_path ON test_keys USING GIST (key_path);
--The command below will work
INSERT INTO test_keys VALUES (2, 'a.b.b');
--And the commands below will fail
INSERT INTO test_keys VALUES (3, 'a.b');
INSERT INTO test_keys VALUES (4, 'a.b.c');
INSERT INTO test_keys VALUES (5, 'a.b.c.d');
Of course I did not bother creating primary key and other constraints for this test. But do not forget to do so. Also, there is much more on ltree module than I'm showing, if you need something different take a look on its docs, perhaps you'll find the answer there.
You can try below trigger. Please note that key is sql reserve word. So I would suggest you avoid using that as column name in your table.
I have added my create table syntax also for testing purpose:
CREATE TABLE my_table
(myid INTEGER, mykey VARCHAR(50));
CREATE FUNCTION check_key_prefix() RETURNS TRIGGER AS $check_key_prefix$
DECLARE
v_match_keys INTEGER;
BEGIN
v_match_keys = 0;
SELECT COUNT(t.mykey) INTO v_match_keys
FROM my_table t
WHERE t.mykey LIKE CONCAT(NEW.mykey, '%')
OR NEW.mykey LIKE CONCAT(t.mykey, '%');
IF v_match_keys > 0 THEN
RAISE EXCEPTION 'Prefix Key Error occured.';
END IF;
RETURN NEW;
END;
$check_key_prefix$ LANGUAGE plpgsql;
CREATE TRIGGER check_key_prefix
BEFORE INSERT OR UPDATE ON my_table
FOR EACH ROW
EXECUTE PROCEDURE check_key_prefix();
Here is a CHECK - based solution - it may satisfy your needs.
CREATE TABLE keys ( id serial primary key, key text );
CREATE OR REPLACE FUNCTION key_check(text)
RETURNS boolean
STABLE STRICT
LANGUAGE SQL AS $$
SELECT NOT EXISTS (
SELECT 1 FROM keys
WHERE key ~ ( '^' || $1 )
OR $1 ~ ( '^' || key )
);
$$;
ALTER TABLE keys
ADD CONSTRAINT keys_cannot_have_common_prefix
CHECK ( key_check(key) );
PS. Unfortunately, it fails in one point (multi - row inserts).
SQL is a very powerful language. Usually you can do most of the things by plain select statements. I.e. if you do not like triggers, you can use a this method for your inserts.
The only assumption is there exists at least 1 row in the table. (*)
The table:
create table my_table
(
id integer primary key,
key varchar(100)
);
Because of the assumption, we'll have at least 1 row.(*)
insert into my_table (id, key) values (1, 'a.b.c');
Now the magic sql. The trick is replace the p_key value by your key value to insert. I have, intentionally, not put that statement into a stored procedure. Because I want it to be straight forward if you want to carry it to your application side. But usually putting sql into stored procedure is better.
insert into my_table (id, key)
select (select max(id) + 1 from my_table), p_key
from my_table
where not exists (select 'p' from my_table where key like p_key || '%' or p_key like key || '%')
limit 1;
Now the tests:
-- 'a.b.b' => Inserts
insert into my_table (id, key)
select (select max(id) + 1 from my_table), 'a.b.b'
from my_table
where not exists (select 'p' from my_table where key like 'a.b.b' || '%' or 'a.b.b' like key || '%')
limit 1;
-- 'a.b' => does not insert
insert into my_table (id, key)
select (select max(id) + 1 from my_table), 'a.b'
from my_table
where not exists (select 'p' from my_table where key like 'a.b' || '%' or 'a.b' like key || '%')
limit 1;
-- 'a.b.c' => does not insert
insert into my_table (id, key)
select (select max(id) + 1 from my_table), 'a.b.c'
from my_table
where not exists (select 'p' from my_table where key like 'a.b.c' || '%' or 'a.b.c' like key || '%')
limit 1;
-- 'a.b.c.d' does not insert
insert into my_table (id, key)
select (select max(id) + 1 from my_table), 'a.b.c.d'
from my_table
where not exists (select 'p' from my_table where key like 'a.b.c.d' || '%' or 'a.b.c.d' like key || '%')
limit 1;
(*) If you wish you can get rid of this existence of the single row by introducing an Oracle like dual table. If you wish modifying the insert statement is straight forward. Let me know if you wish to do so.
One possible solution is to create a secondary table that holds the prefixes of your keys, and then use a combination of unique and exclusion constraints with an insert trigger to enforce the uniqueness semantics you want.
At a high level, this approach breaks each key down into a list of prefixes and applies something similar to readers-writer lock semantics: any number of keys may share a prefix as long as none of the keys equals the prefix. To accomplish that, the list of prefixes includes the key itself with a flag that marks it as a terminal prefix.
The secondary table looks like this. We use a CHAR rather than a BOOLEAN for the flag because later on we’ll be adding a constraint that doesn’t work on boolean columns.
CREATE TABLE prefixes (
id INTEGER NOT NULL,
prefix TEXT NOT NULL,
is_terminal CHAR NOT NULL,
CONSTRAINT prefixes_id_fk
FOREIGN KEY (id)
REFERENCES your_table (id)
ON DELETE CASCADE,
CONSTRAINT prefixes_is_terminal
CHECK (is_terminal IN ('t', 'f'))
);
Now we’ll need to define a trigger on insert into your_table to also insert rows into prefixes, such that
INSERT INTO your_table (id, key) VALUES (1, ‘abc');
causes
INSERT INTO prefixes (id, prefix, is_terminal) VALUES (1, 'a', ‘f’);
INSERT INTO prefixes (id, prefix, is_terminal) VALUES (1, 'ab', ‘f’);
INSERT INTO prefixes (id, prefix, is_terminal) VALUES (1, 'abc', ’t’);
The trigger function might look like this. I’m only covering the INSERT case here, but the function could be made to handle UPDATE as well by deleting the old prefixes and then inserting the new ones. The DELETE case is covered by the cascading foreign-key constraint on prefixes.
CREATE OR REPLACE FUNCTION insert_prefixes() RETURNS TRIGGER AS $$
DECLARE
is_terminal CHAR := 't';
remaining_text TEXT := NEW.key;
BEGIN
LOOP
IF LENGTH(remaining_text) <= 0 THEN
EXIT;
END IF;
INSERT INTO prefixes (id, prefix, is_terminal)
VALUES (NEW.id, remaining_text, is_terminal);
is_terminal := 'f';
remaining_text := LEFT(remaining_text, -1);
END LOOP;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
We add this function to the table as a trigger in the usual way.
CREATE TRIGGER insert_prefixes
AFTER INSERT ON your_table
FOR EACH ROW
EXECUTE PROCEDURE insert_prefixes();
An exclusion constraint and a partial unique index will enforce that a row where is_terminal = ’t’ can't collide with another row of the same prefix regardless of its is_terminal value, and that there's only one row with is_terminal = ’t’:
ALTER TABLE prefixes ADD CONSTRAINT prefixes_forbid_conflicts
EXCLUDE USING gist (prefix WITH =, is_terminal WITH <>);
CREATE UNIQUE INDEX ON prefixes (prefix) WHERE is_terminal = 't';
This allows new rows that don’t conflict but prevents ones that do conflict, including in multi-row INSERTs.
db=# INSERT INTO your_table (id, key) VALUES (1, 'a.b.c');
INSERT 0 1
db=# INSERT INTO your_table (id, key) VALUES (2, 'a.b.b');
INSERT 0 1
db=# INSERT INTO your_table (id, key) VALUES (3, 'a.b');
ERROR: conflicting key value violates exclusion constraint "prefixes_forbid_conflicts"
db=# INSERT INTO your_table (id, key) VALUES (4, 'a.b.c');
ERROR: duplicate key value violates unique constraint "prefixes_prefix_idx"
db=# INSERT INTO your_table (id, key) VALUES (5, 'a.b.c.d');
ERROR: conflicting key value violates exclusion constraint "prefixes_forbid_conflicts"
db=# INSERT INTO your_table (id, key) VALUES (6, 'a.b.d'), (7, 'a');
ERROR: conflicting key value violates exclusion constraint "prefixes_forbid_conflicts"
I have table A(id).
I need to
create table B(id)
add a foreign key to table A that references to B.id
for every row in A, insert a row in B and update A.b_id with the newly inserted row in B
Is it possible to do it without adding a temporary column in B that refers to A? The below does work, but I'd rather not have to make a temporary column.
alter table B add column ref_id integer references(A.id);
insert into B (ref_id) select id from A;
update A set b_id = B.id from B where B.ref_id = A.id;
alter table B drop column ref_id;
Assuming that:
1) you're using postgresql 9.1
2) B.id is a serial (so actually an int with a default value of nextval('b_id_seq')
3) when inserting to B, you actually add other fields from A otherwise the insert is useless
...I think something like this would work:
with n as (select nextval('b_id_seq') as newbid,a.id as a_id from a),
l as (insert into b(id) select newbid from n returning id as b_id)
update a set b_id=l.b_id from l,n where a.id=n.a_id and l.b_id=n.newbid;
Add the future foreign key column, but without the constraint itself:
ALTER TABLE A ADD b_id integer;
Fill the new column with values:
WITH cte AS (
SELECT
id
ROW_NUMBER() OVER (ORDER BY id) AS b_ref
FROM A
)
UPDATE A
SET b_id = cte.b_ref
FROM cte
WHERE A.id = cte.id;
Create the other table:
CREATE TABLE B (
id integer CONSTRAINT PK_B PRIMARY KEY
);
Add rows to the new table using the referencing column of the existing one:
INSERT INTO B (id)
SELECT b_id
FROM A;
Add the FOREIGN KEY constraint:
ALTER TABLE A
ADD CONSTRAINT FK_A_B FOREIGN KEY (b_id) REFERENCES B (id);
PostgeSQL dialect.
You might use an anonymous code block like this
do $$
declare
category_cursor cursor for select id from schema1.categories;
r_category bigint;
setting_id bigint;
begin
open category_cursor;
loop fetch category_cursor into r_category;
exit when not found;
insert into schema2.setting(field)
values ('field_value') returning id into setting_id;
update schema1.categories set category_setting_id = setting_id
where category_id = r_category;
end loop;
end; $$
Let assume we have two tables first - categories, second - settings which must be applied to these categories.
First step - declare cursor(collect ids from categories), and variabels where we store temporary data
Loop cursor inserting values 'field_value' into settings
Store id in variable setting_id
Update table categories with setting_id