Error "duplicate key value violates unique constraint" while updating multiple rows - sql

I created a table in PostgreSQL and Oracle as
CREATE TABLE temp(
seqnr smallint NOT NULL,
defn_id int not null,
attr_id int not null,
input CHAR(50) NOT NULL,
CONSTRAINT pk_id PRIMARY KEY (defn_id, attr_id, seqnr)
);
This temp table has primary key as (defn_id,attr_id,seqnr) as a whole!
Then I inserted the record in the temp table as
INSERT INTO temp(seqnr,defn_id,attr_id,input)
VALUES (1,100,100,'test1');
INSERT INTO temp(seqnr,defn_id,attr_id,input)
VALUES (2,100,100,'test2');
INSERT INTO temp(seqnr,defn_id,attr_id,input)
VALUES (3,100,100,'test3');
INSERT INTO temp(seqnr,defn_id,attr_id,input)
VALUES (4,100,100,'test4');
INSERT INTO temp(seqnr,defn_id,attr_id,input)
VALUES (5,100,100,'test5');
in both Oracle and Postgres!
The table now contains:
seqnr | defn_id | attr_id | input
1 | 100 | 100 | test1
2 | 100 | 100 | test2
3 | 100 | 100 | test3
4 | 100 | 100 | test4
5 | 100 | 100 | test5
When I run the command:
UPDATE temp SET seqnr=seqnr+1
WHERE defn_id = 100 AND attr_id = 100 AND seqnr >= 1;
In case of ORACLE it is Updating 5 Rows and the O/p is
seqnr | defn_id | attr_id | input
2 | 100 | 100 | test1
3 | 100 | 100 | test2
4 | 100 | 100 | test3
5 | 100 | 100 | test4
6 | 100 | 100 | test5
But in case of PostgreSQL it is giving an error!
DETAIL: Key (defn_id, attr_id, seqnr)=(100, 100, 2) already exists.
Why does this happen and how can I replicate the same result in Postgres as Oracle?
Or how can the same result be achieved in Postgres without any errors?

UNIQUE an PRIMARY KEY constraints are checked immediately (for each row) unless they are defined DEFERRABLE - which is the solution you demand.
ALTER TABLE temp
DROP CONSTRAINT pk_id
, ADD CONSTRAINT pk_id PRIMARY KEY (defn_id, attr_id, seqnr) DEFERRABLE
;
Then your UPDATE just works.
db<>fiddle here
This comes at a cost, though. The manual:
Note that deferrable constraints cannot be used as conflict
arbitrators in an INSERT statement that includes an ON CONFLICT DO UPDATE clause.
And for FOREIGN KEY constraints:
The referenced columns must be the columns of a non-deferrable unique
or primary key constraint in the referenced table.
And:
When a UNIQUE or PRIMARY KEY constraint is not deferrable,
PostgreSQL checks for uniqueness immediately whenever a row is
inserted or modified. The SQL standard says that uniqueness should be
enforced only at the end of the statement; this makes a difference
when, for example, a single command updates multiple key values. To
obtain standard-compliant behavior, declare the constraint as
DEFERRABLE but not deferred (i.e., INITIALLY IMMEDIATE). Be aware
that this can be significantly slower than immediate uniqueness
checking.
See:
Constraint defined DEFERRABLE INITIALLY IMMEDIATE is still DEFERRED?
I would avoid a DEFERRABLE PK if at all possible. Maybe you can work around the demonstrated problem? This usually works:
UPDATE temp t
SET seqnr = t.seqnr + 1
FROM (
SELECT defn_id, attr_id, seqnr
FROM temp
WHERE defn_id = 100 AND attr_id = 100 AND seqnr >= 1
ORDER BY defn_id, attr_id, seqnr DESC
) o
WHERE (t.defn_id, t.attr_id, t.seqnr)
= (o.defn_id, o.attr_id, o.seqnr);
db<>fiddle here
But there are no guarantees as ORDER BY is not specified for UPDATE in Postgres.
Related:
UPDATE with ORDER BY

Related

Oracle - fast insert and fast latest records lookup

I have a table with logs which grew in size (~100M records) to the point where querying even the latest entries takes a considerable amount of a time.
I am wondering is there a smart way to make access to latest records fast (largest PK values) while also make inserts (appends) to it fast? I do not want to delete any data if possible, actually there is already a mechanism which monthly deletes logs older than N days.
Ideally what I mean is have the query
select * from t_logs order by log_id desc fetch first 50 rows only
to run in a split second (up to reasonable row count, say 500, if that matters).
The table is defined as follows:
CREATE TABLE t_logs (
log_id NUMBER NOT NULL,
method_name VARCHAR2(128 CHAR) NOT NULL,
msg VARCHAR2(4000 CHAR) NOT NULL,
type VARCHAR2(1 CHAR) NOT NULL,
time_stamp TIMESTAMP(6) NOT NULL,
user_created VARCHAR2(50 CHAR) DEFAULT user NOT NULL
);
CREATE UNIQUE INDEX logs_pk ON t_logs ( log_id ) REVERSE;
ALTER TABLE t_logs ADD (
CONSTRAINT logs_pk PRIMARY KEY ( log_id )
);
I am not really a DBA, so I am not familiar with all the performance tuning methods. I just use logs a lot and I was wondering if I could do something data-not-invasive to ease my pain. Up to my knowledge, what I did: tried re-computing statistics/re-analyze table (no effect), looked into query plan
-------------------------------------------
| Id | Operation | Name |
-------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | VIEW | |
| 2 | WINDOW SORT PUSHED RANK| |
| 3 | TABLE ACCESS FULL | T_LOGS |
-------------------------------------------
I would expect query to leverage index to perform the lookup, why doesn't it? Maybe this is a reason it takes so long to find the results?
Version: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Mr Cave, in the accepted answer, seems to be right
alter table t_logs drop constraint log_pk;
drop index log_pk;
create unique index logs_pk on t_logs ( log_id );
alter table t_logs add (
constraint logs_pk primary key ( log_id )
);
Queries run super fast now, plan looks as expected:
-------------------------------------------------
| Id | Operation | Name |
-------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | VIEW | |
| 2 | WINDOW NOSORT STOPKEY | |
| 3 | TABLE ACCESS BY INDEX ROWID| T_LOGS |
| 4 | INDEX FULL SCAN DESCENDING| LOGS_PK |
-------------------------------------------------
100 million rows isn't that large.
Why are you creating a reverse-key index for your primary key? Sure, that has the potential to reduce contention on inserts but were you really constrained by contention? That would be pretty unusual. Maybe you have an unusual environment. But my guess is that someone was trying to prematurely optimize the design for inserts without considering what that did to queries.
My wager would be that a nice, basic design would be more than sufficient for your needs
CREATE TABLE t_logs (
log_id NUMBER NOT NULL,
method_name VARCHAR2(128 CHAR) NOT NULL,
msg VARCHAR2(4000 CHAR) NOT NULL,
type VARCHAR2(1 CHAR) NOT NULL,
time_stamp TIMESTAMP(6) NOT NULL,
user_created VARCHAR2(50 CHAR) DEFAULT user NOT NULL
);
CREATE UNIQUE INDEX logs_pk ON t_logs ( log_id );
ALTER TABLE t_logs ADD (
CONSTRAINT logs_pk PRIMARY KEY ( log_id )
);
If you can't recreate the primary key for some reason, create an index on time_stamp and change your queries to use that
CREATE INDEX log_ts ON t_logs( time_stamp );
SELECT *
FROM log_ts
ORDER BY time_stamp DESC
FETCH FIRST 100 ROWS ONLY;

Primary key collision in scope of one trasaction

I have a postgresql database, which heavily relies on events from the outside, e.g. administrator changing / adding some fields or records might trigger a change in overall fields structure in other tables.
There lies the problem, however, as sometimes the fields changed by the trigger function are primary key fields. There is a table, which uses two foreign keys ids as the primary key, as in example below:
# | PK id1 | PK id2 | data |
0 | 1 | 1 | ab |
1 | 1 | 2 | cd |
2 | 1 | 3 | ef |
However, within one transaction (if I may call it such, since, in fact, it is a plpgsql function), the structure might be changed to:
# | PK id1 | PK id2 | data |
0 | 1 | 3 | ab |
1 | 1 | 2 | cd |
2 | 1 | 1 | ef |
Which, as you might have noticed, changed the 0th record's second primary key to 3, and the 2nd's to 1, which is the opposite of what they were before.
It is 100% certain that after the function has taken its effect there will be no collisions whatsoever, but I'm wondering, how can this be implemented?
I could, in fact, use a synthetic primary key as a BIGSERIAL, yet there is still a need for those two ids to be UNIQUE constained, so it wouldn't do the trick, unfortunately.
You can declare a constraint as deferrable, for example a primary key:
CREATE TABLE elbat (id int,
nmuloc int,
PRIMARY KEY (id)
DEFERRABLE);
You can then use SET CONSTRAINTS in a transaction to set deferrable constraints as deferred. That means that they can be violated temporarily during the transaction but must be fulfilled at the transaction's COMMIT.
Let's assume we have some data in our example table:
INSERT INTO elbat (id,
nmuloc)
VALUES (1,
1),
(2,
2);
We can now switch the IDs like this:
BEGIN TRANSACTION;
SET CONSTRAINTS ALL DEFERRED;
UPDATE elbat
SET id = 2
WHERE nmuloc = 1;
SELECT *
FROM elbat;
UPDATE elbat
SET id = 1
WHERE nmuloc = 2;
COMMIT;
There's no error even though the IDs are both 2 after the first UPDATE.
db<>fiddle
More on that can be found in the documentation, e.g. in CREATE TABLE (or ALTER TABLE) and SET CONSTRAINTS.

Select count(*) returns 0 but select * returns 2 rows

This is very similar to this question with a full minimal example.
I have a simple select query (from a non-empty table) with only left joins. The last left join happens to be with an empty table.
The query returns 2 non-null rows as it should, but simply changing it to a count(*) query makes it return 0 as the count of rows.
The same SQL works properly on both MySQL and MSSQL (after fixing the PK syntax).
Full (re-runnable if uncomented) example:
-- DROP TABLE first;
-- DROP TABLE second;
-- DROP TABLE empty;
CREATE TABLE first (
pk int,
fk int
);
ALTER TABLE first
ADD CONSTRAINT PK_first PRIMARY KEY (pk);
CREATE TABLE second (
pk int
);
ALTER TABLE second
ADD CONSTRAINT PK_second PRIMARY KEY (pk);
CREATE TABLE empty (
pk int
);
ALTER TABLE first ADD CONSTRAINT FK_first FOREIGN KEY (fk)
REFERENCES second (pk) ENABLE;
INSERT INTO second (pk)
VALUES (5);
INSERT INTO first (pk, fk)
VALUES (1, 5);
INSERT INTO first (pk, fk)
VALUES (2, 5);
SELECT
COUNT(*)
FROM first
LEFT OUTER JOIN second
ON (first.fk = second.pk)
LEFT OUTER JOIN empty
ON (1 = 1);
The last query returns 0 on my machine, but changing the count(*) to just * makes it return 2 rows.
Can anyone reproduce this? My db_version is 11.2.0.2.
Explain plan seems to see the 2 rows that should be returned:
----------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 13 | 3 (0)| 00:00:01 |
| 1 | SORT AGGREGATE | | 1 | 13 | | |
| 2 | MERGE JOIN CARTESIAN| | 2 | 26 | 3 (0)| 00:00:01 |
| 3 | VIEW | | 1 | | 2 (0)| 00:00:01 |
| 4 | TABLE ACCESS FULL | EMPTY | 1 | | 2 (0)| 00:00:01 |
| 5 | BUFFER SORT | | 2 | 26 | 3 (0)| 00:00:01 |
| 6 | INDEX FULL SCAN | PK_FIRST | 2 | 26 | 1 (0)| 00:00:01 |
----------------------------------------------------------------------------------
Note
-----
- dynamic sampling used for this statement (level=2)
I don't know much about dynamic sampling, but if i alter session set OPTIMIZER_DYNAMIC_SAMPLING=0;, then the plan shows 82 rows in each step.
Removing the primary keys fixes the problem on Oracle, but that is hardly a proper solution.
Removing the join into the empty table also fixes the problem, but it is an outer join with tautological filter, so it should be a noop.
Is this actually the intended behavior on Oracle for some reason? Or is my server just bugged?
Both MSSQL and MySQL return 2 as the count.
Edit: Round 2
It was enough to add 2 more tables and the bug shows also in 11.2.0.4. Can anyone reproduce it on more current Oracle versions?
An online fiddle here.
CREATE TABLE first (
pk int,
fk int
);
ALTER TABLE first
ADD CONSTRAINT PK_first PRIMARY KEY (pk);
CREATE TABLE second (
pk int,
fk int
);
ALTER TABLE second
ADD CONSTRAINT PK_second PRIMARY KEY (pk);
CREATE TABLE third (
pk int,
fk int
);
ALTER TABLE third
ADD CONSTRAINT PK_third PRIMARY KEY (pk);
CREATE TABLE fourth (
pk int
);
ALTER TABLE fourth
ADD CONSTRAINT PK_fourth PRIMARY KEY (pk);
CREATE TABLE empty (
pk int
);
ALTER TABLE first ADD CONSTRAINT FK_first FOREIGN KEY (fk)
REFERENCES second (pk) ENABLE;
ALTER TABLE second ADD CONSTRAINT FK_second FOREIGN KEY (fk)
REFERENCES third (pk) ENABLE;
ALTER TABLE third ADD CONSTRAINT FK_third FOREIGN KEY (fk)
REFERENCES fourth (pk) ENABLE;
INSERT INTO fourth (pk)
VALUES (50);
INSERT INTO third (pk, fk)
VALUES (10, 50);
INSERT INTO third (pk, fk)
VALUES (11, 50);
INSERT INTO second (pk, fk)
VALUES (5, 10);
INSERT INTO second (pk, fk)
VALUES (6, 10);
INSERT INTO first (pk, fk)
VALUES (1, 5);
INSERT INTO first (pk, fk)
VALUES (2, 5);
SELECT
COUNT(*)
FROM first
LEFT OUTER JOIN second
ON (first.fk = second.pk)
LEFT OUTER JOIN third
ON (first.pk = third.pk)
LEFT OUTER JOIN fourth
ON (third.fk = fourth.pk)
LEFT OUTER JOIN empty
ON (1 = 1);
Anyway the consensus seems to be that this is a bug in obsolete Oracle releases.
11.2.0.2 is too old version (EOL already )and looks like it even has never been patched.
The obvious workaroud for your bug is the hint no_query_transformation, try:
SELECT--+ no_query_transformation
COUNT(*)
FROM first
LEFT OUTER JOIN second
ON (first.fk = second.pk)
LEFT OUTER JOIN empty
ON (1 = 1);
Update and addition: you can just disable join elimination using hint NO_ELIMINATE_JOIN:
http://sqlfiddle.com/#!4/9cf338/10
SELECT--+ NO_ELIMINATE_JOIN(second)
COUNT(*)
FROM first
LEFT OUTER JOIN second
ON (first.fk = second.pk)
LEFT OUTER JOIN empty e
ON (1 = 1);
or _optimizer_join_elimination_enabled:
http://sqlfiddle.com/#!4/9cf338/10
SELECT--+ opt_param('_optimizer_join_elimination_enabled' 'false')
COUNT(*)
FROM first
LEFT OUTER JOIN second
ON (first.fk = second.pk)
LEFT OUTER JOIN third
ON (first.pk = third.pk)
LEFT OUTER JOIN fourth
ON (third.fk = fourth.pk)
LEFT OUTER JOIN empty
ON (1 = 1);

Increment a column value of an entity in an SQL table

I've set up the following table:
CREATE TABLE transactions (txn_id SERIAL, stock VARCHAR NOT NULL, qty INT NOT NULL, user_id INT NOT NULL);
On inserting few rows, the table looks like this:
txn_id | stock | qty | user_id
--------+--------+-----+---------
1 | APPL | 2 | 1
2 | GOOGLE | 3 | 4
3 | TSLA | 1 | 2
Now, while adding a new insert into the table,
for example - INSERT INTO transactions (stock, qty, user_id) VALUES ('APPL', 3, 1)
if 'stock' and 'user_id' match (as in the above example), i only want the quantity to be updated in the database. Such as in this case, the row 1 entity should have its 'qty' column value incremented by +3.
Is there a solution to this without using any ORM like SQLalchemy etc. ?
You want on conflict. First set up a unique constraint on the stock/user_id columns:
alter table transactions add constraint unq_transactions_stock_user_id unique (stock, user_id);
Then use this with an on conflict statement:
insert into transactions (stock, qty, user_id)
values ('APPL', 3, 1)
on conflict on constraint unq_transactions_stock_user_id do update
set qty = coalesce(transactions.qty, 0) + excluded.qty;

Is it possible to update an "order" column from within a trigger in MySQL?

We have a table in our system that would benefit from a numeric column so we can easily grab the 1st, 2nd, 3rd records for a job. We could, of course, update this column from the application itself, but I was hoping to do it in the database.
The final method must handle cases where users insert data that belongs in the "middle" of the results, as they may receive information out of order. They may also edit or delete records, so there will be corresponding update and delete triggers.
The table:
CREATE TABLE `test` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`seq` int(11) unsigned NOT NULL,
`job_no` varchar(20) NOT NULL,
`date` date NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=7 DEFAULT CHARSET=latin1
And some example data:
mysql> SELECT * FROM test ORDER BY job_no, seq;
+----+-----+--------+------------+
| id | seq | job_no | date |
+----+-----+--------+------------+
| 5 | 1 | 123 | 2009-10-05 |
| 6 | 2 | 123 | 2009-10-01 |
| 4 | 1 | 123456 | 2009-11-02 |
| 3 | 2 | 123456 | 2009-11-10 |
| 2 | 3 | 123456 | 2009-11-19 |
+----+-----+--------+------------+
I was hoping to update the "seq" column from a t rigger, but this isn't allowed by MySQL, with an error "Can't update table 'test' in stored function/trigger because it is already used by statement which invoked this stored function/trigger".
My test trigger is as follows:
CREATE TRIGGER `test_after_ins_tr` AFTER INSERT ON `test`
FOR EACH ROW
BEGIN
SET #seq = 0;
UPDATE
`test` t
SET
t.`seq` = #seq := (SELECT #seq + 1)
WHERE
t.`job_no` = NEW.`job_no`
ORDER BY
t.`date`;
END;
Is there any way to achieve what I'm after other than remembering to call a function after each update to this table?
What about this?
CREATE TRIGGER `test_after_ins_tr` BEFORE INSERT ON `test`
FOR EACH ROW
BEGIN
SET #seq = (SELECT COALESCE(MAX(seq),0) + 1 FROM test t WHERE t.job_no = NEW.job_no);
SET NEW.seq = #seq;
END;
From Sergi's comment above:
http://dev.mysql.com/doc/refman/5.1/en/stored-program-restrictions.html - "Within a stored function or trigger, it is not permitted to modify a table that is already being used (for reading or writing) by the statement that invoked the function or trigger."