I am trying to wrap my head around with this deadlock issue in our production, and now i really need some help.
PostgreSQL 9.5.10
Query 1: (Updating 1000 records)
update entitlements.stream_general sg
set stream_offset_id =nextval( 'entitlements.stream_general_stream_offset_id_seq' ),
should_update_offset_id = false
from (select id, topic, stream_id from entitlements.stream_general where should_update_offset_id = true limit 1000) sg2
where sg.id=sg2.id and sg.topic=sg2.topic and sg.stream_id = sg2.stream_id
Query 2: (Updating a single record)
update entitlements.stream_general set stream_action = $1::entitlements.stream_action_type, update_dt = now(), should_update_offset_id = true where stream_id = $2 and topic = $3 and id = $4
Exception :
Process 60563 waits for ShareLock on transaction 3603536083; blocked
by process 60701. Process 60701 waits for ShareLock on transaction
3603536039; blocked by process 60563.
Since there are only two transactions involved in the deadlock processes, how can one Update can be in deadlock with another Update. According to my understanding, after first Update there will be RowExclusiveLock on all those rows, and second Update should get blocked. How can there be a DEADLOCK?
stream_general table schema :
CREATE TABLE entitlements.stream_general (
stream_id int4 NOT NULL,
id varchar NOT NULL,
topic varchar NOT NULL,
stream_offset_id int8 NOT NULL DEFAULT '-1'::integer,
create_dt timestamptz NOT NULL DEFAULT now(),
update_dt timestamptz NOT NULL DEFAULT now(),
stream_action stream_action_type NOT NULL,
should_update_offset_id bool NULL,
PRIMARY KEY (stream_id, topic, id),
FOREIGN KEY (stream_id) REFERENCES entitlements.stream(stream_id) ON DELETE CASCADE
)
WITH (
OIDS=FALSE
) ;
CREATE INDEX stream_general_id_idx ON entitlements.stream_general USING btree (id, topic) ;
CREATE INDEX stream_general_should_update_offset_id_index ON entitlements.stream_general USING btree (should_update_offset_id) ;
CREATE INDEX stream_general_stream_id_idx ON entitlements.stream_general USING btree (stream_id, topic, stream_offset_id) ;
Note : stream_id is the foreign key.
Only culprit i can think of is subquery in Query1, but how i am not able to figure out how that Select can be problematic. Or may be something is up with foreign constraints.
The first query is first taking a read lock before taking a write lock. Could that cause a deadlock when the second query waits for a write lock too?
Try for update to have the subquery acquire a write lock?
update ...
from (
select ...
from ...
FOR UPDATE
) sg2
Related
Encountered this problem in production in the form of a deadlock. Figured out that if a transaction was inserting a row on my table, and I wanted to select a totally different row from that table, I would get the following error:
245: Could not position within a file via an index.
144: ISAM error: key value locked
Error in line 1
Near character position 70
My select statement was of the form select * from table where bar = 3 and foo = "CCCC";, where "foo" is a foreign key to a table with 18 rows, and "bar" is the first table's primary key. My insert statement was also inserting a row with foo = "CCCC". Curiously, the select query also returned the desired row before outputting the error.
I tried all this on informix 12.10 with isolation level set to repeatable read. I tried it on production, and in a fresh DB I set up with only the two tables mentioned. The lock mode of both tables is "row".
I investigated by modifying the select statement: select * from table where bar = 3; would not fail. Also, select * from table where bar = 3 and foo = "CCCC" order by ber; would not fail (ber being a random field from the table, ber is not indexed).
I would expect all the select statements I tried to return the desired row without error, OR all of them to fail. My solution in production was to order by a random field in the table, which fixed the deadlock issue
Does anyone know why this issue could have happened ? I suspect it is linked to the indexes on the table, which were all created automatically when adding the primary and foreign keys to the table. But I do not know enough about indexes to understand what happened. Could this be a bug ?
Schema of the tables:
create table options (
foo char(4) not null,
fee int not null)
extent size 16 next size 16
lock mode row;
alter table options add constraint (
primary key (foo)
constraint cons1 );
create table decisions (
bar char(3) not null,
foo char(4) not null,
ber int not null)
extent size 131072 next size 65536
lock mode row;
alter table decisions add constraint (
primary key (bar)
constraint cons2 );
alter table decisions add constraint (
foreign key (foo) references options(foo)
constraint cons3 );
Data I inserted into the "options" table:
AAAA|0|
BBBB|0|
CCCC|1|
DDDD|4|
EEEE|1|
FFFF|8|
Data I inserted into the "decisions" table:
QWE|AAAA|0|
WER|AAAA|9|
ERT|CCCC|2|
RTY|AAAA|32|
TYU|CCCC|1234|
YUI|CCCC|42398|
UIO|AAAA|23178|
IOP|CCCC|1233|
OPA|CCCC|11|
PAS|AAAA|890|
ASD|AAAA|90|
SDF|CCCC|2|
DFG|AAAA|4|
FGH|CCCC|7|
Edit: I used set explain on; for the queries.
select * from decisions where foo = "CCCC" and bar = "QWE" order by foo; returned that the index used was on foo="CCCC". However, for select * from decisions where foo = "CCCC" and bar = "QWE" order by ber;, it's indexed on bar="QWE".
I have a SQL table with a column called [applied], only one row from all rows can be applied ( have the value of 1) all other rows should have the value 0
Is there a check constraint that i can write to force such a case?
If you use null instead of 0, it will be much easier.
Have a CHECK constraint to make sure the (non-null) value = 1. Also have a UNIQUE constraint to only allow a single value 1.
create table testtable (
id int primary key,
applied int,
constraint applied_unique unique (applied),
constraint applied_eq_1 check (applied = 1)
);
Core ANSI SQL, i.e. expected to work with any database.
Most databases support filtered indexes:
create unique index unq_t_applied on t(applied) where applied = 1;
To know exactly how to write trigger that will help you an info of a database you use is needed.
You wil need a trigger where this will be your test control:
SELECT COUNT(APPLIED)
FROM TEST
WHERE APPLIED = 1
If it is > 0 then do not allow insert else allow.
While this can be done with triggers and constraints, they probably require an index. Instead, consider a join table.
create table things_applied (
id smallint primary key default 1,
thing_id bigint references things(id) not null,
check(id = 1)
);
Because the primary key is unique, there can only ever be one row.
The first is activated with an insert.
insert into things_applied (thing_id) values (1);
Change it by updating the row.
update things_applied set thing_id = 2;
To deactivate completely, delete the row.
delete things_applied;
To find the active row, join with the table.
select t.*
from things t
join things_applied ta on ta.thing_id = t.id
To check if it's active at all, count the rows.
select count(id) as active
from things_applied
Try it.
A table for storing items, in a particular order, associated with a container. Separate ak_* constraints involving item_id and seq ensure a container contains distinct items and the sequence of those items is distinct.
CREATE TABLE [container_items] (
[container_item_id] INT IDENTITY (1, 1) NOT NULL,
[container_id] INT NOT NULL,
[item_id] INT NOT NULL,
[seq] INT NOT NULL,
CONSTRAINT [pk_container_item] PRIMARY KEY CLUSTERED ([container_item_id] ASC),
CONSTRAINT [ak_container_item_seq] UNIQUE NONCLUSTERED ([container_id] ASC, [seq] ASC),
CONSTRAINT [ak_container_item_item] UNIQUE NONCLUSTERED ([container_id] ASC, [item_id] ASC),
CONSTRAINT [fk_container_item_item] FOREIGN KEY ([item_id]) REFERENCES [items] ([item_id]),
CONSTRAINT [fk_container_item_container] FOREIGN KEY ([container_id]) REFERENCES [containers] ([container_id])
);
Suppose for container_id=1 the original data is
container_item_id, container_id, item_id, seq
1,1,1,1
2,1,3,2
3,1,10,3
4,1,8,4
and some client app for reordering says the new sequence for the item_ids is
8,1
10,2
3,3
1,4
The ak_* constraints make it impossible to update the data base table directly. For instance, trying update in this manner:
update container
container_items
set item_id=8, seq=1
where container_item_id = 1
fails
Violation of UNIQUE KEY constraint ak_container_item_item. Cannot insert duplicate key in object 'container_items'. The duplicate key value is (1, 8).
The statement has been terminated.
Q: Is it worth the effort to find an algorithm that would reuse existing container_item_id records when the seq order is changed ?
A non-reusing approach would be to delete the existing records for survery_id=1 and then append the new sequenced item_ids as new records.
You can encapsulate the whole operation in a simple atomic transaction. Also, you need a at least one 'aux' value, I a couple of aux values with simple * -1 operation in this sample:
begin transaction tx1;
set transaction isolation level serializable;
update container
survey_items
set seq=-1*seq #<-- set aux values
where container_id = 1;
update container
survey_items
set seq=1
where container_id = 1 and item_id = 8;
update container
survey_items
set seq=2
where container_id = 1 and item_id = 10;
#and so on
commit;
Notice than you can work at repeatable read with the same guaranties because no phantoms are made.
We are using mysql as message queue, now we want to change from mysql to redis.
We are facing few difficulties to implement the same logic of mysql in redis.
In Mysql the process as follows:
Bulk insert into mysql database using load data infile.
in other php script we select the records order by priority with conditions and delete those records from database and process those records
How we can achieve same in redis?
in redis we are able to insert bulk data using pipe with lpush like key and json_encoded data
how can we get the data from redis key order by priority with some conditions and delete those records from redis?
We have table structure in mysql as below:
CREATE TABLE `message_queue` (
`sql_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`msgdata` text,
`pid` tinyint(4) DEFAULT NULL,
`receiver` varchar(20) DEFAULT NULL,
`time` bigint(20) DEFAULT '0',
`udhdata` varchar(100) DEFAULT NULL,
PRIMARY KEY (`sql_id`),
KEY `pid` (`pid`),
KEY `time` (`time`)
) ENGINE=MyISAM
SELECT and delete queries
SELECT * FROM message_queue WHERE (time = 0 OR time <= UNIX_TIMESTAMP()) ORDER BY pid DESC, sql_id ASC limit 500;
DELETE FROM message_queue WHERE sql_id in(ids_list)
To satisfy most of the requirements you'll need to decompose your data into several data structures.
Insert:
Every record from your message_queue table should be stored in a Hash - sql_id looks like a good key name candidate.
Keep a Sorted Set, e.g. message_by_time where each member is the sql_id and the score is time.
Query:
Use ZRANGEBYSCORE message_by_time 0 0 and ZRANGEBYSCORE message_by_time -inf <replace-with-timestamp> to get the initial range.
Delete:
Call DEL and ZREM for each element.
Your requirements also specify the need to sort and limit - these can also be done in Redis, but I recommend handling them in the code instead.
I have a situation where I need to ensure that there is only one active record with the same object_id and user_id at any time. Here is a representative table:
CREATE TABLE actions (
id SERIAL PRIMARY KEY,
object_id integer,
user_id integer,
active boolean default true,
created_at timestamptz default now()
);
By only one active record at a time, I mean you could have a sequence of inserts like the following:
insert into actions (object_id, user_id, active) values (1, 1, true);
insert into actions (object_id, user_id, active) values (1, 1, false);
but doing a subsequent
insert into actions (object_id, user_id, active) values (1, 1, true);
should fail because at this point in time, there already exists 1 active tuple with object_id = 1 and user_id = 1.
I'm using PostgreSQL 8.4.
I saw this post which looks interesting, but its Oracle specific.
I also saw this post but it requires more care regarding the transaction isolation level. I don't think it would work as-is in read committed mode.
My question is what other options are available to unsure this kind of constraint?
Edit: Removed the third insert in the first set. I think it was confusing the example. I also added the created_at time stamp to help with the context. To reiterate, there can be multiple (object_id, user_id, false) tuples, but only one (object_id, user_id, true) tuple.
Update: I accepted Craig's answer, but for others who may stumble upon something similar, here is another possible (though suboptimal) solution.
CREATE TABLE action_consistency (
object_id integer,
user_id integer,
count integer default 0,
primary key (object_id, user_id),
check (count >= 0 AND count <= 1)
);
CREATE OR REPLACE FUNCTION keep_action_consistency()
RETURNS TRIGGER AS
$BODY$
BEGIN
IF NEW.active THEN
UPDATE action_consistency
SET count = count + 1
WHERE object_id = NEW.object_id AND
user_id = NEW.user_id;
INSERT INTO action_consistency (object_id, user_id, count)
SELECT NEW.object_id, NEW.user_id, 1
WHERE NOT EXISTS (SELECT 1
FROM action_consistency
WHERE object_id = NEW.object_id AND
user_id = NEW.user_id);
ELSE
-- assuming insert will be active for simplicity
UPDATE action_consistency
SET count = count - 1
WHERE object_id = NEW.object_id AND
user_id = NEW.user_id;
END IF;
RETURN NEW;
END;
$BODY$
LANGUAGE plpgsql;
CREATE TRIGGER ensure_action_consistency AFTER INSERT OR UPDATE ON actions
FOR EACH ROW EXECUTE PROCEDURE keep_action_consistency();
It requires the use of a tracking table. For what I hope are obvious reasons, this is not at all desirable. It means that you have an additional row each distinct (object_id, user_id) in actions.
Another reason why I accepted #Craig Ringer's answer is that there are foreign key references to actions.id in other tables that are also rendered inactive when a given action tuple changes state. This why the history table is less ideal in this scenario. Thank you for the comments and answers.
Given your specification that you want to limit only one entry to being active at a time, try:
CREATE TABLE actions (
id SERIAL PRIMARY KEY,
object_id integer,
user_id integer,
active boolean default true,
created_at timestamptz default now()
);
CREATE UNIQUE INDEX actions_unique_active_y ON actions(object_id,user_id) WHERE (active = 't');
This is a partial unique index, a PostgreSQL specific feature - see partial indexes. It constrains the set such that only one (object_id,user_id) tuple may exist where active is true.
While that strictly answers your question as you explained further in comments, I think wildplasser's answer describes the more correct choice and best approach.
You can use UNIQUE constraint to ensure that the column contains the unique value...
Here, set of object_id and user_id have been made unique....
CREATE TABLE actions (
id SERIAL PRIMARY KEY,
object_id integer,
user_id integer,
active boolean default true,
UNIQUE (object_id , user_id )
);
Check Out SQLFIDDLE
Similary, if you want to make set of object_id,user_id and active as UNIQUE, you can simply add the column name in the list of UNIQUE.
CREATE TABLE actions (
id SERIAL PRIMARY KEY,
object_id integer,
user_id integer,
active boolean default true,
UNIQUE (object_id , user_id,active )
);
Check Out SQLFIDDLE
Original:
CREATE TABLE actions (
id SERIAL PRIMARY KEY,
object_id integer,
user_id integer,
active boolean default true
);
my version:
CREATE TABLE actions (
object_id integer NOT NULL REFERENCES objects (id),
user_id integer NOT NULL REFERENCES users(id),
PRIMARY KEY (user_id, object_id)
);
What are the differences:
omitted the surrogate key. It is useless, it enforces no constraint, and nobody will ever reference it
added a (composite) primary key, which happens to be the logical key
changed the two fields to NOT NULL, and made them into foreign keys (what would be the meaning of a row that would not exist in the users or objects table?
removed the boolean flag. What is the semantic difference between a {user_id,object_id} tuple that does not exist versus one that does exist but has it's "active" flag set to false? Why create three states when you only need two?