Let me start with an example of my "one unit of work":
I need to insert a transaction into the database. Along with that transaction, a couple unique codes are stored into a separate table, the foreign key being the transaction id. I have 4 database interactions: 1) get the next transaction id from a sequence; 2) insert the transaction; 3 & 4) insert 2 unique codes using the transaction id.
This is a multi-threaded Java application. Which route is best?
each database interaction should get its own connection from the pool, commit and close it immediately after each step
a single connection should be retrieved and used for each of the 4 steps, then commit once at the end and close the connection
I worked on a similar issue some time back using 'Akka' framework 'actors 'to help me with the multi threading part.
We settled with your second approach. Using single connection to do the work.
1) get the next transaction id from a sequence; 2) insert the transaction; 3 & 4) insert 2 unique codes using the transaction id.
Lets make this work atomic. If there is a connection failure in any of the four steps roll back that complete work.
Maintain an 'id' that will make sure this piece of work is started again. Id being updated only when all transactions in work complete.
If transactions are of different types then you can have different threads operating on them(each transaction type corresponds to separate work) with new db connections.
Related
I am developing a server application with Spring + Hibernate + SQL-Server and i recognized that all my transactions are blocking other transaction, even if other transactions do not touch the same tables / rows and there are no relationships between this tables.
Here is a screenshot:
Transaction Report
The screenshot shows that a delete statement on table A blocks a select statement on table B. But there is no relationship between the tables.
In my understanding a transaction should only lock a table or row and another transactions that will hit the locked table or row will be blocked.
But why are all transactions blocked?
Do i missunderstand anything?
Solution was found:
The process was a copy-process. At the beginning i opened a transaction for the whole process and copy n-objects in this process. When i need to do selects while copying the selects were blocked. So i decide to open a transaction everytime i copy one object and after that copying-process close the transaction for this one object. So selects are no longer blocked. This design is like the copy-process in Windows. If you decide to cancel the copy-process the already copied files will stay. Like in my case the already copied objects will stay cause the transaction was closed and committed for every object separatly.
Final solution
What we did actually was to bypass the line by line insert by creating a work table & session ID, and having a stored proc that DELETE and INSERT from the temp table to the main table all at once. No more deadlocks!
Final process :
DELETE FROM TheTable_work (with sessionID, just in case…)
INSERT INTO TheTable_work (line by line with ADODB)
I call a stored procedure that does :
BEGIN TRANSACTION
DELETE FROM TheTable (with some
conditions)
INSERT INTO TheTable FROM TheTable_work (1
statement)
COMMIT
DELETE FROM TheTable_work (clean up)
We do think it's because of index lock but I am waiting for confirmation from DBA (explanation here).
Isolation levels did not change nothing (we tried all of possibilities, even turning on and off READ_COMMITTED_SNAPSHOT)
I am having an application that causes a deadlock into my database when 2 users "write" to the database at the same moment. They do not work on the same data, because they have different id, but they work on the same table.
User 1 (id = 1), User 2 (id = 2)
Process
User 1 does : 1 DELETE statement, followed by 3000 INSERT
User 2 does : 1 DELETE statement, followed by 3000 INSERT
User 2 DELETE in the middle of user 1 INSERT (example : after 1500). Result in Deadlock.
Statements (examples, but they work)
DELETE FROM dbo.TheTable WHERE id = 1 /* Delete about 3000 lines */
INSERT INTO dbo.TheTable (id, field2, field3) VALUE (1 , 'value2','value3')
I use ADODB (Microsoft Access...) so I do a simple query execute , example :
ConnectSQLServer.Execute "DELETE or INSERT statement"
Error Message
Run-time error '-2147457259 (80004005)':
Transaction (Process ID76) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Other informations
There is no transaction involved, only simple INSERT or DELETE query,
one by one
There is no RecordSet involved, nor open
We tried to looked for a deadlock trace but no graphic showed up, a DBA tries to understand why
I'd like to understand why they deadlock each other and not just "wait" until the other one is finished! How can I do better operations ? =)
Access : it's a access front-end with SQL-Server backend. No linked tables, vba code pushed queries via ADODB connection.
Edit : Deadlock graph
On the left we have 1 (of 3000) INSERT statement, on the right one DELETE.
UPDATE
I deleted an index on a column that is used in the DELETE statement and I can not reproduce the deadlock anymore! Not a clean solution, but temporary. We think about INSERT 3000 lines in a temp table then copy from temp table to TheTable all at once (DELETE and INSERT in a stored procedure).
Thanks,
Séb
DELETE will hold an Update lock on the pages :
Update (U)
Used on resources that can be updated. Prevents a common form of
deadlock that occurs when multiple sessions are reading, locking, and
potentially updating resources later.
And INSERT will hold an intent/exclusive lock on the same pages.
Exclusive (X)
Used for data-modification operations, such as INSERT, UPDATE, or
DELETE. Ensures that multiple updates cannot be made to the same
resource at the same time.
Intent
Used to establish a lock hierarchy. The types of intent locks are:
intent shared (IS), intent exclusive (IX), and shared with intent
exclusive (SIX).
If you wish to run both queries at the same time,
each will have to wait for the other at a given time.
I suggest you to run each insert in a seperate transaction if no rollback is needed for the whole set of inserts.
Or, try to remove parallelization, or do these operations at different times.
To make short :
1) Try to make sure each INSERT is correctly commited before going to the next one.
2) Or try to avoid doing both operations at the same time
Another guess would be to adjust the Transaction Isolation Level for the sessions that produce the problem. Check the following documentation.
Reference :
Lock Modes
Transaction Isolation Level
SQL Table contains a table with a primary key with two columns: ID and Version. When the application attempts to save to the database it uses a command that looks similar to this:
BEGIN TRANSACTION
INSERT INTO [dbo].[FirstTable] ([ID],[VERSION]) VALUES (41,19)
INSERT INTO [dbo].[SecondTable] ([ID],[VERSION]) VALUES (41,19)
COMMIT TRANSACTION
SecondTable has a foreign key constraint to FirstTable and matching column names.
If two computers execute this statement at the same time, does the first computer lock FirstTable and the second computer waits, then when it is finished waiting it finds that the first insert statement fails and throws an error and does not execute the second statement? Is it possible for the second insert to run successfully on both computers?
What is the safest way to ensure that the second computer does not write anything and returns an error to the calling application?
Since these are transactions, all operations must be successful otherwise everything gets rolled back. Whichever computer completes the first statement of the transaction first will ultimately complete its transaction. The other computer will attempt to insert a duplicate primary key and fail, causing its transaction to roll back.
Ultimately there will be one row added to both tables, via only one of the computers.
Ofcourse if there are 2 statements then one of them is going to be executed and the other one will throw an error . In order to avoid this your best bet would be to use either "if exists" or "if not exists"
What this does is basically checks to see if there is data already present in the table if not then you insert , else just select.
Take a look at the flow shown below :
if not exists (select * from Table with (updlock, rowlock, holdlock) where
...)
/* insert */
else
/* update */
commit /* locks are released here */
The transaction did not rollback automatically if an error was encountered. It needed to have
set xact_abort on
Found this in this question
SQL Server - transactions roll back on error?
I use libpqxx for my connection to postgresql. And everything was ok, until i run serialazable query on one table on one row.
table:
CREATE TABLE t1(id integer primary key);
postgres 9.4.4_x64
pqxx::connection c1(conn_str);
pqxx::connection c2(conn_str);
pqxx::transaction<pqxx::isolation_level::serializable> t1(c1);
t1.exec("INSERT INTO t1 (id) VALUES (25)");
pqxx::transaction<pqxx::isolation_level::serializable> t2(c2);
t2.exec("INSERT INTO t1 (id) VALUES (25)"); //hang here
t2.commit();
t1.commit();
my program hang forever. hang in PQexec function. Why? Is i think it must rollback one of transaction? but no? just hang.
UPDATE: same result for pure libpq:
c1 = PQconnectdb(conninfo);
c2 = PQconnectdb(conninfo);
res1 = PQexec(c1, "BEGIN");
PQclear(res1);
res1 = PQexec(c1, "INSERT INTO t1 (id) VALUES (104)");
PQclear(res1);
res2 = PQexec(c2, "BEGIN");
PQclear(res2);
res2 = PQexec(c2, "INSERT INTO t1 (id) VALUES (104)");
PQclear(res2);
res2 = PQexec(c2, "END");
PQclear(res2);
res1 = PQexec(c1, "END");
PQclear(res1);
postgresql 9.1 - same hang
The hang has nothing to do with the serializable isolation level.
I'm no libpqxx expert, but your example appears to be running both transactions in a single thread. That's your problem.
t2.exec("INSERT INTO t1 (id) VALUES (25)");
The above statement has to wait for t1 to commit or rollback before completing, but t1.commit() never gets a chance to execute. Deadlock! This is absolutely normal, and will happen regardless of your chosen isolation level. This is just a consequence of trying to run statements from 2 concurrent transactions in the same thread of execution, not a good idea.
Try running both transactions on different threads, and your hang will go away.
If only the two transaction are involved, you should get a unique violation error for transaction t2 - exactly the same as with default READ COMMITTED isolation level:
ERROR: duplicate key value violates unique constraint "t1_pkey"
DETAIL: Key (id)=(25) already exists.
t1 tried the insert first and wins regardless which transaction tries to commit first. All following transactions trying to insert the same key wait for the first. Again, this is valid for READ COMMITTED and SERIALIZABLE alike.
A possible explanation would be that a third transaction is involved, which tried to insert the same key first and is still open. Or several such transactions, artefacts of your tests.
All transactions wait for the first one that tried to insert the same key. If that one commits, all other get a unique violation. If it rolls back, the next in line gets its chance.
To check look at pg_stat_activity (being connected to the same database):
SELECT * FROM pg_stat_activity;
More specifically:
SELECT * FROM pg_stat_activity WHERE state = 'idle in transaction';
Then commit / rollback the idle transaction. Or brute-force: terminate that connection. Details:
psql: FATAL: too many connections for role
form postgres team:
"In concurrent programming, a deadlock is a situation in which two or more competing actions are each waiting for the other to finish, and thus neither ever does."
https://en.wikipedia.org/wiki/Deadlock
definition it is not a deadlock in the first place. "c1" is not waiting for "c2" to finish and can happily go about its business with id=104. However, your application never gives c1 the next command because it is stubbornly waiting for "c2" to perform its insert - which it cannot due to the lock held by "c1".
Lock testing can be done within a single thread but deadlock testing implies that both connections need to simultaneously be attempting to execute a command - which a single program cannot accomplish without threading.
David J.
I am not able to understand how select will behave while its part of exclusive transaction. Please consider following scenarios –
Scenario 1
Step 1.1
create table Tmp(x int)
insert into Tmp values(1)
Step 1.2 – session 1
begin tran
set transaction isolation level serializable
select * from Tmp
Step 1.3 – session 2
select * from Tmp
Even first session hasn't been finished, session 2 will be able to read tmp table. I thought Tmp will have exclusive lock and shared lock should not be issued to select query in session 2. And it’s not happening. I have made sure that default isolation level is READ COMMITED.
Thanks in advance for helping me in understanding this behavior.
EDIT : Why I need select in exclusive lock?
I have a SP which actually generate sequential values. So flow is -
read max values from Table and store value in variables
Update table set value=value+1
This SP is executed in parallel by several thousand instances. If two instances execute SP at same time, then they will read same value and will update value+1. Though I would like to have sequential value for every execution. I think its possible only if select is also part of exclusive lock.
If you want a transaction to be serializable, you have to change that option before you start the outermost transaction. So your first session is incorrect and is still actually running under read committed (or whatever other level was in effect for that session).
But even if you correct the order of statements, it still will not acquire an exclusive lock for a plain SELECT statement.
If you want the plain SELECT to acquire an exclusive lock, you need to ask for it:
select * from Tmp with (XLOCK)
or you need to execute a statement that actually requires an exclusive lock:
update Tmp set x = x
Your first session doesn't need an exclusive lock because it's not changing the data. If your first (serializable) session had run to completion and either rolled back or committed, before your second session was started, that session's results would still be the same because your first session didn't change the data - and so the "serializable" nature of the transaction was correct.