Having trouble with locks on a related table via triggers.
I'm inserting into table tCatalog (this has a trigger to simply insert a record in another table tSearchQueue). The insert to tCatalog is inside a transaction that has a lot of other functions that sometimes takes several seconds. However, the tSearchQueue table is locked until the transaction can be committed. Is there a way to avoid this?
INSERT INTO [dbo].[tSearchQueue] (Processed, SQL, sys_CreateDate)
SELECT
0, 'Test ' + cast(CatalogID as varchar(10)), getdate()
FROM
inserted
BEGIN TRAN t1
DECLARE #catalogid int
INSERT INTO tCatalog (ProgramID, sys_CreatedBy, ItemNumber, Description, UOMID)
VALUES (233, 1263, 'brian catalog4', 'brian catalog4', 416)
SELECT #catalogid = SCOPE_IDENTITY()
INSERT INTO tCustomAttributeCatalog (CatalogID, CustomAttributeID, DefaultValue, DefaultValueRead, sys_CreatedBy)
VALUES (#catalogid, 299, 'No', 'No', 1263)
INSERT INTO tCustomAttributeCatalog (CatalogID, CustomAttributeID, DefaultValue, DefaultValueRead, sys_CreatedBy)
VALUES (#catalogid, 300, null, null, 1263)
COMMIT TRAN t1
It looks like you have a background process which wants to be notified of changes so it can do some sort of re-indexing. If that's the case it's not necessarily wrong that it is blocked, since if the transaction does not commit then it shouldn't index it anyway.
So the sequence is:
Begin transaction
insert tCatalog
trigger inserts tSearchQueue
Insert some other tables
Perform a long-running operation
Commit transaction.
And the problem is another process wants to read tSearchQueue but cannot since it is locked.
Option 1: Perform the background operation in batches.
If the process is getting behind because it has too few opportunities to read the table, then maybe reading multiple rows at a time would solve the issue. I.e. each time it gets a chance to read the queue it should read many rows, process them all at once, and then mark them all as done together (or delete them as the case may be).
Option 2: Perform the long running operation first, if possible:
begin transaction
perform long-running operation
insert tCatalog
trigger inserts tSearchQueue
insert some other tables
commit transaction
Other process now finds tSearchQueue is locked only for a short time.
Note that if the long-running operation is a file copy, these can be included in the transaction using CopyFileTransacted, or the copy can be rolled back in a "catch" statement if the operation fails.
Option 3: Background process avoids the lock
If the other process is trying primarily to read the table, then snapshot isolation may solve your problem. This will return only committed rows, as they existed at the point in time. Combined with row-level locking this may solve your problem.
Alternatively the background process might read with the NOLOCK hint (dirty reads). This may result in reading data from transactions which are later rolled back. However if the data is being validated in a separate step (e.g. you are just writing the identifier for the object which needs reindexing) this is not necessarily a problem. If the indexing process can cope with entries which no longer exist, or which haven't actually changed, then having spurious reads won't matter.
Related
Final solution
What we did actually was to bypass the line by line insert by creating a work table & session ID, and having a stored proc that DELETE and INSERT from the temp table to the main table all at once. No more deadlocks!
Final process :
DELETE FROM TheTable_work (with sessionID, just in case…)
INSERT INTO TheTable_work (line by line with ADODB)
I call a stored procedure that does :
BEGIN TRANSACTION
DELETE FROM TheTable (with some
conditions)
INSERT INTO TheTable FROM TheTable_work (1
statement)
COMMIT
DELETE FROM TheTable_work (clean up)
We do think it's because of index lock but I am waiting for confirmation from DBA (explanation here).
Isolation levels did not change nothing (we tried all of possibilities, even turning on and off READ_COMMITTED_SNAPSHOT)
I am having an application that causes a deadlock into my database when 2 users "write" to the database at the same moment. They do not work on the same data, because they have different id, but they work on the same table.
User 1 (id = 1), User 2 (id = 2)
Process
User 1 does : 1 DELETE statement, followed by 3000 INSERT
User 2 does : 1 DELETE statement, followed by 3000 INSERT
User 2 DELETE in the middle of user 1 INSERT (example : after 1500). Result in Deadlock.
Statements (examples, but they work)
DELETE FROM dbo.TheTable WHERE id = 1 /* Delete about 3000 lines */
INSERT INTO dbo.TheTable (id, field2, field3) VALUE (1 , 'value2','value3')
I use ADODB (Microsoft Access...) so I do a simple query execute , example :
ConnectSQLServer.Execute "DELETE or INSERT statement"
Error Message
Run-time error '-2147457259 (80004005)':
Transaction (Process ID76) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Other informations
There is no transaction involved, only simple INSERT or DELETE query,
one by one
There is no RecordSet involved, nor open
We tried to looked for a deadlock trace but no graphic showed up, a DBA tries to understand why
I'd like to understand why they deadlock each other and not just "wait" until the other one is finished! How can I do better operations ? =)
Access : it's a access front-end with SQL-Server backend. No linked tables, vba code pushed queries via ADODB connection.
Edit : Deadlock graph
On the left we have 1 (of 3000) INSERT statement, on the right one DELETE.
UPDATE
I deleted an index on a column that is used in the DELETE statement and I can not reproduce the deadlock anymore! Not a clean solution, but temporary. We think about INSERT 3000 lines in a temp table then copy from temp table to TheTable all at once (DELETE and INSERT in a stored procedure).
Thanks,
Séb
DELETE will hold an Update lock on the pages :
Update (U)
Used on resources that can be updated. Prevents a common form of
deadlock that occurs when multiple sessions are reading, locking, and
potentially updating resources later.
And INSERT will hold an intent/exclusive lock on the same pages.
Exclusive (X)
Used for data-modification operations, such as INSERT, UPDATE, or
DELETE. Ensures that multiple updates cannot be made to the same
resource at the same time.
Intent
Used to establish a lock hierarchy. The types of intent locks are:
intent shared (IS), intent exclusive (IX), and shared with intent
exclusive (SIX).
If you wish to run both queries at the same time,
each will have to wait for the other at a given time.
I suggest you to run each insert in a seperate transaction if no rollback is needed for the whole set of inserts.
Or, try to remove parallelization, or do these operations at different times.
To make short :
1) Try to make sure each INSERT is correctly commited before going to the next one.
2) Or try to avoid doing both operations at the same time
Another guess would be to adjust the Transaction Isolation Level for the sessions that produce the problem. Check the following documentation.
Reference :
Lock Modes
Transaction Isolation Level
I just want to ask if it is always the first query will be executed when encapsulate to a transaction? for example i got 500 k records to be deleted and 500 k to be inserted, is there a possibility of locking?
Actually I already test this query and it works fine but i want to make sure if my assumption is correct.
Note: this will Delete and Insert the same record with possible update on other columns.
BEGIN TRAN;
DELETE FROM OUTPUT TABLE WHERE ID = (1,2,3,4 etc)
INSERT INTO OUTPUT TABLE Values (1,2,3,4 etc)
COMMIT TRAN;
Within a transaction all write locks (all locks acquired for modifications) must obey the strict two phase locking rule. One of the consequences is that a write (X) lock acquired in a transaction cannot be released until the transaction commits. So yes, the DELETE and INSERT will execute sequentially and all locks acquired during the DELETE will be retained while executing the INSERT.
Keep in mind that deleting 500k rows in a transaction will escalate the locks to one table lock, see Lock Escalation.
Deleting 500k rows and inserting 500k rows in a single transaction, while maybe correct, is a bad idea. You should avoid such large units of works, long transaction, if possible. Long transactions pin the log in place, create blocking and contention, increase recovery and DB startup time, increase SQL Server resource consumption (locks require memory).
You should consider doing the operation in small batches (perhaps 10000 rows at time), use MERGE instead of DELETE/INSERT (if possible) and, last but not least, consider a partitioned sliding window
implementation, see How to Implement an Automatic Sliding Window in a Partitioned Table.
From the documentation on TRANSACTION (emphasis mine):
BEGIN TRANSACTION represents a point at which the data referenced by a
connection is logically and physically consistent. If errors are
encountered, all data modifications made after the BEGIN TRANSACTION
can be rolled back to return the data to this known state of
consistency. Each transaction lasts until either it completes without
errors and COMMIT TRANSACTION is issued to make the modifications a
permanent part of the database, or errors are encountered and all
modifications are erased with a ROLLBACK TRANSACTION statement.
BEGIN TRANSACTION starts a local transaction for the connection
issuing the statement. Depending on the current transaction isolation
level settings, many resources acquired to support the Transact-SQL
statements issued by the connection are locked by the transaction
until it is completed with either a COMMIT TRANSACTION or ROLLBACK
TRANSACTION statement. Transactions left outstanding for long periods
of time can prevent other users from accessing these locked resources,
and also can prevent log truncation.
Although BEGIN TRANSACTION starts a local transaction, it is not
recorded in the transaction log until the application subsequently
performs an action that must be recorded in the log, such as executing
an INSERT, UPDATE, or DELETE statement. An application can perform
actions such as acquiring locks to protect the transaction isolation
level of SELECT statements, but nothing is recorded in the log until
the application performs a modification action.
Considering a forum table and many users simultaneously inserting messages into it, how safe is this transaction?
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRANSACTION
DECLARE #LastMessageId SMALLINT
SELECT #LastMessageId = MAX(MessageId)
FROM Discussions
WHERE ForumId = #ForumId AND DiscussionId = #DiscussionId
INSERT INTO Discussions
(ForumId, DiscussionId, MessageId, ParentId, MessageSubject, MessageBody)
VALUES
(#ForumId, #DiscussionId, #LastMessageId + 1, #ParentId, #MessageSubject, #MessageBody)
IF ##ERROR = 0
BEGIN
COMMIT TRANSACTION
RETURN 0
END
ROLLBACK TRANSACTION
RETURN 1
Here I read last MessageId and increment it. I can't use Identity field because it needs to be incremented for every message inserted in a group (not every message insert into table.)
Your transaction should be quite safe indeed - check out the MSDN docs on the SERIALIZABLE transaction level:
SERIALIZABLE
Specifies the following:
Statements cannot read data that has been modified but not yet
committed by other transactions.
No other transactions can modify data that has been read by the
current transaction until the current
transaction completes.
Other transactions cannot insert new rows with key values that
would fall in the range of keys read
by any statements in the current
transaction until the current
transaction completes.
Range locks are placed in the range of key values that match the
search conditions of each statement
executed in a transaction. This blocks
other transactions from updating or
inserting any rows that would qualify
for any of the statements executed by
the current transaction. This means
that if any of the statements in a
transaction are executed a second
time, they will read the same set of
rows. The range locks are held until
the transaction completes. This is the
most restrictive of the isolation
levels because it locks entire ranges
of keys and holds the locks until the
transaction completes. Because
concurrency is lower, use this option
only when necessary. This option has
the same effect as setting HOLDLOCK on
all tables in all SELECT statements in
a transaction.
The main problem with this transaction isolation level is that it's a pretty heavy load on the server, and serializes (as the name implies) any access, so your server performance and scalability will suffer, e.g. with very high numbers of users, you'll possibly get lots of timeouts for users waiting for a transaction to finish.
So using the more lightweight approach of a global message id as INT IDENTITY is definitely much better!
I have a random question. If I were to do a sql select and while the sql server was querying my request someone else does a insert statement... could that data that was inputted in that insert statement also be retrieved from my select statement?
Queries are queued, so if the SELECT occurs before the INSERT there's no possibility of seeing the newly inserted data.
Using default isolation levels, SELECT is generally given higher privilege over others but still only reads COMMITTED data. So if the INSERT data has not been committed by the time the SELECT occurs--again, you wouldn't see the newly inserted data. If the INSERT has been committed, the subsequent SELECT will include the newly inserted data.
If the isolation level allowed reading UNCOMMITTED (AKA dirty) data, then yes--a SELECT occurring after the INSERT but before the INSERT data was committed would return that data. This is not recommended practice, because UNCOMMITTED data could be subject to a ROLLBACK.
If the SELECT statement is executed before the INSERT statement, the selected data will certainly not include the new inserted data.
What happens in MySQL with MyISAM, the default engine, is that all INSERT statements require a table lock; as a result, once an INSERT statement is executed, it first waits for all existing SELECTs to complete before locking the table, performs the INSERT, and then unlocks it.
For more information, see: Internal Locking Methods in the MySQL manual
No, a SELECT that is already executing that the moment of the INSERT will never gather new records that did not exist when the SELECT statement started executing.
Also if you use the transactional storage engine InnoDB, you can be assured that your SELECT will not include rows that are currently being inserted. That's the purpose of transaction isolation, or the "I" in ACID.
For more details see http://dev.mysql.com/doc/refman/5.1/en/set-transaction.html because there are some nuances about read-committed and read-uncommitted transaction isolation modes.
I don't know particulars for MySQL, but in SQL Server it would depend on if there were any locking hints used, and the default behavior for locks. You have a couple of options:
Your SELECT locks the table, which means the INSERT won't process until your select is finished.
Your SELECT is able to do a "dirty read" which means the transaction doesn't care if you get slightly out-of-date data, and you miss the INSERT
Your SELECT is able to do a "dirty read" but the INSERT happens before the SELECT hits that row, and you get the result that was added.
The only way you do that is with a "dirty read".
Take a look at MYSql's documentation on TRANSACTION ISOLATION LEVELS to get a better understanding of what that is.
I understand how a transaction might be useful for co-ordinating a pair of updates. What I don't understand is wrapping single statements in transactions, which is 90% of what I've ever seen. In fact, in real life code it is more common in my experience to find a series of logically related transactions each wrapped in their own transaction, but the whole is not wrapped in a transaction.
In MS-SQL, is there any benefit from wrapping single selects, single updates, single inserts or single deletes in a transaction?
I suspect this is superstitious programming.
It does nothing. All individual SQL Statements, (with rare exceptions like Bulk Inserts with No Log, or Truncate Table) are automaticaly "In a Transaction" whether you explicitly say so or not.. (even if they insert, update, or delete millions of rows).
EDIT: based on #Phillip's comment below... In current versions of SQL Server, Even Bulk Inserts and Truncate Table do write some data to the transaction log, although not as much as other operations do. The critical distinction from a transactional perspective, is that in these other types of operations, the data in your database tables being modified is not in the log in a state that allows it to be rolled back.
All this means is that the changes the statement makes to data in the database are logged to the transaction log so that they can be undone if the operation fails.
The only function that the "Begin Transaction", "Commit Transaction" and "RollBack Transaction" commands provide is to allow you to put two or more individual SQL statements into the same transaction.
EDIT: (to reinforce marks comment...) YES, this could be attributed to "superstitious" programming, or it could be an indication of a fundamental misunderstanding of the nature of database transactions. A more charitable interpretation is that it is simply the result of an over-application of consistency which is inappropriate and yet another example of Emersons euphemism that:
A foolish consistency is the hobgoblin of little minds,
adored by little statesmen and philosophers and divines
As Charles Bretana said, "it does nothing" -- nothing in addition to what is already done.
Ever hear of the "ACID" requirements of a relational database? That "A" stands for Atomic, meaning that either the statement works in its entirety, or it doesn't--and while the statement is being performed, no other queries can be done on the data affected by that query. BEGIN TRANSACTION / COMMIT "extends" this locking functionality to the work done by multiple statements, but it adds nothing to single statements.
However, the database transaction log is always written to when a database is modified (insert, update, delete). This is not an option, a fact that tends to irritate people. Yes, there's wierdness with bulk inserts and recovery modes, but it still gets written to.
I'll name-drop isolation levels here too. Fussing with this will impact individual commands, but doing so will still not make a declared-transaction-wrapped query perform any differently than a "stand-alone" query. (Note that they can be very powerful and very dangeroug with multi-statement declared transactions.) Note also that "nolock" does not apply to inserts/updates/deletes -- those actions always required locks.
For me, wrapping a single statement in a transaction means that I have the ability to roll it back if I, say, forget a WHERE clause when executing a manual, one-time UPDATE statement. It has saved me a few times.
e.g.
--------------------------------------------------------------
CREATE TABLE T1(CPK INT IDENTITY(1,1) NOT NULL, Col1 int, Col2 char(3));
INSERT INTO T1 VALUES (101, 'abc');
INSERT INTO T1 VALUES (101, 'abc');
INSERT INTO T1 VALUES (101, 'abc');
INSERT INTO T1 VALUES (101, 'abc');
INSERT INTO T1 VALUES (101, 'abc');
INSERT INTO T1 VALUES (101, 'abc');
INSERT INTO T1 VALUES (101, 'abc');
SELECT * FROM T1
--------------------------------------------------------------
/* MISTAKE SCENARIO (run each row individually) */
--------------------------------------------------------------
BEGIN TRAN YOUR_TRANS_NAME_1; /* open a trans named YOUR_TRANS_NAME_1 */
UPDATE T1 SET COL2 = NULL; /* run some update statement */
SELECT * FROM T1; /* OOPS ... forgot the where clause */
ROLLBACK TRAN YOUR_TRANS_NAME_1; /* since it did bad things, roll it back */
SELECT * FROM T1; /* tans rolled back, data restored. */
--------------------------------------------------------------
/* NO MISTAKES SCENARIO (run each row individually) */
--------------------------------------------------------------
BEGIN TRAN YOUR_TRANS_NAME_2;
UPDATE T1 SET COL2 = 'CBA' WHERE CPK = 4; /* run some update statement */
SELECT * FROM T1; /* did it correctly this time */
COMMIT TRAN YOUR_TRANS_NAME_2 /* commit (close) the trans */
--------------------------------------------------------------
DROP TABLE T1
--------------------------------------------------------------
One possible excuse is that that single statement could cause a bunch of other SQL to run via triggers, and that they're protecting against something going bad in there, although I'd expect any DBMS to have the common sense to use implicit transactions in the same way already.
The other thing I can think of is that some APIs allow you to disable autocommit, and the code's written just in case someone does that.
When you start an explicit transaction and issue a DML, the resources being locked by the statement remain locked, and the results of statement are not visible from outside the transaction until you manually commit or rollback it.
This is what you may or may not need.
For instance, you may want to show preliminary results to outer world while still keeping a lock on them.
In this case, you start another transaction which places a lock request before the first one commits, thus avoiding race condition
Implicit transactions are commited or rolled back immediatley after the DML statement completes or fails.
SQL Server has a setting which allows turning autocommit off for a session. It's even the default for some clients (see https://learn.microsoft.com/en-us/sql/t-sql/statements/set-implicit-transactions-transact-sql?view=sql-server-2017)
Depending on a framework and/or a database client you use, not putting each individual command into its own transaction might cause them to be all lumped together into a default transaction. Explicitly wrapping each of them in a transaction clearly declares the intent and actually makes sure it happens the way the programmer intended, regardless of the current autocommit setting, especially if there isn't a company-wide policy on autocommit.
If the begin tran / commit tran commands are being observed in the database (as per your comment here), it is also possible that a framework is generating them on behalf of an unsuspecting programmer. (How many developers closely inspect SQL code generated by their framework?)
I hope this is still relevant, despite the question being somewhat ancient.