I have a complex Query (Without any locking hints) that takes data from many tables say Table1,Table2,Table3
Below is the code the code to retrieve data (there are no transactions)
IDbCommand sqlCmd = dbHelper.CreateCommand(Helper.MyConnString, sbSQL.ToString(), CommandType.Text, arParms);
sqlCmd.CommandTimeout = 300;
ds = dbHelper.ExecuteDataset(sqlCmd);
In an application this query runs every 2 minutes
When i fire a simple update query say
Update Table1 set Col1='abc' where ID=100
(where ID is int and primary key + clustered index)
the update query gets delayed and many times it is timeout
Below is the log
How can i fix this.
You could execute your query inside a transaction that has an isolation level of SNAPSHOT. That way, your query won't acquire any (shared) locks and your UPDATE doesn't have to wait for the exclusive lock (given that the source of the locks on the table that block your UPDATE is really your query that the application runs every two minutes...)
For reference, have a look at Working with Snapshot Isolation and SET TRANSACTION ISOLATION LEVEL on MSDN.
EDIT due to comment:
First, turn on ALLOW_SNAPSHOT_ISOLATION for your database:
ALTER DATABASE YourDB SET ALLOW_SNAPSHOT_ISOLATION ON
Then, write your query as follows:
SET TRANSACTION ISOLATION LEVEL SNAPSHOT;
SELECT Column1, Columns2, ...
FROM Table1
LEFT JOIN Table2
ON Table1.Columns45 = Table2.Column3
[... your complex query ...]
Is that enough for an example?
If you don't want wait reading queries while modifing data, it would be best to use READ_COMMITTED_SNAPSHOT.
It can be transparently on, and don't affect you app code and has no side effect.
SNAPSHOT has many side effects, for example, while you don't have locks on data modifications, you can have conflicting data problems on commiting, this problems very dificult to deal with.
Related
Test setup
I have a SQL Server 2014 and a simple table MyTable that contains columns Code (int) and Data (nvarchar(50)), no indexes created for this table.
I have 4 records in the table in the following manner:
1, First
2, Second
3, Third
4, Fourth
Then I run the following query in a transaction:
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRANSACTION
DELETE FROM dbo.MyTable
WHERE dbo.MyTable.Code = 2
I have one affected row and I don't issue either Commit or Rollback.
Next I start yet another transaction:
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRANSACTION
SELECT TOP 10 Code, Data
FROM dbo.MyTable
WHERE Code = 3
At this step the transaction with the SELECT query hangs waiting for completion of the transaction with the DELETE query.
My question
I don't understand why the transaction with SELECT query is waiting for the transaction with the DELETE query. In my understanding the deleted row (with code 2) has nothing to do with the selected row (with code 3) and as far as I understand the specific of isolation level SERIALIZABLE SQL Server shouldn't lock entire table in this case. Maybe this happens because the minimal locking amount for SERIALIZABLE is a page? Then it could produce an inconsistent behavior for selecting rows from some other pages if the table would have more rows, say 1000000 (some rows from other pages wouldn't be locked then). Please help to figure out why the locking takes place in my case.
Under locking READ COMMITTED, REPEATABLE READ, or SERIALIZABLE a SELECT query must place Shared (S) locks for every row the query plan actually reads. The locks can be placed either at the row-level, page-level, or table-level. Additionally SERIALIZABLE will place locks on ranges, so that no other session could insert a matching row while the lock is held.
And because you have "no indexes created for this table", this query:
SELECT TOP 10 Code, Data
FROM dbo.MyTable
WHERE Code = 3
Has to be executed with a table scan, and it must read all the rows (even those with Code=2) to determine whether they qualify for the SELECT.
This is one reason why you should almost always use Row-Versioning, either by setting the database to READ COMMITTED SNAPSHOT, or by coding read-only transactions to use SNAPSHOT isolation.
Given an UPDATE execution which takes 5 minutes or so, what happens when SELECT tries to retrieve data from the same table? For different Transaction Isolation Levels and SELECT WITH (NOLOCK), does SELECT wait for UPDATE? If not, does SELECT return old data (data before the UPDATE) or part of the currently inserted records (such as 50% of the records currently being inserted) ?
If found the following question, but it only describes what happens when you execute and UPDATE during a long SELECT.
SQL Server - does [SELECT] lock [UPDATE]?
I am using MS SQL Server 2012. Hopefully, this behaviour is consistent for different implementations.
This post by Gavin Draper explains it quite well and contains some example query's.
SQL Server Isolation Levels By Example
Isolation levels in SQL Server control the way locking works between
transactions.
SQL Server 2008 supports the following isolation levels
Read Uncommitted
Read Committed (The default)
Repeatable Read
Serializable
Snapshot
Before I run through each of these in detail you may want to create a
new database to run the examples, run the following script on the new
database to create the sample data. Note : You’ll also want to drop
the IsolationTests table and re-run this script before each example to
reset the data.
CREATE TABLE IsolationTests
(
Id INT IDENTITY,
Col1 INT,
Col2 INT,
Col3 INTupdate te
)
INSERT INTO IsolationTests(Col1,Col2,Col3)
SELECT 1,2,3
UNION ALL SELECT 1,2,3
UNION ALL SELECT 1,2,3
UNION ALL SELECT 1,2,3
UNION ALL SELECT 1,2,3
UNION ALL SELECT 1,2,3
UNION ALL SELECT 1,2,3
Also before we go any further it is important to understand these two
terms….
Dirty Reads – This is when you read uncommitted data, when doing this there is no guarantee that data read will ever be committed
meaning the data could well be bad.
Phantom Reads – This is when data that you are working with has been changed by another transaction since you first read it in.
This
means subsequent reads of this data in the same transaction could
well be different.
Read Uncommitted
This is the lowest isolation level there is. Read uncommitted causes
no shared locks to be requested which allows you to read data that is
currently being modified in other transactions. It also allows other
transactions to modify data that you are reading.
As you can probably imagine this can cause some unexpected results in
a variety of different ways. For example data returned by the select
could be in a half way state if an update was running in another
transaction causing some of your rows to come back with the updated
values and some not to.
To see read uncommitted in action lets run Query1 in one tab of
Management Studio and then quickly run Query2 in another tab before
Query1 completes.
Query1
BEGIN TRAN
UPDATE IsolationTests SET Col1 = 2
--Simulate having some intensive processing here with a wait
WAITFOR DELAY '00:00:10'
ROLLBACK
Query2
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
SELECT * FROM IsolationTests
Notice that Query2 will not wait for Query1 to finish, also more
importantly Query2 returns dirty data. Remember Query1 rolls back all
its changes however Query2 has returned the data anyway, this is
because it didn't wait for all the other transactions with exclusive
locks on this data it just returned what was there at the time.
There is a syntactic shortcut for querying data using the read
uncommitted isolation level by using the NOLOCK table hint. You
could change the above Query2 to look like this and it would do the
exact same thing.
SELECT * FROM IsolationTests WITH(NOLOCK)
Read Committed
This is the default isolation level and means selects will only return
committed data. Select statements will issue shared lock requests
against data you’re querying this causes you to wait if another
transaction already has an exclusive lock on that data. Once you have
your shared lock any other transactions trying to modify that data
will request an exclusive lock and be made to wait until your Read
Committed transaction finishes.
You can see an example of a read transaction waiting for a modify
transaction to complete before returning the data by running the
following Queries in separate tabs as you did with Read Uncommitted.
Query1
BEGIN TRAN
UPDATE Tests SET Col1 = 2
--Simulate having some intensive processing here with a wait
WAITFOR DELAY '00:00:10'
ROLLBACK
Query2
SELECT * FROM IsolationTests
Notice how Query2 waited for the first transaction to complete before
returning and also how the data returned is the data we started off
with as Query1 did a rollback. The reason no isolation level was
specified is because Read Committed is the default isolation level for
SQL Server. If you want to check what isolation level you are running
under you can run DBCC useroptions. Remember isolation levels are
Connection/Transaction specific so different queries on the same
database are often run under different isolation levels.
Repeatable Read
This is similar to Read Committed but with the additional guarantee
that if you issue the same select twice in a transaction you will get
the same results both times. It does this by holding on to the shared
locks it obtains on the records it reads until the end of the
transaction, This means any transactions that try to modify these
records are forced to wait for the read transaction to complete.
As before run Query1 then while its running run Query2
Query1
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRAN
SELECT * FROM IsolationTests
WAITFOR DELAY '00:00:10'
SELECT * FROM IsolationTests
ROLLBACK
Query2
UPDATE IsolationTests SET Col1 = -1
Notice that Query1 returns the same data for both selects even though
you ran a query to modify the data before the second select ran. This
is because the Update query was forced to wait for Query1 to finish
due to the exclusive locks that were opened as you specified
Repeatable Read.
If you rerun the above Queries but change Query1 to Read Committed you
will notice the two selects return different data and that Query2 does
not wait for Query1 to finish.
One last thing to know about Repeatable Read is that the data can
change between 2 queries if more records are added. Repeatable Read
guarantees records queried by a previous select will not be changed or
deleted, it does not stop new records being inserted so it is still
very possible to get Phantom Reads at this isolation level.
Serializable
This isolation level takes Repeatable Read and adds the guarantee that
no new data will be added eradicating the chance of getting Phantom
Reads. It does this by placing range locks on the queried data. This
causes any other transactions trying to modify or insert data touched
on by this transaction to wait until it has finished.
You know the drill by now run these queries side by side…
Query1
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRAN
SELECT * FROM IsolationTests
WAITFOR DELAY '00:00:10'
SELECT * FROM IsolationTests
ROLLBACK
Query2
INSERT INTO IsolationTests(Col1,Col2,Col3)
VALUES (100,100,100)
You’ll see that the insert in Query2 waits for Query1 to complete
before it runs eradicating the chance of a phantom read. If you change
the isolation level in Query1 to repeatable read, you’ll see the
insert no longer gets blocked and the two select statements in Query1
return a different amount of rows.
Snapshot
This provides the same guarantees as serializable. So what's the
difference? Well it’s more in the way it works, using snapshot doesn't
block other queries from inserting or updating the data touched by the
snapshot transaction. Instead row versioning is used so when data is
changed the old version is kept in tempdb so existing transactions
will see the version without the change. When all transactions that
started before the changes are complete the previous row version is
removed from tempdb. This means that even if another transaction has
made changes you will always get the same results as you did the first
time in that transaction.
So on the plus side your not blocking anyone else from modifying the
data whilst you run your transaction but…. You’re using extra
resources on the SQL Server to hold multiple versions of your changes.
To use the snapshot isolation level you need to enable it on the
database by running the following command
ALTER DATABASE IsolationTests
SET ALLOW_SNAPSHOT_ISOLATION ON
If you rerun the examples from serializable but change the isolation
level to snapshot you will notice that you still get the same data
returned but Query2 no longer waits for Query1 to complete.
Summary
You should now have a good idea how each of the different isolation
levels work. You can see how the higher the level you use the less
concurrency you are offering and the more blocking you bring to the
table. You should always try to use the lowest isolation level you can
which is usually read committed.
READ UNCOMMITTED: The SELECT can read all kinds of nasty inconsistencies. Old rows, new rows, duplicate rows, missing rows. It can also totally error out with the famous "data movement" error.
READ COMMITTED: Will block without snapshot isolation. Will return the old state with snapshot isolation in perfect consistency.
REPEATABLE READ/SERIALIZABLE: Will block.
SNAPSHOT: Will return the old state with snapshot isolation in perfect consistency.
It sounds like you should read a few concurrency tutorials. I have written these brief facts to get you started. To really understand what's going on to the point that you can make predictions (that come true) you need to go deeper than an answer on Stack Overflow can provide.
Most of the time, you want to use SNAPSHOT for read-only transactions. It takes away all concurrency concerns. Be aware that it has a few drawbacks.
I am pulling the data from several tables and then passing the data to a long running process. I would like to be able to record what data was used for the process and then query the database to check if any of the tables have changed since the process was last run.
Is there a method of solving this problem that should work across all sql databases?
One possible solution that I've thought of is having a separate table that is only used for keeping track of whether the data has changed since the process was run. The table contains a "stale" flag. When I start running the process, stale is set to false. If any creation, update, or deletion occurs in any of the tables on which the operation depends, I set stale to true. Is this a valid solution? Are there better solutions?
One concern with my solution is situations like this:
One user starts inserting a new row into one of the tables. Stale gets set to true, but the new row has not actually been added yet. Another user has simultaneously started the long running process, pulling the data from the tables and setting the flag to false. The row is finally added. Now the data used for the process is out of date but the flag indicates it is not stale. Would transactions be able to solve this problem?
EDIT:
This is some SQL for my idea. Not sure if it works, but just to give you a better idea of what I was thinking:
# First transaction reads the data and sets the flag to false
BEGIN TRANSACTION
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
UPDATE flag SET stale = false
SELECT * FROM DATATABLE
COMMIT TRANSACTION
# Second transaction updates the data and sets the flag to true
BEGIN TRANSACTION
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
UPDATE data SET val = 15 WHERE ID = 10
UPDATE flag SET stale = true
COMMIT TRANSACTION
I do not have much experience with transactions or handwriting xml, so there are probably issues with this. From what I understand two serializable transactions can not be interleaved. Please correct me if I'm wrong.
Is there a way to accomplish this with only the first transaction? The process will be run rarely, but the updates to the data table will occur more frequently, so it would be nice to not lock up the data table when performing updates.
Also, is the SET TRANSACTION ISOLATION syntax specific to MS?
The stale flag will probably work, but a timestamp would be better since it provides more metadata about the age of the records which could be used to tune your queries, e.g., only pull data that is over 5 minutes old.
To address your concern about inserting a row at the same time a query is run, transactions with an appropriate isolation level will help. For row inserts, updates, and selects, at least use a transaction with an isolation level that prevents dirty reads so that no other connections can see the updated data until the transaction is committed.
If you are strongly concerned about the case where an update happens at the same time as a record pull, you could use the REPEATABLE READ or even SERIALIZABLE isolation levels, but this will slow DB access down.
Your SQLServer sampled should work. For alternate databases, Here's an example that works in PostGres:
Transaction 1
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
-- run queries that update the tables, then set last_updated column
UPDATE sometable SET last_updatee = now() WHERE id = 1;;
COMMIT;
Transaction 2
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
-- select data from tables, then set last_queried column
UPDATE sometable SET last_queried = now() WHERE id = 1;
COMMIT;
If transaction 1 starts, and then transaction 2 starts before transaction 1 has completed, transaction 2 will block during on the update, and then will throw an error when transaction 1 is committed. If transaction 2 starts first, and transaction 1 starts before that has finished, then transaction 1 will error. Your application code or process should be able to handle those errors.
Other databases use similar syntax - MySQL (with InnoDB plugin) requires you to set the isolation level before you start the transaction.
I have two transaction: T1 with SERIALIZABLE isolation level and T2 (I think - with default READ COMMITTED isolation level, but it doesn't matter).
Transaction T1 performs SELECT then WAITFOR 2 seconds then SELECT.
Transaction T2 performs UPDATE on data which T1 read.
It causes deadlock, why transaction T2 don't wait for end of T1?
When T1 has REPEATABLE READ isolation level everything is OK i.e. phantom rows occur.
I thought when I raise isolation level up to SERIALIZABLE, T2 will wait for end of T1.
This is a part of my college exercise. I have to show negative effects in two parallel transactions which have incorrect isolation level and absence of these effects with correct isolation level.
Here's the code, unfortunately names of fields are in Polish.
T1:
USE MR;
SET IMPLICIT_TRANSACTIONS OFF;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
BEGIN TRANSACTION;
-- 1. zapytanie
SELECT
www.IdSamochodu, s.Model, s.Marka, s.NrRejestracyjny, o.PESEL, o.Nazwisko, o.Imie, o.NrTelefonu
FROM
WizytyWWarsztacie www
JOIN
Samochody s
ON s.IdSamochodu = www.IdSamochodu
JOIN
Osoby o
ON o.PESEL = s.PESEL
WHERE
www.[Status] = 'gotowy_do_odbioru'
ORDER BY www.IdSamochodu ASC
;
WAITFOR DELAY '00:00:02';
-- 2. zapytanie
SELECT
u.IdSamochodu, tu.Nazwa, tu.Opis, u.Oplata
FROM
Uslugi u
JOIN
TypyUslug tu
ON tu.IdTypuUslugi = u.IdTypuUslugi
JOIN
WizytyWWarsztacie www
ON www.IdSamochodu = u.IdSamochodu AND
www.DataOd = u.DataOd
WHERE
www.[Status] = 'gotowy_do_odbioru'
ORDER BY u.IdSamochodu ASC, u.Oplata DESC
;
COMMIT;
T2:
USE MR;
SET IMPLICIT_TRANSACTIONS OFF;
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
BEGIN TRANSACTION;
UPDATE
Uslugi
SET
[Status] = 'wykonano'
WHERE
IdUslugi = 2
;
UPDATE
www
SET
www.[Status] = 'gotowy_do_odbioru'
FROM
WizytyWWarsztacie www
WHERE
www.[Status] = 'wykonywanie_usług' AND
EXISTS (
SELECT 1
FROM
Uslugi u
WHERE
u.IdSamochodu = www.IdSamochodu AND
u.DataOd = www.DataOd AND
u.[Status] = 'wykonano'
GROUP BY u.IdSamochodu, u.DataOd
HAVING COUNT(u.IdUslugi) = (
SELECT
COUNT(u2.IdUslugi)
FROM
Uslugi u2
WHERE
u2.IdSamochodu = www.IdSamochodu AND
u2.DataOd = www.DataOd
GROUP BY u2.IdSamochodu, u2.DataOd
)
)
;
COMMIT;
I use SQL Management Studio and I have each transaction in different file. I run this by clicking F5 in T1 then quickly switch to file which contains T2 and again - F5.
I read about deadlocks and locking mechanism in mssql but apparently, I haven't understand this topic yet.
Deadlock issue in SQL Server 2008 R2 (.Net 2.0 Application)
SQL Server deadlocks between select/update or multiple selects
Deadlock on SELECT/UPDATE
http://msdn.microsoft.com/en-us/library/ms173763(v=sql.105).aspx
http://www.sql-server-performance.com/2004/advanced-sql-locking/
edit
I figure out the first UPDATE statement in T2 causes the problem, why?
Troubleshooting deadlocks starts with obtaining the deadlock graph. This is an xml document that tells you the relevant bits about the transactions and resources involved. You can get it through Profiler, extended events, or event notifications (I'm sure that there are other methods, but this will do for now). Once you have the graph, examine it to see what each transaction had what type of locks on what resources. Where you go from there really depends on what's going on in the graph so I'll stop there. Bottom line: obtain the deadlock graph and mine it for details.
As an aside, to say that one or the other transaction is "causing" the deadlock is somewhat misleading. All transactions involved in the deadlock were necessary to cause the deadlock situation so neither is more at fault.
I had some problems with my SQL Managmenet Studio (Profiler didn't work) but finally I've obtained deadlock graph. This article was helpful for me.
To understand this graph I had to learn about locking mechanism and symbols.
I think here it is explained quite clearly.
Now, when I know about all these stuff, the cause of deadlock is quite obvious.
I've made sequence diagram for the described situation:
As I wrote earlier, when we get rid of the first UPDATE statement from transaction T2, deadlock does not occur.
In this situation T2 does not acquire a lock on the pk_uslugi index, thus second SELECT statement from transaction T1 will execute successfully and index pk_wizytywwarsztacie will be unlocked. After that, also T2 will be finished.
The problem could be this:
The T1 Select S-locks the row
The T2 Update U-locks the row (succeeds)
The T2 Update X-locks the row (waits, lock is queued)
T2 tries to S-lock again, but the S-lock is incompatible with the queued X-lock.
Locks in SQL Server are queued. If the head of the queue waits, everything else behind it also waits.
Actually, I'm not entirely sure that this is the cause because the same problem should occur with REPEATABLE READ. I'm still posting this idea hoping that it helps.
I ran into a similar issue where I was selecting from a list of available items and then inserting those items into a holding queue table. When I had too many concurrent requests, the select statement would return items that were also concurrently selected during another parallel request. When attempting to insert them into the holding queue table, I would receive a Unique Constraint error (because the same item couldn't go into the holding table twice).
I then tried wrapping a SERIALIZABLE transaction around the whole thing but then I ran into DEADLOCK errors because both transactions were holding onto a lock on the UC index (determined by my Deadlock graph).
I was finally able to resolve the issue by using an exclusive Row lock within the select statement.
You could try using an exclusive row lock on the table/rows in question. This would ensure that the lock on the rows within T1 will complete before T2 attempts to update the same rows.
EXAMPLE:
SELECT *
FROM Uslugi u WITH (XLOCK, ROWLOCK)
I'm not sure yet of the performance impact of this but while running load testing using multiple threads, it doesn't appear to have a negative impact.
we have a ASP.NET site with a SQL2008 DB.
I'm trying to implement transactions in our current system.
We have a method that is inserting a new row to a table (Table_A), and it works fine with transaction. If an exception is throwed, it will do the rollback.
My problem is, that another user can't do anything that invole with Table_A in the same time - probably because it's locked.
I can understand why the transaction have to lock the new row, but I don't want it to lock the whole table - forcing user_B to wait for user_A to finish.
I have tryed to set isolationlevel ReadUncommitted and Serializable on the transaction, but it didn't work.
So i'm wondering if I also need to rewrite all my select methods, so they are able to select all the rows except the new row there is in the making?
Example:
Table T has 10 rows(records?)
User A is inserting a new row into table T. This take some time.
At the same time, User B want to select the 10 rows, and dosn't care about the new row.
So I guess I somehow need to rewrite user B's select query from a "select all query" to a "select all rows that isnt bind to a transaction query". :)
Any tips is much appriciated, thanks.
You'll need to make changes to your selects, not to the transaction that is doing the insert.
You can
set isolation level read uncommitted for the connection doing the select, or
specify the WITH (NOLOCK) hint against the table that is being locked, or
specify the WITH (READPAST) hint against the table being locked
As I say, all 3 of these options are things that you'd apply to the SELECT, not the INSERT.
A final option may be to enable SNAPSHOT isolation, and change the database default to use that instead (There are probably many warnings I should include here, if the application hasn't been built/tested with snapshot isolation turned on)
Probably what you need is to set your isolation level to Read Committed Snapshot.
Although it needs more space for the tempdb it is great when you don't want your selects to lock the tables.
Beware of posible problems when using this isolation level.