I have a query that I've spent some time optimizing, it will normally run in less than a second. Occasionally, though, it will take a minute or two to run which causes the calling application to throw a timeout error.
It is just a report query, it's not updating. I'm using SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED.
Question: How can I tell what is blocking this query and get it fixed?
When you notice a long-running query, try this in a new query window:
exec sp_who2
Peruse the BlkBy column to see if you're getting blocked by something, then find its SPID in the left column. This will let you know the SPID of the blocker, as well as some general context about the cause of the block. If you think it advisable, you could run KILL to stop the blocking SPID, but keep in mind it will disrupt whomever initiated that query. First, you might want to check with whomever's Login or HostName is blocking.
Alternatively, if you have a cached execution plan for an expensive query, it may have expired.
If there's any particular pattern to the performance dive, that will be a helpful clue, but if it seems random, then I'd keep an eye on sp_who2 so you can fire that off when you notice it happening.
Related
In what situation would a simple update statement
UPDATE [BasicUserTable]
SET [DateTimeCol] = '9/6/2022'
WHERE [UniqueIntPKCol] = 123
take 1m 30s to complete, AND THEN all subsequent updates using the same statement and lines of code (except for id and datetime), execute in < 100 ms?
The table has less than 10,000 records, standard int auto incrementing primary key.
Background: our app was timing out (standard 30 sec timeout) while it waited for SQL Server to execute the statement above. We manually tried the statement using SSMS on the same server, and it took ~1m 30s to execute.
Immediately afterward, all other attempts to run the same code were blazing fast as expected. We can't walk past this issue without knowing the real reason that it happened, so we can prevent it in the future.
After looking at logs, there were no apparent blocking locks on the records, nor code that could intervene an cause issue.
SQL Logs did not have any errors
Microsoft.EntityFrameworkCore.DbUpdateException
Inner exception: Microsoft.Data.SqlClient.SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Has anyone run into this before, or do you have a plausible working theory? (index rebuild, caching, etc.)
A lock wait is the only thing I can imagine that would cause this.
After looking at logs, there were no apparent blocking locks
Lock waits don't cause any logging. You might see logs if you configure the blocked process report, but it's not on by default.
Turning on the Query Store can help by helping track query resource utilization and waits.
Although extremely unlikely here, file growth can also cause sporadic delays, as the statement that needs the additional log file or data file space has to wait for the file to be resized.
I am doing a concurrency test in sql server 2019, I have SQLTest tool that runs concurrent queries, in my test I am using one single SELECT query (star schema) and on SSMS I have while loop that updates fact table records. while running both process I am seeing some of the threads/queries cancelled because of deadlock, which is expected but the option that I am looking or is there a possibility to add a wait time on my select before deadlock? in other words how much time SQL server waits before it creates deadlock error.
In this case I know constant updates are happening but we know that updates are for a fewer seconds so if SQL server can wait for some seconds before creating deadlock.
any suggestions or thoughts ?
I would suggest changing up your testing strategy a little.
Within your test harness, I would SET DEADLOCK_PRIORITY LOW;, so that when a deadlock is detected, your testing process voluntarily takes one for the team, allows itself to become the deadlock victim, and allows the conflicting process to continue.
Then, wrap the testing script in a TRY...CATCH. In the CATCH clause, check to see if the cause of the error is a deadlock (error code 1205), and if it is, retry your test. It's probably a good idea to also build a incremental counter into that so that you don't end up in an infinite deadlock loop.
is there a possibility to add a wait time on my select before deadlock?
No. It would make no sense.
A deadlock is defined as a dead end of locking, which will not, under no circumstrances, be fixed by simply waiting. One of the sides has to cancel.
I.e.
Tx1 has lock on table a, waits for lock on table b
Tx2 has lock on table b, waitf for lock on table a
Normally SQL Server waits (timeout) and cancels. In this case the deadlock detection steps up and realizes that no, unless a side is thrown out there is no way this gets resolved, so - it cancels one side. There is no waiting, because this is actually a programming bug. No joke.
Up there, Tx2 should FIRST ask for a lock on table a. It is good practice to get locks in a transaction in a defined order so this does not happen.
On a client is being raised the error "Timeout" to trigger some commands against the database.
My first test option for correction is to increase the CommandTimeout to 99999 ... but I am afraid that this treatment generates further problems.
Have experienced it ...?
I wonder if my question is relevant, and/or if there is another option more robust and elegant correction.
You are correct to assume that upping the timeout is not the correct approach. Typically, I look for log running queries that are running around the timeouts. They will typically stand out in the areas of duration and reads.
Then I'll work to reduce the query run time using this method:
https://www.simple-talk.com/sql/performance/simple-query-tuning-with-statistics-io-and-execution-plans/
If it's a report causing issues and you can't get it running faster, you may need to start thinking about setting up a reporting database.
CommandTimeout is a time, that the client is waiting for a response from server. If the query is run in the main VCL thread then the whole application is "frozen" and might be marked "not responding" by Windows. So, would you expect your users to wait at frozen app for 99999 sec?
Generally, leave the Timeout values at default and rather concentrate on tunning the queries as Sam suggests. If you happen to have long running queries (ie. some background data movement, calculations etc in Stored Procedures) set the CommandTimeout to 0 (=INFINITE) but run them in a separate thread.
In my client application I have a method like this (in practice it's more complex, but I've left the main part):
public void btnUpdate_Click(...)
{
...
dataAdapter.Update(...);
...
dataAdapter.Fill(...); // here I got exception one time
}
The exception I found in logs says "Deadlock found when trying to get lock; try restarting transaction". I met this exception only time, so it wasn't repeated.
As I understand, DataAdapter.Fill() method executes only select query. I don't make an explicit transaction and I have autocommit enabled.
So how can I get dead lock on a simple select query which is not a part of bigger transaction?
As I understand, to get a dead lock, two transactions should wait for each other. How is that possible with a single select not inside a transaction? Maybe it's a bug in MySql?
Thank you in advance.
You are right it takes two transactions to make a deadlock. That is to say, No statement or statements within a single transaction can deadlock with other statements within the same transaction.
But it only take one transaction to notice a report of a deadlock. How do you know that the transaction you are seeing the deadlock reported in is the only transaction being executed in the database? Isn't there other activity going on in this database?
Also. your statement "I don't make an explicit transaction", and "... which is not a part of bigger transaction" implies that you do not understand that every SQL statement executed is always in an implicit transaction, even if you do not explicitly start one.
Most databases have reporting mechanisms specifically designed to track, report and/or log instances of deadlocks for diagnostic purposes. In SQL server there is a trace flag that causes a log entry with much detail about each deadlock that occurs, including details about each of the two transactions involved, like what sql statements were being executed, what objects in the database were being locked, and why the lock could not be obtained. I'd guess mySQL has similar disgnostic tool. Find out what it is and turn it on so that the next time this occurs you can look and find out exactly what happened.
You can deadlock a simple SELECT against other statements, like an UPDATE. On my blog I have an example explaining a deadlock between two well tunned statements: Read/Write deadlock. While the example is SQL Server specific, the principle is generic. I don't have enough knowledge of MySQL to claim this is necessarily the case or not, specially in the light of various engines MySQL can deploy, but none the less a simple SELECT can be the victim of a deadlock.
I haven't looked into how MySQL transaction works, but this is based on how MSSQL transactions work:
If you are not using a transaction, each query has a transaction by itself. Otherwise you would get a mess every time an update failed in the middle.
The reason for the deadlock might be lock escalation. The database tries to lock as little as possible for each query, so it starts out by locking only the single rows affected. When most of the rows in a page is locked by the query it might decide that escalating the lock into locking the entire page would be better, which may have the side effect of locking some rows not otherwise affected by the query.
If a select query and an update query are trying to escalate locks on the same table, they may cause a deadlock eventhough only a single table is involved.
I agree that in this particular issue this is unlikely to be the issue but this is supplemental to the other answers in terms of limiting their scope, recorded for posterity in case someone finds it useful.
MySQL can in rare cases have single statements periodically deadlock against themselves. This seems to happen particularly on bulk inserts and the issues are almost certainly a deadlock between different threads relating to the operation. I would expect bulk updates to have the same problem. In the past when faced with this sort of issue I have generally just cut down on the number of rows being inserted (or updated) in a single statement. You won't usually get a deadlock when trying to obtain the lock in this case but other messages.
A colleague of mine and I were discussing similar problems in MS SQL Server (so this is not unique to MySQL!) and he pointed out that the solution there is to tell the server not to parallelize the insert or update. The problems here are spinlock-related deadlocks, not logical lock deadlocks in the RDBMS.
Do you know of any ORM tool that offers deadlock recovery? I know deadlocks are a bad thing but sometimes any system will suffer from it given the right amount of load. In Sql Server, the deadlock message says "Rerun the transaction" so I would suspect that rerunning a deadlock statement is a desirable feature on ORM's.
I don't know of any special ORM tool support for automatically rerunning transactions that failed because of deadlocks. However I don't think that a ORM makes dealing with locking/deadlocking issues very different. Firstly, you should analyze the root cause for your deadlocks, then redesign your transactions and queries in a way that deadlocks are avoided or at least reduced. There are lots of options for improvement, like choosing the right isolation level for (parts) of your transactions, using lock hints etc. This depends much more on your database system then on your ORM. Of course it helps if your ORM allows you to use stored procedures for some fine-tuned command etc.
If this doesn't help to avoid deadlocks completely, or you don't have the time to implement and test the real fix now, of course you could simply place a try/catch around your save/commit/persist or whatever call, check catched exceptions if they indicate that the failed transaction is a "deadlock victim", and then simply recall save/commit/persist after a few seconds sleeping. Waiting a few seconds is a good idea since deadlocks are often an indication that there is a temporary peak of transactions competing for the same resources, and rerunning the same transaction quickly again and again would probably make things even worse.
For the same reason you probably would wont to make sure that you only try once to rerun the same transaction.
In a real world scenario we once implemented this kind of workaround, and about 80% of the "deadlock victims" succeeded on the second go. But I strongly recommend to digg deeper to fix the actual reason for the deadlocking, because these problems usually increase exponentially with the number of users. Hope that helps.
Deadlocks are to be expected, and SQL Server seems to be worse off in this front than other database servers. First, you should try to minimize your deadlocks. Try using the SQL Server Profiler to figure out why its happening and what you can do about it. Next, configure your ORM to not read after making an update in the same transaction, if possible. Finally, after you've done that, if you happen to use Spring and Hibernate together, you can put in an interceptor to watch for this situation. Extend MethodInterceptor and place it in your Spring bean under interceptorNames. When the interceptor is run, use invocation.proceed() to execute the transaction. Catch any exceptions, and define a number of times you want to retry.
An o/r mapper can't detect this, as the deadlock is always occuring inside the DBMS, which could be caused by locks set by other threads or other apps even.
To be sure a piece of code doesn't create a deadlock, always use these rules:
- do fetching outside the transaction. So first fetch, then perform processing then perform DML statements like insert, delete and update
- every action inside a method or series of methods which contain / work with a transaction have to use the same connection to the database. This is required because for example write locks are ignored by statements executed over the same connection (as that same connection set the locks ;)).
Often, deadlocks occur because either code fetches data inside a transaction which causes a NEW connection to be opened (which has to wait for locks) or uses different connections for the statements in a transaction.
I had a quick look (no doubt you have too) and couldn't find anything suggesting that hibernate at least offers this. This is probably because ORMs consider this outside of the scope of the problem they are trying to solve.
If you are having issues with deadlocks certainly follow some of the suggestions posted here to try and resolve them. After that you just need to make sure all your database access code gets wrapped with something which can detect a deadlock and retry the transaction.
One system I worked on was based on “commands” that were then committed to the database when the user pressed save, it worked like this:
While(true)
start a database transaction
Foreach command to process
read data the command need into objects
update the object by calling the command.run method
EndForeach
Save the objects to the database
If not deadlock
commit the database transaction
we are done
Else
abort the database transaction
log deadlock and try again
EndIf
EndWhile
You may be able to do something like with any ORM; we used an in house data access system, as ORM were too new at the time.
We run the commands outside of a transaction while the user was interacting with the system. Then rerun them as above (when you use did a "save") to cope with changes other people have made. As we already had a good ideal of the rows the command would change, we could even use locking hints or “select for update” to take out all the write locks we needed at the start of the transaction. (We shorted the set of rows to be updated to reduce the number of deadlocks even more)