I have been having deadlock issues. I've been on working some retry approaches. My retry code is currently just a 'for' statement that tries 5 times. I understand i need to use the 'Evit' nhibernate method to clear the session. I am using a session factory and use a transaction for each request.
In the below example if i experience a deadlock on the first retry will the orderNote property remain the same on the second retry?
private ActionResult OrderDetails(int id)
{
var order = _orderRepository.Get(id);
order.OrderNote = "will this text remain";
Retry.Times(5).Do(() => _orderRepository.Update(order));
return View();
}
Edit
1) Finding it hard to trace the cause. I'm getting about 10 locks a day all over my application. Just set up a profiler. Are there any other useful methods for tracing
http://msdn.microsoft.com/en-us/library/ms190465.aspx
I think the main issue is that i'm using auto increament. I'm in the process of moving to hilo.
2) Using a different transation mode. I'm not defining any at the moment. What is recommended.
5) Long running operations. Yes i do. And i think because i'm using auto increament lazy loading is ignored. Does that sound correct?
In my opinion your code is trying to fix the symptoms instead of the cause.
You will be better off doing some of the following things:
Find out why you are getting deadlocks and fix the core issue
Use a different transaction mode to read past locks
Look at delegating the update into a queue structure to be background processed
Understand the update execution plan and perhaps add indexing to speed up queries
Do you have any "long" running operations in your Controller action which is keeping the transaction open for longer than it should be?
Even if the operation did deadlock, why don't you return an friendly error back to the calling page and let them manually retry.
Update:
1.) Useful methods for tracing
I have used this method for tracing deadlocks which should give you an idea of the resources which are in contention: Tracing Deadlocks
You can also look at the concurreny models available to you: NHibernate Concurrency
2.) Transaction Isolation Levels
Depending on your DB this Question has some useful information: Transaction Isolation Mode
3.) Long Running Operations
I have to use Identity Columns as my primary keys in NHibernate and I don't think these are going to be source of your problem in an update scenario as the Id/PK is already set by this point. Try to minimise the long running operations which will shorten the amount of time your transaction is held open.
Related
Recently we see a huge amount of errors that pertain to Serializable isolation violation on table we have some base tables that forms our core data and we extract values from these tables to run our business logic in Lambda's.
Scenario :
Lambda 1 : Runs every 15 mins and gets the latest data from the source (RDBMS) into Redshift which forms our base tables (Does a DELETE and INSERT)
Lambda 2 : Triggered after successful run of the above Lambda and this is where the business logic is written and is normal SELECT statements
Lambda 3 : Triggered every 15 mins and is also run using the base tables and has only SELECT statements
When the Lambda 1 is triggered for its next run at instances we see that it fails with the Serializable isolation violation error.
Based on most of the posts putting a LOCK on the table might solve the issue but will increase the wait time for the other queries to run longer than expected and due to the constraint of the Lambda it will timeout after 15 mins which is not ideal. And I did see posts that stated putting a LOCK didn't entirely solve it too so skeptical to use to.
So something that struck me is would creating a VIEW on top of the base table and use the view in all the SELECT statements would that help here, if someone has any insights on this would really be helpful.
So the issue is with the locks each transaction is creating and being unable to determine the correct order the locks need to be resolved in. See: https://aws.amazon.com/premiumsupport/knowledge-center/redshift-serializable-isolation/
Now your description doesn't have enough writes in flight as stated to see how you are getting this from one pass of the Lambdas runs. So either this description isn't complete (multiple Lambdas updating tables) or the issue is coming between runs of Lambdas. A possibility is that the transaction aren't being closed and the Lambda invocations are having locks that cannot be resolved. Do you have COMMITs closing transactions? More info is needed to know which.
You can inspect pg_locks between Lambdas to see what locks are left around. The XID is the transaction that has the lock. I'd guess you have many more open transactions than you expect. Are your Lambda sessions in autocommit mode? Are you updating tables from multiple sessions? Are you COMMITting your changes? Are you reusing scratch tables?
Adding an explicit LOCK to serialize things can work if you know why / what tables are causing the serialization issue. This is also not likely the best solution and a better approach will likely be apparent when the issue is understood.
Adding a view to the mix won't resolve the issue (though it might move it if it changes the timing of events).
We are trying to implement retry logic to recover from transient errors in Azure environment.
We are using long-running sessions to keep track and commit the whole bunch of changes at the end of application transaction (which may spread over several web-requests). Along the way we need to get additional data from database. Our main problem is that we can't easily recover from db error because we can't "replay" all user actions.
So far we used straightforward recovery algorithm:
Try to perform operation in long-running session
In case of error, close the session, open a new one and merge entities into it
Retry the operation
It's very expensive approach in terms of time (merge is really long for big entity hierarchies). So we'd like to optimize things a little.
We'd like to perform query operations in separate session (to keep long running one untouched and safe) and on success, merge results back to the long-running session. Retry is relatively simple here - we just need to open new session and run query once more. However, with this approach we have an issue with initializing lazy properties/collections:
If we do this in separate session, we need to merge results back (a lot of entities) but merge could fail and break the long-running session
We tried different ways of "moving" original entity to different session, loading details and returning it back, but without success (evict, replicate, etc.)
There is known statement that session should be discarded in case of exception. However, the example shows write operation. Is it still true for read ones? I mean if I guarantee that no data is written back to the database, can I reuse the same session to run query again?
Do you have any other suggestions about retry logic with long-running sessions?
IMO there's no way to solve your issue. It's gonna take a lot of time to commit everything or you will have to do a lot of work to break it up into smaller sessions and handle every error that can occur while merging.
To answer your question about using the session after an exception: you cannot trust ANYTHING anymore inside this session, not even loaded entities.
Read this paragraph from Ayende's article about building a simple todo app with a recoveryplan in case of an exception in the session:
Then there is the problem of error handling. If you get an exception
(such as StaleObjectStateException, because of concurrency conflict),
your session and its loaded entities are toast, because with
NHibernate, an exception thrown from a session moves that session into
an undefined state. You can no longer use that session or any loaded
entities. If you have only a single global session, it means that you
probably need to restart the application, which is probably not a good
idea.
Looking through my logs, I can see that my app is vulnerable to deadlocks. They are occurring in many parts of my application.
1) Is there way to replicate this issue. ie: I have only seen this in logs.
2) What is the best/simplest way to retry if the transaction is locked
3) If I wrapped the call in a try/catch. What would the exception type be.
There is a lot written about the issue. I concluded the best option is to try and shorten the transactions as much as possible. Should I change the isolation levels?
Finding Deadlocks
deadlocks are very hard to find. If you know why they occur, you may reproduce it in integration tests. In real environments you can use Profiler to observe dead locks. It shows a graph which displays how the deadlock is formed.
Retry
You should actually throw away the transaction and start again. The NHibernate session is out of synch after any database exception.
We have a delay before restarting to avoid more stress to the database. It waits for a certain time containing a random number, to avoid that the parallel transactions are synchronizing again.
Avoiding Deadlocks
Reducing Lock Time
If you are using Sql Server, it is very vulnerable to dead locks because of its pessimistic locking mechanism (in contrast to Oracle databases). The newer Snapshot isolation level is something similar to what Oracle is doing and may fix the problem to some degree, but I never used until now so I can't say much about it.
NHibernate fixes the problem as far as possible by caching changes to persistent data and store it at the end of a transaction. But there are some limits and some ways to break it.
Using identity ("auto numbers") as primary keys is probably the most famous mistake. It forces NH to insert entities when they are put into the session which produces a lock of the whole table (in SQL Server).
More complicated to fix is the flushing problem. NH needs to flush changes before executing queries, to ensure consistency. You can get around this by setting FlushMode to Never, which may cause consistency problems, so only do it when you exactly know what you do. The best solution is to only use Get or Load or navigate to properties of a root entity instead of performing queries in the middle of a transaction.
By doing all this, NH is able to wait for any Insert, Update and Delete command to the database until the end of the transaction. The reduces lock time a lot and therefore it also reduces the risk of dead locks.
General Rules To Avoid Deadlock
The general rules to avoid deadlocks also apply when using NHibernate. Most important: lock resources in a certain order, lock resources not on by one but all at the beginning. The latter is contradictory to what I said above to reduce lock time. It would mean that you lock resources at the beginning of a transaction to make other transactions wait until it is finished. This may reduce deadlocks but also reduces parallel execution.
This is the solution that we opted to use in legacy system where we could not fix the root cause of these deadlocks as it would mean rewriting a lot of existing and poorly documented code. The system was using DataSets and ADO.NET classes, so if you intent to use NHibernate I fear you would have to research its internals and/or develop your own extension or for if existing functionality is not available for that.
1) If the code is prone to deadlocks they should start appearing at sufficient database load. You need many simultaneous connections working with the same tables using the problematic procedures.
It is difficult to reproduce deadlocks in the exact places you want, but if you want general deadlocks for your retry procedure testing do simultaneous reads/inserts into same tables from 10+ threads with differing access order (e.g. table A then B in some of them, table B and then A in others) with small delays and you get one soon.
2) You need to retry the entire code fragment that works with the transaction and retry data initialization too. Meaning, if you are filling datasets within transaction you have to clear them at the beginning of retryable code block.
3) It is .Numer=1205 of SqlException. In general, you can also retry on timeout and network errors:
switch (sqlEx.Number)
{
case 1205:
{
DebugLog("DEADLOCK!");
canRetry = true;
break;
}
case -2:
case -2147217871:
{
DebugLog("TIMEOUT!");
canRetry = true;
break;
}
case 11:
{
DebugLog("NETWORK ERROR!");
canRetry = true;
break;
}
default:
{
DebugLog(string.Format("SQL ERROR: {0}", sqlEx.Number));
break;
}
}
In my experience when retrying on deadlock it is best to discard the connection from the pool with SqlConnection.ClearPool(connection) because it might not be reset properly for the next time.
This is a common question, but the explanations found so far and observed behaviour are some way apart.
We have want the following nHibernate strategy in our MVC website:
A SessionScope for the request (to track changes)
An ActiveRecord.TransactonScope to wrap our inserts only (to enable rollback/commit of batch)
Selects to be outside a Transaction (to reduce extent of locks)
Delayed Flush of inserts (so that our insert/updates occur as a UoW at end of session)
Now currently we:
Don't get the implied transaction from the SessionScope (with FlushAction Auto or Never)
If we use ActiveRecord.TransactionScope there is no delayed flush and any contained selects are also caught up in a long-running transaction.
I'm wondering if it's because we have an old version of nHibernate (it was from trunk very near 2.0).
We just can't get the expected nHibernate behaviour, and performance sucks (using NHProf and SqlProfiler to monitor db locks).
Here's what we have tried since:
Written our own TransactionScope (inherits from ITransactionScope) that:
Opens a ActiveRecord.TransactionScope on the Commit, not in the ctor (delays transaction until needed)
Opens a 'SessionScope' in the ctor if none are available (as a guard)
Converted our ids to Guid from identity
This stopped the auto flush of insert/update outside of the Transaction (!)
Now we have the following application behaviour:
Request from MVC
SELECTs needed by services are fired, all outside a transaction
Repository.Add calls do not hit the db until scope.Commit is called in our Controllers
All INSERTs / UPDATEs occur wrapped inside a transaction as an atomic unit, with no SELECTs contained.
... But for some reason nHProf now != sqlProfiler (selects seems to happen in the db before nHProf reports it).
NOTE
Before I get flamed I realise the issues here, and know that the SELECTs aren't in the Transaction. That's the design. Some of our operations will contain the SELECTs (we now have a couple of our own TransactionScope implementations) in serialised transactions. The vast majority of our code does not need up-to-the-minute live data, and we have serialised workloads with individual operators.
ALSO
If anyone knows how to get an identity column (non-PK) refreshed post-insert without a need to manually refresh the entity, and in particular by using ActiveRecord markup (I think it's possible in nHibernate mapping files using a 'generated' attribute) please let me know!!
Do you know of any ORM tool that offers deadlock recovery? I know deadlocks are a bad thing but sometimes any system will suffer from it given the right amount of load. In Sql Server, the deadlock message says "Rerun the transaction" so I would suspect that rerunning a deadlock statement is a desirable feature on ORM's.
I don't know of any special ORM tool support for automatically rerunning transactions that failed because of deadlocks. However I don't think that a ORM makes dealing with locking/deadlocking issues very different. Firstly, you should analyze the root cause for your deadlocks, then redesign your transactions and queries in a way that deadlocks are avoided or at least reduced. There are lots of options for improvement, like choosing the right isolation level for (parts) of your transactions, using lock hints etc. This depends much more on your database system then on your ORM. Of course it helps if your ORM allows you to use stored procedures for some fine-tuned command etc.
If this doesn't help to avoid deadlocks completely, or you don't have the time to implement and test the real fix now, of course you could simply place a try/catch around your save/commit/persist or whatever call, check catched exceptions if they indicate that the failed transaction is a "deadlock victim", and then simply recall save/commit/persist after a few seconds sleeping. Waiting a few seconds is a good idea since deadlocks are often an indication that there is a temporary peak of transactions competing for the same resources, and rerunning the same transaction quickly again and again would probably make things even worse.
For the same reason you probably would wont to make sure that you only try once to rerun the same transaction.
In a real world scenario we once implemented this kind of workaround, and about 80% of the "deadlock victims" succeeded on the second go. But I strongly recommend to digg deeper to fix the actual reason for the deadlocking, because these problems usually increase exponentially with the number of users. Hope that helps.
Deadlocks are to be expected, and SQL Server seems to be worse off in this front than other database servers. First, you should try to minimize your deadlocks. Try using the SQL Server Profiler to figure out why its happening and what you can do about it. Next, configure your ORM to not read after making an update in the same transaction, if possible. Finally, after you've done that, if you happen to use Spring and Hibernate together, you can put in an interceptor to watch for this situation. Extend MethodInterceptor and place it in your Spring bean under interceptorNames. When the interceptor is run, use invocation.proceed() to execute the transaction. Catch any exceptions, and define a number of times you want to retry.
An o/r mapper can't detect this, as the deadlock is always occuring inside the DBMS, which could be caused by locks set by other threads or other apps even.
To be sure a piece of code doesn't create a deadlock, always use these rules:
- do fetching outside the transaction. So first fetch, then perform processing then perform DML statements like insert, delete and update
- every action inside a method or series of methods which contain / work with a transaction have to use the same connection to the database. This is required because for example write locks are ignored by statements executed over the same connection (as that same connection set the locks ;)).
Often, deadlocks occur because either code fetches data inside a transaction which causes a NEW connection to be opened (which has to wait for locks) or uses different connections for the statements in a transaction.
I had a quick look (no doubt you have too) and couldn't find anything suggesting that hibernate at least offers this. This is probably because ORMs consider this outside of the scope of the problem they are trying to solve.
If you are having issues with deadlocks certainly follow some of the suggestions posted here to try and resolve them. After that you just need to make sure all your database access code gets wrapped with something which can detect a deadlock and retry the transaction.
One system I worked on was based on “commands” that were then committed to the database when the user pressed save, it worked like this:
While(true)
start a database transaction
Foreach command to process
read data the command need into objects
update the object by calling the command.run method
EndForeach
Save the objects to the database
If not deadlock
commit the database transaction
we are done
Else
abort the database transaction
log deadlock and try again
EndIf
EndWhile
You may be able to do something like with any ORM; we used an in house data access system, as ORM were too new at the time.
We run the commands outside of a transaction while the user was interacting with the system. Then rerun them as above (when you use did a "save") to cope with changes other people have made. As we already had a good ideal of the rows the command would change, we could even use locking hints or “select for update” to take out all the write locks we needed at the start of the transaction. (We shorted the set of rows to be updated to reduce the number of deadlocks even more)