RetryPolicy.Retrying event not firing during Transient Error Handling for SQL Azure? - vb.net

Question

I've got website hosted in azure (as a web service) that is using sql azure as it's backend.
My error logs have been heavily populated with what looks to be transient network and sql connection errors.
As such I've implemented the Enterprise Library Transient Error Handling Block. In tests it would seem that its operating correctly.
The issue I'm encountering is that I would like to log instances where this retry logic is occurring. From the documentation RetryPolicy.Retrying looks to be the event I'm after but in tests it doesn't fire. There's plenty of examples in C# that follow the following pattern to fire this event:
var retryPolicy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>(retryStrategy);
// Receive notifications about retries.
retryPolicy.Retrying += (sender, args) =>
{
// Log details of the retry.
var msg = String.Format("Retry - Count:{0}, Delay:{1}, Exception:{2}",
args.CurrentRetryCount, args.Delay, args.LastException);
Trace.WriteLine(msg, "Information");
};
I thought I'd adapted this correctly but in short, what's wrong with the code below?!
Private RetryManager As RetryManager
Private WithEvents RetryPolicy As RetryPolicy
Private Sub RetryPolicy_Retrying(ByVal sender As Object, ByVal args As RetryingEventArgs)
' Log details of the retry.
Dim msg = String.Format("Retry - Count:{0}, Delay:{1}, Exception:{2}", args.CurrentRetryCount, args.Delay, args.LastException)
Trace.TraceInformation(msg)
End Sub
Private Sub SetupRetryPolicy()
'If its already set then lets not do it again
If RetryPolicy Is Nothing Then
RetryManager = EnterpriseLibraryContainer.Current.GetInstance(Of RetryManager)()
RetryPolicy = RetryManager.GetRetryPolicy(Of SqlAzureTransientErrorDetectionStrategy)("Exponential Backoff Retry Strategy")
' connect sub as handler to event when retry occurs
AddHandler RetryPolicy.Retrying, AddressOf RetryPolicy_Retrying
End If
End Sub
Public Sub ExecuteAndDoStuff(ByVal connString As String, ByVal cmdText As String)
SetupRetryPolicy()
'get a connection with retry
Using conn As New ReliableSqlConnection(connString, RetryPolicy, RetryPolicy)
conn.Open()
Using cmd As SqlCommand = conn.CreateCommand
Try
cmd.CommandText = cmdText
' this might be overkill, do I need to pass the retry policy in again for the command?
Dim dr As SqlDataReader = cmd.ExecuteReaderWithRetry(RetryPolicy, RetryPolicy)
'... do something with this datareader
Catch ex As Exception
'log error
Trace.TraceError("Query failed to execute despite retry logic: " & ex.ToString)
'continue to throw the error (picked up higher up the chain)
Throw ex
End Try
End Using
End Using
End Sub
I'm totally new to at least half of what's going on in this block of code but before anyone thows a rtfm at me - I tried!

Answer 1

It's hard to tell if anything is wrong in your code; it could be that there are no transient errors being detected at all. How are you making the determination that there are transient errors? The first thing I would do is to make sure you have a repeatable way of creating a transient error.
The way I would setup the test is to have a 1GB database in Azure SQL Database, fill it up with data up until it has reached its storage limit, and then try to add more data (which will generate a transient error everytime).
There are two things to keep in mind with Azure SQL transient errors:
1) They are very hard to test because many of them depend on variables that are outside of your control; one of the easiest transient error to replicate is out of space (the suggestion above)
2) There are a few other types of errors that can be fired, such as router swicthing conditions in Azure, that are not considered transient; for example IOException errors are not captured by the SQL Transient Strategy. As a result, you either need to account for those kinds of errors seperately, or customize the strategy to include these errors. Your catch block should trap these errors in your current implementation.