Problems getting a reliable unpooled connection to SQL Server 2008r2 - sql

Question

I am having trouble getting my application to reliably connect to a mirrored SQL Server 2008r2. I've created a tool to help diagnose the problem and found some interesting things.
The connection string I'm using is
Data Source=IADB01;Failover Partner=IBDB01;Database=XXX;
Integrated Security=SSPI;Connection Timeout=150;Pooling=false;
It seems to be the non-pooled connection string. If I leave pooling on, I got no problems at all and I can generate several hundred connections a second with no problems. But this causes problems with the automatic failover. With pooling turned off, I get between 0 and 100 or so successful connections and then a long delay before finally this SqlException:
System.Data.SqlClient.SqlException (0x80131904): Login failed for user 'INTERNET\71260'.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
at System.Data.SqlClient.SqlInternalConnectionTds.LoginWithFailover(Boolean useFailoverHost, ServerInfo primaryServerInfo, String failoverHost, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at DatabaseConnectionTest.Form1.MakeConnection()
ClientConnectionId:dc55aea9-ea36-4812-b628-3dc7f096f0e4
I'm can't figure out this as I don't know why the exception is Login failed. If I start connecting again, I get anywhere between 0 and roughly 100 or so successful connections before the Login Failed again. I can't see any pattern as to how many successful connections I get before the failure either, but it's usually only 5-10 successes though. We've shifted the client to be on the same network switch as the database server to avoid problems with firewalls etc and it hasn't changed anything. The DC is on the same switch too.
The successful connections are taking less than a second. When the a connection fails it only does so after at least 5 seconds sometimes up to 20 seconds. I'm convinced the problem is somewhere else, but I need ideas as to where to look.
I can see in the event logs SQL Server making it's calls, but the entries are always the same. When the failure happens I've got the same set of calls in the event logs. I want to use a profiler on SQL Server to see but as yet I haven't been able to due to not having sysadmin.
So, as far as the stack trace goes, it looks like it's made it as far as methods Run and TryRun. How should I interpret this? The connection has actually been established but then the command is failing? Thanks for the help. I can post more if needed.

Answer 1

Sorted. This is a known problem caused by TCP Chimney Offloading. The solution is to turn off the offloading.