How do you resolve ODATA returning an open bracket of array and 'StatusCode could not be set' error - sql

I am working on a .NET ODATA API that exposes data from SQL Views and is consumed by a 'Salesforce Connect' Connector.
If I execute a query with an ODATA filter applied my query works fine but when I test some filters i get the following output:
Please note the 'open bracket response'. The response is 200 so I get an OK. But in my .NET logging I see the following error in my console:
fail: BARCO.ODataApi.Api.Web.AppExceptionHandlerMiddleware[0]
Request resulted in SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite, String method)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.ExecuteReader()
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.InitializeReader(Enumerator enumerator)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.<>c.<MoveNext>b__19_0(DbContext _, Enumerator enumerator)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.MoveNext()
at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteResourceSetAsync(IEnumerable enumerable, IEdmTypeReference resourceSetType, ODataWriter writer, ODataSerializerContext writeContext)
at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteObjectInlineAsync(Object graph, IEdmTypeReference expectedType, ODataWriter writer, ODataSerializerContext writeContext)
at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteObjectAsync(Object graph, Type type, ODataMessageWriter messageWriter, ODataSerializerContext writeContext)
at Microsoft.AspNetCore.OData.Formatter.ODataOutputFormatterHelper.WriteToStreamAsync(Type type, Object value, IEdmModel model, ODataVersion version, Uri baseAddress, MediaTypeHeaderValue contentType, HttpRequest request, IHeaderDictionary requestHeaders, IODataSerializerProvider serializerProvider)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultAsync>g__Logged|22_0(ResourceInvoker invoker, IActionResult result)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at BARCO.ODataApi.Api.Web.UnitOfWorkMiddleware.Invoke(HttpContext context) in C:\Github\BARCO\BarcoODataApi\BarcoODataApi\src\BARCO.ODataApi.Api\Web\UnitOfWorkMiddleWare.cs:line 46
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at BARCO.ODataApi.Api.Web.ClientConfigurationMiddleware.Invoke(HttpContext context) in C:\Github\BARCO\BarcoODataApi\BarcoODataApi\src\BARCO.ODataApi.Api\Web\ClientConfigurationMiddleware.cs:line 66
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at BARCO.ODataApi.Api.Web.AppExceptionHandlerMiddleware.Invoke(HttpContext context) in C:\Github\BARCO\BarcoODataApi\BarcoODataApi\src\BARCO.ODataApi.Api\Web\AppExceptionHandlerMiddleware.cs:line 37
ClientConnectionId:6b3c22ec-a535-4e66-99df-0cf41053a3f9
Error Number:-2,State:0,Class:11
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HMFRFUHVRJMQ", Request id "0HMFRFUHVRJMQ:00000002": An unhandled exception was thrown by the application.
System.InvalidOperationException: StatusCode cannot be set because the response has already started.
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ThrowResponseAlreadyStartedException(String value)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.set_StatusCode(Int32 value)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.Microsoft.AspNetCore.Http.Features.IHttpResponseFeature.set_StatusCode(Int32 value)
at Microsoft.AspNetCore.Http.DefaultHttpResponse.set_StatusCode(Int32 value)
at BARCO.ODataApi.Api.Web.AppExceptionHandlerMiddleware.SetResponseExceptionStatusCode(HttpContext context, Exception exception) in C:\Github\BARCO\BarcoODataApi\BarcoODataApi\src\BARCO.ODataApi.Api\Web\AppExceptionHandlerMiddleware.cs:line 101
at BARCO.ODataApi.Api.Web.AppExceptionHandlerMiddleware.Invoke(HttpContext context) in C:\Github\BARCO\BarcoODataApi\BarcoODataApi\src\BARCO.ODataApi.Api\Web\AppExceptionHandlerMiddleware.cs:line 47
at Microsoft.AspNetCore.Watch.BrowserRefresh.BrowserRefreshMiddleware.InvokeAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
Is there a way to provide more detailed logging so I can understand what exactly is going wrong in my application or where the error is? Since receiving a response of "value":"[" is really weird and I have never seen this kind of response. I searched online already but there does not seem to be any questions out there with the same issue concerning this response.
One of my first questions i posted so please let me know if there is a need for additional information on my question.
Thanks in advance!
EDIT:
Salesforce Connect Backend Query translation:
SELECT TOP(#__TypedProperty_2) ...
FROM ... AS [v]
WHERE CASE
WHEN ( #__TypedProperty_0 LIKE N'' )
OR ( Charindex(#__TypedProperty_0, [v].[assetid]) > 0 ) THEN
Cast(1 AS BIT)
ELSE Cast(0 AS BIT)
END = #__TypedProperty_1
ORDER BY ...

Related

Azure SQL - System.ComponentModel.Win32Exception: The wait operation timed out

I have an MVC Azure Web App, using an Azure SQL Database.
I've just finished developing a new version on my local machine, and released it to my Azure test environment, but it's behaving a bit strangely: some parts of the system work, but one particular part (which hasn't changed for this release) is giving me the following error:
System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTdsWithTransparentParameterEncryption(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest, Task describeParameterEncryptionTask)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func`3 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed)
at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext)
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQueryInternal[TElement](String commandText, String entitySetName, ExecutionOptions executionOptions, Object[] parameters)
at System.Data.Entity.Core.Objects.ObjectContext.<>c__DisplayClass69`1.<ExecuteStoreQueryReliably>b__68()
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
at System.Data.Entity.Core.Objects.ObjectContext.<>c__DisplayClass69`1.<ExecuteStoreQueryReliably>b__67()
at System.Data.Entity.Infrastructure.DbExecutionStrategy.Execute[TResult](Func`1 operation)
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQueryReliably[TElement](String commandText, String entitySetName, ExecutionOptions executionOptions, Object[] parameters)
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQuery[TElement](String commandText, ExecutionOptions executionOptions, Object[] parameters)
at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
The part that's failing calls a stored procedure to generate some management information. It works fine in my local environment and on the production site, but fails in the test environment. In the environments that work, it takes a fraction of a second to run.
Other parts of the test system are working fine, and the performance is the same as usual.
It turned out that the problem was the pricing tier of the database.
Until now I've been running the test database at Basic level, to keep costs down. Basic level supports databases up to 2GB.
When I imported the most recent copy of the production database it was less than 2GB, so I was allowed to select Basic, but that didn't leave enough space for the temporary tables used by the stored procedure that was causing the problem.
As soon as I switched to the Standard pricing tier and increased the maximum size of my database to 5GB, the problem went away.

Hangfire - Background job creation failed. See inner exception for details

I create a hangfire jobs as `BackgroundJob.Enqueue(x => x.Recalculate(itemID)); to queue a job. When I try to save consecutively few times I get following error from Hangfire.
Here is the stack trace,
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader)
at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command)
at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable`1 commandTimeout, Nullable`1 commandType)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<>c__DisplayClass7_0.<SetJobState>b__0(SqlConnection x)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<Commit>b__4_0(SqlConnection connection)
at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass17_0.<UseTransaction>b__0(SqlConnection connection)
at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](Func`2 func)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction[T](Func`2 func, Nullable`1 isolationLevel)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction(Action`1 action)
at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass7_0.<CreateWithFilters>b__0()
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
Exception message: Background job creation failed. See inner exception for details.(BackgroundJobClientException)
Inner Ex: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Has anyone came across this issue? This works fine in locally. But issue happens in Azure Deployment. I am using a v12 database. Appreciate any help. I noticed that all my hangfire job creation takes like 20 seconds on average to queue the background job. When I try to create the job again and again it times out. Time out seems to be 30 seconds.
I tried to increase the connection timeout but what ever the value I put in connection string, it seems it times out after 30 seconds. But increasing the time out won't solve the underlying issue anyways.
I figured out that DTU usage is at 100% most of the time, which was unacceptable as there was nothing much happening in the Test environment that I was testing this. Then I looked in to performance data in Azure database which gave me worst performing queries, which was really handy. Handfire delete jobs query was causing this performance issue. It was an issue with the Hangfire version we were using. Solution was to upgrade the Hangfire to a latest version.

The wait operation timed out Win32Exception (0x80004005): The wait operation timed out azure

Following error comes when retrieving large amount of data from sql azure. I have already implement the Transient Fault Handling but still getting this error
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.ComponentModel.Win32Exception: The wait operation timed out
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[Win32Exception (0x80004005): The wait operation timed out]
[SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) +1789270
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) +5340622
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +244
System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) +1691
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) +275
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds) +1421
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean asyncWrite) +177
System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) +208
System.Data.SqlClient.SqlCommand.ExecuteNonQuery() +163
System.Web.SessionState.SqlSessionStateStore.SqlExecuteNonQueryWithRetry(SqlCommand cmd, Boolean ignoreInsertPKException, String id) +98
[HttpException (0x80004005): Unable to connect to SQL Server session database.]
System.Web.SessionState.SqlSessionStateStore.ThrowSqlConnectionException(SqlConnection conn, Exception e) +235
System.Web.SessionState.SqlSessionStateStore.SqlExecuteNonQueryWithRetry(SqlCommand cmd, Boolean ignoreInsertPKException, String id) +390
System.Web.SessionState.SqlSessionStateStore.SetAndReleaseItemExclusive(HttpContext context, String id, SessionStateStoreData item, Object lockId, Boolean newItem) +589
System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs) +565
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +136
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +69
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.34009
http://i.stack.imgur.com/8BloW.png
The SqlClient.SqlCommand object has a property CommandTimeout. It's default value is 30 (seconds). You should set it to a higher value.
You can set the timeout value higher or you can turn off the timeout value, as I did in the below code. Note that you may encounter errors if the object that holds the data exceeds 2Gb. You may want to consider redesigning your query to take smaller chunks of data at a time.
// I'm populating an ADO.Net DataTable for this demo but populate whatever object you'd like
DataTable DtFromSQL = new DataTable();
SqlConnection myConnection = new SqlConnection("ConnectionString");
myConnection.Open();
SqlDataReader myReader = null;
SqlCommand myCommand = new SqlCommand("enter some SQL query here", myConnection);
// A CommandTimeout Value of 0 turns the timout off, otherwise you can set it to some value in seconds
myCommand.CommandTimeout = 0;
myReader = myCommand.ExecuteReader();
DtFromSQL.Load(myReader8);
myConnection.Close();
DtFromSQL;

Timeout error in VB.NET while running a sql script

I was wondering if someone can help me figure why I am getting this error from a simple stored procedure that I created in SQL Server 2008 R2. I am trying to obtain data and insert it into a temporary table. if it is a timeout issue, how can I increase the timeout length to 420 seconds. to my understanding the timeout is good for only 30 seconds.
this is the error in its entirety:
Server Error in '/WebSite9' Application.
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) +1951450
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) +4849003
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) +194
System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) +2394
System.Data.SqlClient.SqlDataReader.ConsumeMetaData() +33
System.Data.SqlClient.SqlDataReader.get_MetaData() +83
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) +297
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async) +954
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result) +162
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) +32
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) +141
System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior) +12
System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior) +10
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +130
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +287
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, String srcTable) +92
System.Web.UI.WebControls.SqlDataSourceView.ExecuteSelect(DataSourceSelectArguments arguments) +1297
System.Web.UI.WebControls.BaseDataList.GetData() +38
System.Web.UI.WebControls.DataList.CreateControlHierarchy(Boolean useDataSource) +153
System.Web.UI.WebControls.BaseDataList.OnDataBinding(EventArgs e) +54
System.Web.UI.WebControls.BaseDataList.DataBind() +55
System.Web.UI.WebControls.BaseDataList.EnsureDataBound() +60
System.Web.UI.WebControls.BaseDataList.CreateChildControls() +69
System.Web.UI.Control.EnsureChildControls() +87
System.Web.UI.Control.PreRenderRecursiveInternal() +44
System.Web.UI.Control.PreRenderRecursiveInternal() +171
System.Web.UI.Control.PreRenderRecursiveInternal() +171
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +842
Version Information: Microsoft .NET Framework Version:2.0.50727.4952; ASP.NET Version:2.0.50727.4955
Dim cmd = connection.CreateCommand()
cmd.CommandTimeout = 420
Use the CommandTimeout property.
Set the timeout in ASP code
VB.NET:
Dim objCmd = conn.CreateCommand()
objCmd.CommandTimeout = 420
or Classic:
set con = createObject("ADODB.Connection")
con.open connectionString
con.commandTimeout = 420
I would review your execution plan on your query as well to determine if you have some table scans going on resulting in a poor performing query.

WCF Service creates wcf Client proxy, starts transaction, calls proxy method but fails on Message Security

I am calling a WCF service from a WCF proxy(client) withing another WCF service as follows:
Client -> Service1 -> Proxy.Method -> Service2
I use ASPNet Compatability everywhre with ASP.Net Membership configured and working everywhere. If I call the Service2 from the proxy using a TransactionSope I always get a message security error on Service2 which is unable to verify the message.
Looking at a WCF service trace (below) it looks like the problem seems to be the SqlProvider in Service2. I can only think that somehow a transaction might be deadlocked but calls to other services BEFORE the transaction have all completed successfully and they DON'T use transactions at all.
If I don't wrap the service call in a TransactionScope everything works fine! Anyone got any idea please?
Brian
131075
3
0
2
BRIANFURLONG-HP
http://msdn.microsoft.com/en-IE/library/System.ServiceModel.Diagnostics.ThrowingException.aspx
Throwing an exception.
/LM/W3SVC/1/ROOT/Nad.CheckoutServices.Host-20-129243786426619315
System.ServiceModel.Security.MessageSecurityException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
Message security verification failed.
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessage(Message& message, TimeSpan timeout)
at System.ServiceModel.Security.SecurityProtocol.VerifyIncomingMessage(Message& message, TimeSpan timeout, SecurityProtocolCorrelationState[] correlationStates)
at System.ServiceModel.Channels.SecurityChannelListener1.ServerSecurityChannel1.VerifyIncomingMessage(Message& message, TimeSpan timeout, SecurityProtocolCorrelationState[] correlationState)
at System.ServiceModel.Channels.SecurityChannelListener1.SecurityReplyChannel.ProcessReceivedRequest(RequestContext requestContext, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelListener1.ReceiveRequestAndVerifySecurityAsyncResult.ProcessInnerItem(RequestContext innerItem, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelListener1.ReceiveItemAndVerifySecurityAsyncResult2.OnInnerReceiveDone()
at System.ServiceModel.Channels.SecurityChannelListener1.ReceiveItemAndVerifySecurityAsyncResult2.StartInnerReceive()
at System.ServiceModel.Channels.SecurityChannelListener1.ReceiveItemAndVerifySecurityAsyncResult2.Start()
at System.ServiceModel.Channels.SecurityChannelListener1.SecurityReplyChannel.BeginTryReceiveRequest(TimeSpan timeout, AsyncCallback callback, Object state)
at System.ServiceModel.Dispatcher.ReplyChannelBinder.BeginTryReceive(TimeSpan timeout, AsyncCallback callback, Object state)
at System.ServiceModel.Dispatcher.ErrorHandlingReceiver.BeginTryReceive(TimeSpan timeout, AsyncCallback callback, Object state)
at System.ServiceModel.Dispatcher.ChannelHandler.BeginTryReceive()
at System.ServiceModel.Dispatcher.ChannelHandler.EnsurePump()
at System.ServiceModel.Dispatcher.ChannelHandler.OpenAndEnsurePump()
at System.ServiceModel.Dispatcher.ChannelHandler.OpenAndEnsurePump(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object o)
at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace>
**<ExceptionString>System.ServiceModel.Security.MessageSecurityException: Message security verification failed. ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.**
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at System.Web.Security.SqlMembershipProvider.GetPasswordWithFormat(String username, Boolean updateLastLoginActivityDate, Int32& status, String& password, Int32& passwordFormat, String& passwordSalt, Int32& failedPasswordAttemptCount, Int32& failedPasswordAnswerAttemptCount, Boolean& isApproved, DateTime& lastLoginDate, DateTime& lastActivityDate)
at System.Web.Security.SqlMembershipProvider.CheckPassword(String username, String password, Boolean updateLastLoginActivityDate, Boolean failIfNotApproved, String& salt, Int32& passwordFormat)
at System.Web.Security.SqlMembershipProvider.CheckPassword(String username, String password, Boolean updateLastLoginActivityDate, Boolean failIfNotApproved)
at System.Web.Security.SqlMembershipProvider.ValidateUser(String username, String password)
at System.IdentityModel.Selectors.UserNamePasswordValidator.MembershipProviderValidator.Validate(String userName, String password)
at System.IdentityModel.Selectors.CustomUserNameSecurityTokenAuthenticator.ValidateUserNamePasswordCore(String userName, String password)
at System.IdentityModel.Selectors.UserNameSecurityTokenAuthenticator.ValidateTokenCore(SecurityToken token)
at System.IdentityModel.Selectors.SecurityTokenAuthenticator.ValidateToken(SecurityToken token)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlReader reader, SecurityTokenResolver tokenResolver, IList1 allowedTokenAuthenticators, SecurityTokenAuthenticator& usedTokenAuthenticator)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlDictionaryReader reader, Int32 position, Byte[] decryptedBuffer, SecurityToken encryptionToken, String idInEncryptedForm, TimeSpan timeout)
at System.ServiceModel.Security.ReceiveSecurityHeader.ExecuteFullPass(XmlDictionaryReader reader)
at System.ServiceModel.Security.StrictModeSecurityHeaderElementInferenceEngine.ExecuteProcessingPasses(ReceiveSecurityHeader securityHeader, XmlDictionaryReader reader)
at System.ServiceModel.Security.ReceiveSecurityHeader.Process(TimeSpan timeout, ChannelBinding channelBinding, ExtendedProtectionPolicy extendedProtectionPolicy)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessageCore(Message& message, TimeSpan timeout)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessage(Message& message, TimeSpan timeout)
--- End of inner exception stack trace ---
System.Data.SqlClient.SqlException, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at System.Web.Security.SqlMembershipProvider.GetPasswordWithFormat(String username, Boolean updateLastLoginActivityDate, Int32& status, String& password, Int32& passwordFormat, String& passwordSalt, Int32& failedPasswordAttemptCount, Int32& failedPasswordAnswerAttemptCount, Boolean& isApproved, DateTime& lastLoginDate, DateTime& lastActivityDate)
at System.Web.Security.SqlMembershipProvider.CheckPassword(String username, String password, Boolean updateLastLoginActivityDate, Boolean failIfNotApproved, String& salt, Int32& passwordFormat)
at System.Web.Security.SqlMembershipProvider.CheckPassword(String username, String password, Boolean updateLastLoginActivityDate, Boolean failIfNotApproved)
at System.Web.Security.SqlMembershipProvider.ValidateUser(String username, String password)
at System.IdentityModel.Selectors.UserNamePasswordValidator.MembershipProviderValidator.Validate(String userName, String password)
at System.IdentityModel.Selectors.CustomUserNameSecurityTokenAuthenticator.ValidateUserNamePasswordCore(String userName, String password)
at System.IdentityModel.Selectors.UserNameSecurityTokenAuthenticator.ValidateTokenCore(SecurityToken token)
at System.IdentityModel.Selectors.SecurityTokenAuthenticator.ValidateToken(SecurityToken token)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlReader reader, SecurityTokenResolver tokenResolver, IList1 allowedTokenAuthenticators, SecurityTokenAuthenticator& usedTokenAuthenticator)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlDictionaryReader reader, Int32 position, Byte[] decryptedBuffer, SecurityToken encryptionToken, String idInEncryptedForm, TimeSpan timeout)
at System.ServiceModel.Security.ReceiveSecurityHeader.ExecuteFullPass(XmlDictionaryReader reader)
at System.ServiceModel.Security.StrictModeSecurityHeaderElementInferenceEngine.ExecuteProcessingPasses(ReceiveSecurityHeader securityHeader, XmlDictionaryReader reader)
at System.ServiceModel.Security.ReceiveSecurityHeader.Process(TimeSpan timeout, ChannelBinding channelBinding, ExtendedProtectionPolicy extendedProtectionPolicy)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessageCore(Message& message, TimeSpan timeout)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessage(Message& message, TimeSpan timeout)
</StackTrace>
<ExceptionString>System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at System.Web.Security.SqlMembershipProvider.GetPasswordWithFormat(String username, Boolean updateLastLoginActivityDate, Int32& status, String& password, Int32& passwordFormat, String& passwordSalt, Int32& failedPasswordAttemptCount, Int32& failedPasswordAnswerAttemptCount, Boolean& isApproved, DateTime& lastLoginDate, DateTime& lastActivityDate)
at System.Web.Security.SqlMembershipProvider.CheckPassword(String username, String password, Boolean updateLastLoginActivityDate, Boolean failIfNotApproved, String& salt, Int32& passwordFormat)
at System.Web.Security.SqlMembershipProvider.CheckPassword(String username, String password, Boolean updateLastLoginActivityDate, Boolean failIfNotApproved)
at System.Web.Security.SqlMembershipProvider.ValidateUser(String username, String password)
at System.IdentityModel.Selectors.UserNamePasswordValidator.MembershipProviderValidator.Validate(String userName, String password)
at System.IdentityModel.Selectors.CustomUserNameSecurityTokenAuthenticator.ValidateUserNamePasswordCore(String userName, String password)
at System.IdentityModel.Selectors.UserNameSecurityTokenAuthenticator.ValidateTokenCore(SecurityToken token)
at System.IdentityModel.Selectors.SecurityTokenAuthenticator.ValidateToken(SecurityToken token)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlReader reader, SecurityTokenResolver tokenResolver, IList1 allowedTokenAuthenticators, SecurityTokenAuthenticator& usedTokenAuthenticator)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlDictionaryReader reader, Int32 position, Byte[] decryptedBuffer, SecurityToken encryptionToken, String idInEncryptedForm, TimeSpan timeout)
at System.ServiceModel.Security.ReceiveSecurityHeader.ExecuteFullPass(XmlDictionaryReader reader)
at System.ServiceModel.Security.StrictModeSecurityHeaderElementInferenceEngine.ExecuteProcessingPasses(ReceiveSecurityHeader securityHeader, XmlDictionaryReader reader)
at System.ServiceModel.Security.ReceiveSecurityHeader.Process(TimeSpan timeout, ChannelBinding channelBinding, ExtendedProtectionPolicy extendedProtectionPolicy)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessageCore(Message& message, TimeSpan timeout)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessage(Message& message, TimeSpan timeout)
This error looks like you are using calling a service on http but it wants to use https.
In the end I realised the transaction was blocking the access to membership which was caused by accessing the Service2. Tnis was an issue with the way in which Membership handles transactions. It specifically forces "New Transaction" when it should simply enlist in the ambient transaction if it exists