we have a problem that we can't reproduce, it happens randomly on our web server (Windows Server 2008 Datacenter 64-bit, hosted on Amazon EC2) running ASP.NET 4.
It starts with a ASP.NET warning in the error log (a strange GET request with a really long URL):
Exception information:
Exception type: HttpException
Exception message: The length of the URL for this request exceeds the configured maxUrlLength value.
at System.Web.HttpRequest.ValidateInputIfRequiredByConfig()
at System.Web.HttpApplication.PipelineStepManager.ValidateHelper(HttpContext context)
Then an error:
Application ID: /LM/W3SVC/2/ROOT
Process ID: 4604
Exception: System.Threading.ThreadAbortException
Message: Thread was being aborted.
StackTrace: at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
Then another error:
Application ID: DefaultDomain
Process ID: 4604
Exception: System.Threading.ThreadAbortException
Message: Thread was being aborted.
StackTrace:
And then the application error:
Faulting application w3wp.exe, version 7.0.6002.18005, time stamp 0x49e03238, faulting module kernel32.dll, version 6.0.6002.18005, time stamp 0x49e041d1, exception code 0xe0434352, fault offset 0x00000000000176fd, process id 0x%9, application start time 0x%10.
Windbg
I managed to get a dump for a crash with adplus, but I really don't know what to look for. I've troubleshooted some stackoverflows with Windbg before, but I don't know which approach to use for this error.
!pe on the thread with the error:
Exception object: 00000001c323d948
Exception type: System.Threading.ThreadAbortException
Message: Thread was being aborted.
InnerException:
StackTrace (generated):
StackTraceString:
HResult: 80131530
!clrstack
Child SP IP Call Site
0000000015f7f078 0000000076c176fd [GCFrame: 0000000015f7f078]
0000000015f7f258 0000000076c176fd [GCFrame: 0000000015f7f258]
k
Child-SP RetAddr Call Site
0000000015f7edf0 000007fef2826e39 kernel32!RaiseException+0x39
0000000015f7eec0 000007fef2bbbfb4 clr!RaiseTheExceptionInternalOnly+0x363
0000000015f7eff0 000007fef2bbc906 clr!RaiseTheException+0xa4
0000000015f7f020 000007fef2c3b99b clr!BStrFromString+0x66
0000000015f7f050 000007fef2c3b9a9 clr!RealCOMPlusThrow+0x3b
0000000015f7f0c0 000007fef2c93156 clr!RealCOMPlusThrow+0x9
0000000015f7f0f0 000007fef2b229f3 clr!Thread::RaiseCrossContextException+0x2a7
0000000015f7f310 000007fef2830886 clr! ?? ::FNODOBFM::string'+0xafb03
0000000015f7f4c0 000007fef27fcce3 clr!UM2MDoADCallBack+0x9e
0000000015f7f530 000007fef845ba59 clr!UMThunkStubAMD64+0x273
0000000015f7f5c0 000007fef8458f02 webengine4!W3_MGD_HANDLER::ProcessNotification+0x79
0000000015f7f5f0 000007fef27d4595 webengine4!ProcessNotificationCallback+0x43
0000000015f7f620 000007fef27d3ac8 clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x181
0000000015f7f6d0 000007fef294658f clr!ThreadpoolMgr::NewWorkerThreadStart+0x2e5
0000000015f7f770 000007fef29447c6 clr!ThreadpoolMgr::WorkerThreadStart+0x3b
0000000015f7f810 0000000076c1be3d clr!Thread::intermediateThreadProc+0x7d
0000000015f7fbd0 0000000076d56a51 kernel32!BaseThreadInitThunk+0xd
0000000015f7fc00 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
Anybody have a clue what this could be? Or can point me in the right direction for analyzing it with Windbg?
Edit:
The incoming urls usually looks like this:
foo.bar.com/wEPDwULLTE1MTk5MzIzMTFkGAMFFmN0bDAwJGRiMSRkZGxEYXRhYmFzZXMPFCsAAmRkZAU7Y3RsMDAkU2VhcmNoQ2xvdWQxJF9SaWdodENvbHVtbiRfU2VhcmNoQ2xvdWQkbHN2U2VhcmNoVGVybXMPFCsADmRkZGRkZGQ8KwAUAAIUZGRkZgL/D2QFK2N0bDAwJHN1cnZleTEkX1JpZ2h0Q29sdW1uJF9JUiR1c2VyQ29tbWVudHMPFCsAA2VnZGQeuUcvQDsShDIp1k7YjJw70Ry 9/Q1B9Sd1egrovYgkw==/
but I found in the event log that this happen for this urls like this aswell:
foo.bar.com/&
("dangerous request" in .NET 4)
You should at least load symbols and then see the correct strings displayed in call stack.
If you are not capable of handling such analysis, I think opening a support case via http://support.microsoft.com is good for you.
I have been trying to work out a similar issue.
I found this article which purports a process for determining the cause, I think it actually makes sense:
http://blogs.msdn.com/b/asiatech/archive/2012/06/21/how-to-troubleshoot-httpexception-request-timed-out-asp-net-4-0-64-bit.aspx
Caveat: I have not been able to sucessfully complete the steps described. I will be posting here asking for help.
Related
We are experiencing a specific statefull service that is unable to fully "go green", the partitions keep reshuffling and we are not seeing any indications of the errors in our own logs. After lots of digging we found something suspicious in the EventLogs on one of the WMs (pasted below)
Application: (hidden).exe
Framework Version: v4.0.30319
Description: The application requested process termination through System.Environment.FailFast(string message).
Message: ((copyMode & CopyMode.FalseProgress) == 0) || (sourceStartingLsn < targetStartingLsn). Source starting lsn : 2018, target starting lsn :2018
Stack:
at System.Environment.FailFast(System.String)
at Microsoft.ServiceFabric.Replicator.Utility.CodingError(System.String, System.Object[])
at Microsoft.ServiceFabric.Replicator.Utility.Assert(Boolean, System.String, ...)
at Microsoft.ServiceFabric.Replicator.LoggingReplicator.GetLogRecordsToCopy(Microsoft.ServiceFabric.Replicator.ProgressVector, System.Fabric.Epoch, Microsoft.ServiceFabric.Replicator.LogicalSequenceNumber, Microsoft.ServiceFabric.Replicator.LogicalSequenceNumber, Int64, Int64, Microsoft.ServiceFabric.Replicator.LogicalSequenceNumber ByRef, Microsoft.ServiceFabric.Replicator.LogicalSequenceNumber ByRef, Microsoft.ServiceFabric.Data.IAsyncEnumerator'1 ByRef, Microsoft.ServiceFabric.Replicator.BeginCheckpointLogRecord ByRef)
at Microsoft.ServiceFabric.Replicator.LoggingReplicatorCopyStream+d__3.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.ServiceFabric.Replicator.LoggingReplicatorCopyStream+d__3, Microsoft.ServiceFabric.Data.Impl, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__3 ByRef)
at Microsoft.ServiceFabric.Replicator.LoggingReplicatorCopyStream.GetNextAsync(System.Threading.CancellationToken)
at System.Fabric.StateProviderBroker+AsyncEnumerateOperationDataBroker.b__8(System.Threading.CancellationToken)
at System.Fabric.Interop.Utility.WrapNativeAsyncMethodImplementation(System.Func`2, IFabricAsyncOperationCallback, System.String, System.Fabric.Interop.InteropApi)
We are not sure what to make of this. Seems related to state replication but we don't think we've changed anything related to the state of the service. Since the service is exiting with FailFast, we don't get a chance to do anything in our code to remedy this so we are basically stuck in this loop right now (luckily on a non-Live environment but still...)
Does anyone have any idea what this is related to specifically and how we can recover the service and the data?
I have the following vb.net code:
Imports IBM.WMQ
[...]
MQEnvironment.Hostname = hostName
MQEnvironment.Port = portNumber
MQEnvironment.Channel = channelName
queueManager = New MQQueueManager(queueManagerName) ' error here
which is throwing the following error:
System.IO.FileNotFoundException occurred
FileName=C:\Users\User\Documents\Visual Studio 2012\Projects\[...]\bin\Debug\mqclient.ini
HResult=-2147024894
Message=Could not find file 'C:\Users\User\Documents\Visual Studio 2012\Projects\[...]\bin\Debug\mqclient.ini'.
Source=mscorlib
StackTrace:
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
I am not using any ini files in the construction of my queue manager, so does anyone have any idea what's going on - why is it even looking for one, and why in the same directory as the program? I have installed the MQ client, and afaik I have all the environmental variables, etc. set up properly.
Thanks for any help you can give
Is that an unhandled or a first chance exception? Internally, the MQ .net layer will try to read a MQClient.ini but should function quite happily without it. It reads the file for compatibility with the C client, and can handle some of the MQClient.ini stanzas. I would not have expected an absence of such a file to cause problems, but it will try to open it internally. Was that the full callstack, as I'd have expected some MQ libraries on the stack otherwise.
We are recently upgraded a web/mvc application to use StrucutreMap 3.0.4
Now, when attempting to profile the application using RedGate Ant's profiler at the "Line Level Timings, All Methods with Source" or greater level, we get the following error
Operation could destabilize the runtime.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Security.VerificationException: Operation could destabilize the runtime.
Source Error:
Line 174:
Line 175: Initialise(context);
Line 176: ObjectFactory.Initialize(x => { x.AddRegistry<BWebsiteIoc>(); });
Line 177: }
Line 178: }
Source File: c:\_Development\NT\Platform\WebSite\Global.asax.cs Line: 176
Stack Trace:
[VerificationException: Operation could destabilize the runtime.]
StructureMap.ObjectFactory.Initialize(Action`1 action) in c:\BuildAgent\work\996e173a8ceccdca\src\StructureMap\ObjectFactory.cs:42
Nga.Platform.Website.FirstRequestInitialisation.Initialise(HttpContext context) in c:\_Development\NT\Platform\WebSite\Global.asax.cs:176
Nga.Platform.Website.MvcApplication.Application_BeginRequest(Object source, EventArgs e) in c:\_Development\NT\Platform\WebSite\Global.asax.cs:106
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +182
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +165
I've tried adding to system.web in web.config as per http://documentation.red-gate.com/display/APP8/Operation+could+destabilize+the+runtime+error+profiling+ASP.NET
May be an old question, but it's still one of the first for me on google results, so here goes.
I was able to solve the issue by deleting structuremap's pdb files before profiling.
Added
<assemblyName>StructureMap</assemblyName> to %LOCALAPPDATA%\Red Gate\ANTS Performance Profiler 9\LineLevelBlacklist.xml which did the trick.
I was able to work around the problem by adding the following to the AssemblyInfo.cs of the class library causing the problem.
[assembly: System.Security.SecurityRules(System.Security.SecurityRuleSet.Level1)]
I found this solution in a Redgate forum discussion.
I am trying to debug a WCF service which is crashing from time to time. I have created a crash dump file using adplus, that was the easy part, I used this command.
adplus.exe -crash -pmn myservicehost.exe -o c:\dump
I am opening the file .dmp file with winDbg x64 version 6.2.9200. When I look at the threads !Threads I see there are a bunch of threads that that are waiting for a callback to complete (I think).
0:031> ~~[1b00]s
ntdll!NtWaitForMultipleObjects+0xa:
00000000`778d18ca c3 ret
0:029> ~~[1b00]s
ntdll!NtWaitForMultipleObjects+0xa:
00000000`778d18ca c3 ret
Eventually it crashes. Here is the output when I look at the call stack from that exception using !PrintException /d -nested 00000002814ad6d0
Exception object: 00000002814af4c8
Exception type: System.Runtime.CallbackException
Message: A user callback threw an exception. Check the exception stack and inner exception to determine the callback that failed.
InnerException: System.ServiceModel.CommunicationObjectAbortedException, Use !PrintException 00000002814ad6d0 to see more.
StackTrace (generated):
SP IP Function
000000000FA89D40 000007FEDD9AD3E2 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnClosed()+0x262
000000000FA8C240 000007FEDD93759D System_ServiceModel_ni!System.ServiceModel.ServiceHostBase.OnClosed()+0x6d
000000000FA8C290 000007FEDD9433D0 System_ServiceModel_ni!System.ServiceModel.ServiceHost.OnClosed()+0x10
000000000FA8C2C0 000007FEDE185B86 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Abort()+0x2b6
000000000FA8C3C0 000007FE8ABAC89C MyCompany_WcfApp_WcfAppServiceHost!MyCompany.WcfApp.WcfAppServiceHost.WcfAppServiceHost.FaultedServiceHandler(System.Object, System.EventArgs)+0x26c
000000000FA8C5F0 0000000000000000 mscorlib_ni!System.EventHandler.Invoke(System.Object, System.EventArgs)+0x1
000000000FA8C5F0 000007FEDE184E9A System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnFaulted()+0x1ca
000000000FA8C670 000007FEDE184784 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Fault()+0x94
000000000FA8C6E0 000007FEDE184E9A System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnFaulted()+0x1ca
000000000FA8C760 000007FEDE184784 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Fault()+0x94
000000000FA8C7D0 000007FEDE184E9A System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnFaulted()+0x1ca
000000000FA8C850 000007FEDE184784 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Fault()+0x94
000000000FA8C8C0 000007FEDE475407 System_ServiceModel_ni!System.ServiceModel.Channels.MsmqInputChannelBase.TryReceive(System.TimeSpan, System.ServiceModel.Channels.Message ByRef)+0x4f7
000000000FA8EBF0 000007FEDE5409AE System_ServiceModel_ni!System.ServiceModel.Dispatcher.InputChannelBinder.TryReceive(System.TimeSpan, System.ServiceModel.Channels.RequestContext ByRef)+0x2e
000000000FA8EC50 000007FEDEAC29E2 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ErrorHandlingReceiver.TryReceive(System.TimeSpan, System.ServiceModel.Channels.RequestContext ByRef)+0x646022
000000000FA8ECB0 000007FEDE47C8D6 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.TryTransactionalReceive(System.Transactions.Transaction, System.ServiceModel.Channels.RequestContext ByRef)+0x396
000000000FA8ED70 000007FEDE47BE07 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.TransactedLoop()+0xb7
000000000FA8EDF0 000007FEDE47BD31 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.SyncTransactionalMessagePump()+0x21
000000000FA8EE20 000007FEDE47A829 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.OnStartSyncMessagePump(System.Object)+0x209
000000000FA8EED0 000007FEDB6DE651 System_ServiceModel_Internals_ni!System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x71
000000000FA8EF30 000007FEDB77A260 System_ServiceModel_Internals_ni!System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x9bcd0
000000000FA8EF90 000007FEDF225C26 mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x96
Problem is, this isn't really helping me. I need to find out what is creating all these locks. Does anyone have any suggestions on how I might do that? I have never had an issue that required this level of debugging and to be honest I am not 100% sure what I am doing. If I haven't given enough info please let me know I will happily provide you with whatever else I can.
Thanks
Try using !SyncBlk command of SOS to determine deadlocks if any.
I'm trying to write a windows service. It installs fine, but fails when I run it with the following exception. I've searched for the string "MyNewProgramService", but I can't find any conversions that would throw this error. I've also added try/catch blocks to a bunch of code with custom exception handling without finding where this exception is occuring. I'm thinking it's somewhere in the auto-generated configuartion/setup code. Any ideas?
Event Type: Error
Event Source: MyNewProgram Event
Category: None Event
ID: 0
Date: 4/15/2010
Time: 12:48:34 PM
User: N/A
Computer: 20F7KF1
Description: Service cannot be started. System.InvalidCastException:
Conversion from string "MyNewProgramService" to type 'Integer' is not valid. --->
System.FormatException: Input string was not in a correct format.
at Microsoft.VisualBasic.CompilerServices.Conversions.ParseDouble(String Value,
NumberFormatInfo NumberFormat)
at Microsoft.VisualBasic.CompilerServices.Conversions.ToInteger(String Value)
--- End of inner exception stack trace ---
at Microsoft.VisualBasic.CompilerServices.Conversions.ToInteger(String Value)
at TaskManagerFailureHandlerService.MyNewProgramService.OnStart(String[] args)
at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)
it falls in method MyNewProgramService.OnStart, there Conversions.ToInteger is called somewhere. Just search for it in method. Then check parameter. It might come from config or something, thats why you don't see string in text.
Check your configuration file.
You might also check the account the service is running as. Just as a test go to the login tab in the services list and have it login under your own account. If the problem vanishes it's a permissions problem.