I see following call stack for finalizer thread. Is it normal to have a call to WaitForSingleObject at top in finalizer? Is there anyway I can determine if its not deadlocked or waiting for really long time?
0:009> k
Child-SP RetAddr Call Site
00000000`0a56e5c8 000007fe`fd5010dc ntdll!NtWaitForSingleObject+0xa
00000000`0a56e5d0 000007fe`fdfabeb5 KERNELBASE!WaitForSingleObjectEx+0x79
00000000`0a56e670 000007fe`fe04a576 rpcrt4!UTIL_GetOverlappedResultEx+0x45
00000000`0a56e6b0 000007fe`fdfaf0dd rpcrt4!WS_SyncRecv+0xf6
00000000`0a56e720 000007fe`fdfe7a29 rpcrt4!OSF_CCONNECTION::TransSendReceive+0x18d
00000000`0a56e780 000007fe`fdfa7f61 rpcrt4!OSF_CCONNECTION::SendBindPacket+0xa5c
00000000`0a56e930 000007fe`fdfa8e27 rpcrt4!OSF_CCONNECTION::ActuallyDoBinding+0xc1
00000000`0a56e9d0 000007fe`fdfa8bb6 rpcrt4!OSF_CCONNECTION::OpenConnectionAndBind+0x207
00000000`0a56ea90 000007fe`fdfa8acd rpcrt4!OSF_CCALL::BindToServer+0xc6
00000000`0a56eb40 000007fe`fdfadaeb rpcrt4!OSF_BINDING_HANDLE::InitCCallWithAssociation+0xa5
00000000`0a56eba0 000007fe`fdfad9d0 rpcrt4!OSF_BINDING_HANDLE::AllocateCCall+0x102
00000000`0a56ecd0 000007fe`fdfc74eb rpcrt4!OSF_BINDING_HANDLE::NegotiateTransferSyntax+0x30
00000000`0a56ed20 000007fe`ff462271 rpcrt4!I_RpcNegotiateTransferSyntax+0x9f
00000000`0a56edb0 000007fe`ff45d185 ole32!CRpcChannelBuffer::NegotiateSyntax+0x69
00000000`0a56ee20 000007fe`fe05ba22 ole32!NdrExtNegotiateTransferSyntax+0xe5
00000000`0a56ee60 000007fe`fe05cbbb rpcrt4!Ndr64pClientSetupTransferSyntax+0x453
00000000`0a56eec0 000007fe`ff4621d0 rpcrt4!NdrpClientCall3+0xcb
00000000`0a56f180 000007fe`ff31d8a2 ole32!ObjectStublessClient+0x11d
00000000`0a56f510 000007fe`ff321bb3 ole32!ObjectStubless+0x42
00000000`0a56f560 000007fe`ff321b22 ole32!RemoteReleaseRifRefHelper+0x57
00000000`0a56f5b0 000007fe`ff3217eb ole32!RemoteReleaseRifRef+0xca
00000000`0a56f620 000007fe`ff321417 ole32!CStdMarshal::DisconnectCliIPIDs+0x4c2
00000000`0a56f720 000007fe`ff3194fa ole32!CStdMarshal::Disconnect+0x40c
00000000`0a56f780 000007fe`ff319428 ole32!CStdIdentity::~CStdIdentity+0xa6 [d:\w7rtm\com\ole32\com\dcomrem\stdid.cxx # 312]
00000000`0a56f7b0 000007fe`ff319b49 ole32!CStdIdentity::`scalar deleting destructor'+0x14
00000000`0a56f7e0 000007fe`f2a79f94 ole32!CStdIdentity::CInternalUnk::Release+0xdc [d:\w7rtm\com\ole32\com\dcomrem\stdid.cxx # 767]
00000000`0a56f810 000007fe`f2ba8bea clr!SafeReleasePreemp+0x74
00000000`0a56f860 000007fe`f2ba8acc clr!RCW::ReleaseAllInterfaces+0xda
00000000`0a56f8b0 000007fe`f2ba8c14 clr!RCW::ReleaseAllInterfacesCallBack+0x54
00000000`0a56f930 000007fe`f2ba937e clr!RCW::Cleanup+0x25
00000000`0a56f980 000007fe`f2ba9214 clr!RCWCleanupList::ReleaseRCWListRaw+0x16
00000000`0a56f9b0 000007fe`f2bb005a clr!RCWCleanupList::ReleaseRCWListInCorrectCtx+0x94
00000000`0a56f9f0 000007fe`f2be326e clr!RCWCleanupList::CleanupAllWrappers+0xe5
00000000`0a56fa70 000007fe`f2be319f clr!SyncBlockCache::CleanupSyncBlocks+0xc2
00000000`0a56fae0 000007fe`f2be47c7 clr!Thread::DoExtraWorkForFinalizer+0xdc
00000000`0a56fb10 000007fe`f2a5458c clr!SVR::GCHeap::FinalizerThreadWorker+0x78
00000000`0a56fb50 000007fe`f2a5451a clr!Frame::Pop+0x50
00000000`0a56fb90 000007fe`f2a54491 clr!COMCustomAttribute::PopSecurityContextFrame+0x192
00000000`0a56fc90 000007fe`f2b31bfe clr!COMCustomAttribute::PopSecurityContextFrame+0xbd
00000000`0a56fd20 000007fe`f2b45020 clr!ManagedThreadBase_NoADTransition+0x3f
00000000`0a56fd80 000007fe`f2ab33de clr!SVR::GCHeap::FinalizerThreadStart+0xb4
00000000`0a56fdc0 00000000`772459ed clr!Thread::intermediateThreadProc+0x7d
00000000`0a56fe80 00000000`7747c541 kernel32!BaseThreadInitThunk+0xd
00000000`0a56feb0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
0:009> !CLRStack
OS Thread Id: 0x1338 (6)
Child SP IP Call Site
000000000b67fcd8 00000000774a12fa [DebuggerU2MCatchHandlerFrame: 000000000b67fcd8]
This is a finalizer worker and it is blocked for a RPC response. In this case if the thread is not waiting for more than a few milliseconds (depending on how long the COM server takes to respond to the request) it is normal. But if we are waiting longer then you might want to investigate what happened to the response packet.
There is no way in user mode to understand what's the tick-count of the thread you may want to examine the same thread using livekd on a running system if the problem is re-producible
It is this indication of a blocked finalizer. A very simple example if you use COM to interact between Excel and C#.
You have a Module of Excel that is being made a member variable of C# and then the issue might arise as the control of that module is within GC's hands but excel will be waiting for it to be released by the C#.
Simple way is to just clear any unmanaged code you have in your C# component before letting the unmanaged code to be cleared.
Related
I'm maintaining an antedeluvian Notes application which connects to a SAP back-end via a manually done 'Webservice'
The server is running Domino Release 7.0.4FP2 HF97.
The Webservice is not the more recently Webservice Consumer, but a large Java agent which is using Apache soap.jar (org.apache.soap). Below an example of the calling code.
private Call setupSOAPCall() {
Call call = new Call();
SOAPHTTPConnection conn = new SOAPHTTPConnection();
call.setSOAPTransport(conn);
call.setEncodingStyleURI(Constants.NS_URI_SOAP_ENC);
There has been a change in the SAP system which is now taking 8 minutes to complete (verified by SAP Team).
I'm getting an error message as follows:
[SOAPException: faultCode=SOAP-ENV:Client; msg=For input string: "906 "; targetException=java.lang.NumberFormatException: For input string: "906 "]
I found a blog article describing the error message quite closely:
https://thejavablog.wordpress.com/category/jmeter/
and I've come to the hypothesis that it is a timeout message that is returning to my Call object and that this timeout message is being incorrectly parsed, hence the NumberFormat Exception.
Looking at my logs I can see that there is a time difference of 62 seconds between my call and the response.
I recommended that the server setting in the server document, tab Internet Protocols/HTTP/Timeouts/Request timeouts be changed from 60 seconds to 600 seconds, and the http task restarted with
tell http restart
I've re-run the tests and I am getting the same error, and the time difference is still slightly more than 60 seconds, which is not what I was expecting.
I read Michael Rulnau's blog entry
http://www.mruhnau.net/2014/06/how-to-overcome-domino-webservice.html
which points to this APR
http://www-01.ibm.com/support/docview.wss?uid=swg1LO48272
but I'm not convinced that this would apply in this case, since there is no way that IBM would know that my Java agent is in fact making a Soap call.
My current hypothesis is that I have to use either the setTimeout() method on
org.apache.axis.client.Call
https://axis.apache.org/axis/java/apiDocs/org/apache/axis/client/Call.html
or on the org.apache.soap.transport.http.SOAPHTTPConnection
https://docs.oracle.com/cd/B13789_01/appdev.101/b12024/org/apache/soap/transport/http/SOAPHTTPConnection.html
and that the timeout value is an apache default, not something that is controlled by the Domino server.
I'd be grateful for any help.
I understand your approach, and I hope this is the correct one to solve your problem.
Add a debug (console write would be fine) that display the default Timeout then try to increase it to 10 min.
SOAPHTTPConnection conn = new SOAPHTTPConnection();
System.out.println("time out is :" + conn.getTimeout());
conn.setTimeout(600000);//10 min in ms
System.out.println("after setting it, time out is :" + conn.getTimeout());
call.setSOAPTransport(conn);
Now keep in mind that Dommino has also a Max LotusScript/Java execution time, check this value and (at least for a try) change it: http://www.ibm.com/support/knowledgecenter/SSKTMJ_9.0.1/admin/othr_servertasksagentmanagertab_r.html (it's version 9 help but this part should be identical)
I've since discovered that it wasn't my code generating the error; the default timeout for the apache axis SOAPHTTPConnetion is 0, i.e. no timeout.
I have a job that periodically does some work involving ServerXmlHttpRquest to perform an HTTP POST. The job runs every 60 seconds.
And normally it runs without issue. But there's about a 1 in 50,000 chance (every two or three months) that it will hang:
IXMLHttpRequest http = new ServerXmlHttpRequest();
http.open("POST", deleteUrl, false, "", "");
http.send(stuffToDelete); <---hang
When it hangs, not even the Task Scheduler (with the option enabled to kill the job if it takes longer than 3 minutes to run) can end the task. I have to connect to the remote customer's network, get on the server, and use Task Manager to kill the process.
And then its good for another month or three.
Eventually i started using Task Manager to create a process dump,
so i could analyze where the hang is. After five crash dumps (over the last 11 months or so) i get a consistent picture:
ntdll.dll!_NtWaitForMultipleObjects#20()
KERNELBASE.dll!_WaitForMultipleObjectsEx#20()
user32.dll!MsgWaitForMultipleObjectsEx()
user32.dll!_MsgWaitForMultipleObjects#20()
urlmon.dll!CTransaction::CompleteOperation(int fNested) Line 2496
urlmon.dll!CTransaction::StartEx(IUri * pIUri, IInternetProtocolSink * pOInetProtSink, IInternetBindInfo * pOInetBindInfo, unsigned long grfOptions, unsigned long dwReserved) Line 4453 C++
urlmon.dll!CTransaction::Start(const wchar_t * pwzURL, IInternetProtocolSink * pOInetProtSink, IInternetBindInfo * pOInetBindInfo, unsigned long grfOptions, unsigned long dwReserved) Line 4515 C++
msxml3.dll!URLMONRequest::send()
msxml3.dll!XMLHttp::send()
Contoso.exe!FrobImporter.TFrobImporter.DeleteFrobs Line 971
Contoso.exe!FrobImporter.TFrobImporter.ImportCore Line 1583
Contoso.exe!FrobImporter.TFrobImporter.RunImport Line 1070
Contoso.exe!CommandLineProcessor.TCommandLineProcessor.HandleFrobImport Line 433
Contoso.exe!CommandLineProcessor.TCommandLineProcessor.CoreExecute Line 71
Contoso.exe!CommandLineProcessor.TCommandLineProcessor.Execute Line 84
Contoso.exe!Contoso.Contoso Line 167
kernel32.dll!#BaseThreadInitThunk#12()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart#8()
So i do a ServerXmlHttpRequest.send, and it never returns. It will sit there for days (causing the system to miss financial transactions, until come Sunday night i get a call that it's broken).
It is of no help unless someone knows how to debug code, but the registers in the stalled thread at the time of the dump are:
EAX 00000030
EBX 00000000
ECX 00000000
EDX 00000000
ESI 002CAC08
EDI 00000001
EIP 732A08A7
ESP 0018F684
EBP 0018F6C8
EFL 00000000
Windows Server 2012 R2
Microsoft IIS/8.5
Default timeouts of ServerXmlHttpRequest
You can use serverXmlHttpRequest.setTimeouts(...) to configure the four classes of timeouts:
resolveTimeout: The value is applied to mapping host names (such as "www.microsoft.com") to IP addresses; the default value is infinite, meaning no timeout.
connectTimeout: A long integer. The value is applied to establishing a communication socket with the target server, with a default timeout value of 60 seconds.
sendTimeout: The value applies to sending an individual packet of request data (if any) on the communication socket to the target server. A large request sent to a server will normally be broken up into multiple packets; the send timeout applies to sending each packet individually. The default value is 30 seconds.
receiveTimeout: The value applies to receiving a packet of response data from the target server. Large responses will be broken up into multiple packets; the receive timeout applies to fetching each packet of data off the socket. The default value is 30 seconds.
The KB305053 (a server that decides to keep the connection open will cause serverXmlHttpRequest to wait for the connection to close) seems like it plausibly could be the issue. But the 30 second default timeout would have taken care of that.
Possible workaround - Add myself to a Job
The Windows Task Scheduler is unable to terminate the task; even though the option is enabled to do do.
I will look into using the Windows Job API to add my self process to a job, and use SetInformationJobObject to set a time limit on my process:
CreateJobObject
AssignProcessToJobObject
SetInformationJobObject
to limit my process to three minutes of execution time:
PerProcessUserTimeLimit
If LimitFlags specifies
JOB_OBJECT_LIMIT_PROCESS_TIME, this member is the per-process
user-mode execution time limit, in 100-nanosecond ticks. Otherwise,
this member is ignored.
The system periodically checks to determine
whether each process associated with the job has accumulated more
user-mode time than the set limit. If it has, the process is
terminated.
If the job is nested, the effective limit is the most
restrictive limit in the job chain.
Although since Task Scheduler uses Job objects to also limit a task's time, i'm not hopeful that the Job Object can limit a job either.
Edit: Job objects cannot limit a process by process time - only user time. And with a process idle waiting for an object, it will not accumulate any user time - certainly not three minutes worth.
Bonus Reading
How can a ServerXMLHTTP GET request hang? (GET, not POST)
KB305053: ServerXMLHTTP Stops Responding When You Send a POST Request (which says the timeout should expire; where mine does not)
MS Forums: oHttp.Send - Hangs (HEAD, not POST)
MS Forums: ASP to test SOAP WebService using MSXML2.ServerXMLHTTP Send hangs
CC to MS Support Forums
Consider switching to a newer, supported API.
msxml6.dll using MSXML2.ServerXMLHTTP.6.0
winhttpcom.dll using WinHttp.WinHttpRequest.5.1.
The msxml3.dll library is no longer supported and is only kept around for compatibility reasons. Plus, there were a number of security and stability improvements included with msxml4.dll (and newer) that you are missing out on.
I am trying to debug a WCF service which is crashing from time to time. I have created a crash dump file using adplus, that was the easy part, I used this command.
adplus.exe -crash -pmn myservicehost.exe -o c:\dump
I am opening the file .dmp file with winDbg x64 version 6.2.9200. When I look at the threads !Threads I see there are a bunch of threads that that are waiting for a callback to complete (I think).
0:031> ~~[1b00]s
ntdll!NtWaitForMultipleObjects+0xa:
00000000`778d18ca c3 ret
0:029> ~~[1b00]s
ntdll!NtWaitForMultipleObjects+0xa:
00000000`778d18ca c3 ret
Eventually it crashes. Here is the output when I look at the call stack from that exception using !PrintException /d -nested 00000002814ad6d0
Exception object: 00000002814af4c8
Exception type: System.Runtime.CallbackException
Message: A user callback threw an exception. Check the exception stack and inner exception to determine the callback that failed.
InnerException: System.ServiceModel.CommunicationObjectAbortedException, Use !PrintException 00000002814ad6d0 to see more.
StackTrace (generated):
SP IP Function
000000000FA89D40 000007FEDD9AD3E2 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnClosed()+0x262
000000000FA8C240 000007FEDD93759D System_ServiceModel_ni!System.ServiceModel.ServiceHostBase.OnClosed()+0x6d
000000000FA8C290 000007FEDD9433D0 System_ServiceModel_ni!System.ServiceModel.ServiceHost.OnClosed()+0x10
000000000FA8C2C0 000007FEDE185B86 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Abort()+0x2b6
000000000FA8C3C0 000007FE8ABAC89C MyCompany_WcfApp_WcfAppServiceHost!MyCompany.WcfApp.WcfAppServiceHost.WcfAppServiceHost.FaultedServiceHandler(System.Object, System.EventArgs)+0x26c
000000000FA8C5F0 0000000000000000 mscorlib_ni!System.EventHandler.Invoke(System.Object, System.EventArgs)+0x1
000000000FA8C5F0 000007FEDE184E9A System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnFaulted()+0x1ca
000000000FA8C670 000007FEDE184784 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Fault()+0x94
000000000FA8C6E0 000007FEDE184E9A System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnFaulted()+0x1ca
000000000FA8C760 000007FEDE184784 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Fault()+0x94
000000000FA8C7D0 000007FEDE184E9A System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnFaulted()+0x1ca
000000000FA8C850 000007FEDE184784 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Fault()+0x94
000000000FA8C8C0 000007FEDE475407 System_ServiceModel_ni!System.ServiceModel.Channels.MsmqInputChannelBase.TryReceive(System.TimeSpan, System.ServiceModel.Channels.Message ByRef)+0x4f7
000000000FA8EBF0 000007FEDE5409AE System_ServiceModel_ni!System.ServiceModel.Dispatcher.InputChannelBinder.TryReceive(System.TimeSpan, System.ServiceModel.Channels.RequestContext ByRef)+0x2e
000000000FA8EC50 000007FEDEAC29E2 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ErrorHandlingReceiver.TryReceive(System.TimeSpan, System.ServiceModel.Channels.RequestContext ByRef)+0x646022
000000000FA8ECB0 000007FEDE47C8D6 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.TryTransactionalReceive(System.Transactions.Transaction, System.ServiceModel.Channels.RequestContext ByRef)+0x396
000000000FA8ED70 000007FEDE47BE07 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.TransactedLoop()+0xb7
000000000FA8EDF0 000007FEDE47BD31 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.SyncTransactionalMessagePump()+0x21
000000000FA8EE20 000007FEDE47A829 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.OnStartSyncMessagePump(System.Object)+0x209
000000000FA8EED0 000007FEDB6DE651 System_ServiceModel_Internals_ni!System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x71
000000000FA8EF30 000007FEDB77A260 System_ServiceModel_Internals_ni!System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x9bcd0
000000000FA8EF90 000007FEDF225C26 mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x96
Problem is, this isn't really helping me. I need to find out what is creating all these locks. Does anyone have any suggestions on how I might do that? I have never had an issue that required this level of debugging and to be honest I am not 100% sure what I am doing. If I haven't given enough info please let me know I will happily provide you with whatever else I can.
Thanks
Try using !SyncBlk command of SOS to determine deadlocks if any.
I have erlang application. In this application i run process with spawn(?MODULE, my_foo, [my_param1, my_param2, my_param3]).
And my_foo:
my_foo(my_param1, my_param2, my_param3) ->
...
some code here
...
ok.
When i open etop i see that this my_foo/3 function status: proc_lib:sync_wait/2
Than i try to put exit(self(), normal) in the end of my function, but i see same behavior: proc_lib:sync_wait/2 in etop.
How can i kill or exit process correctly?
Thank you.
Note that exit(Pid, Reason) and exit(Reason) do NOT do the same thing if Pid is the process itself. exit/1 tells the current process to exit - from the inside if you like - while exit/2 sends an exit signal to the process, even if the process is itself. So when you do exit(self(), normal) you are actually sending the normal exit signal to yourself, which is ignored.
In this case putting the exit call at the end of the function should not make any difference as the process automatically dies (with reason normal) when the function with which it was started ends. It seems like the process is suspended somewhere before that.
proc_lib:sync_wait/2 is called inside proc_lib:start/start_link and sits and waits for the spawned process to do proc_lib:init_ack/1/2 to return the return value for start. It would appear that your process does not call init_ack.
Based on the limited information that you give in the question I would suspect that your process hasn't finished running yet.
Normally you don't need to add exit/2 to your process. It will exit automatically when the function has finished running.
You probably have a long running call in some code here that has not finished running. I recommend that you add logging information and see where you are stuck.
I am implementing multi-threading concepts using thread-pooling to pick up messages from queues. There is a necessity that the id/number of the thread which picks up a particular message has to be accessed from outside the thread method. Supposes if there are three threads and they have to pick up 5 messages from the queue. the first thread picks up the first message processes it and it is released. meanwhile the second thread would have picked up some other message and would've started to process it(multi-threading).
Here from some other method, i want to know which thread actually picks up message1, and the consistency has to maintained not only inside the threading function but throughout the application(from outside the thread function also).
i have implemented multi-threading as follows
For m_intThread1 = 0 To m_stuTPConfig.intNumThreads - 1
ThreadPool.QueueUserWorkItem(New WaitCallback(AddressOf **MultiProcessMQ**), m_intThread1)
System.Threading.Thread.Sleep(1000)
Next m_intThread1 'm_intThread
Public Shared Sub MultiProcessMQ(ByVal state As Object)
Dim objParentProcess As New ParentProcess
objParentProcess.ProcessThread(CType(state, String))
If Interlocked.Decrement(CInt(m_stuTPConfig.intNumThreads)) = 0 Then
m_asyncOpsAreDone.Set()
End If 'Interlocked.Decrement(CInt(m_stuTPConfig.intNumThreads)) = 0
End Sub 'MultiProcessMQ()
Public Sub ProcessThread(ByVal strThread As String)
intThrd = Convert.ToInt32(strThread)
Console.WriteLine("Parent Thread started " & strThread)
End Sub
How do i access the variable strThread From another method and maintain its consistency.This is required for logging purposes.
Thanks for formatting. We are using WCFMQAdapter services which is a for configuring the app in accordance with the input and output queues. once I host this service from ProcessThread the control goes to some other file, say a service.vb file wherein all the operations pertaining to the thread are performed and never comes back. The actual usage of threads here are for logging the processing details onto a file in the service.vb file. Now the PROCESSTHREAD method is called. till now the multithreading properties are maintained. once the service is hosted how to pass on these thread info to the service.vb. Or should there be a parent threading/child threading concept? And, if i obtain a threadid using managedThreadId, dynamic thread ids are generated. Are there any chances to obtain them and convert it into user managed ids. say, instead of thread 21,thread 44,thread66,thread89........ can we convert them like thread1, thread 2, thread 3, thread 4? Thanks in Advance :-)
You can use System.Threading.Thread.CurrentThread.ManagedThreadId to get a unique id for each thread. I'd use that to identify your threads.