windbg - module not loading "dll not found in the image list" - .net-4.0

I am trying to get a proper callstack for an unhandled exception in my VS2010 .net4 application using windbg.
The main program is a console application. This dll loads with it's symbols properly.
In the same directory, I have a dll+matching pdb which won't load.
I am running the app on a windows server 2008 R2, 64 bit (no VS installed). But the app was compiled on 32bit. I am using the winX86 debugger to attach to the process.
I have downloaded sosex that supports .net 4 from
http://www.stevestechspot.com/ (the 32 bit version)
the sos.dll (from "C:\Program Files (x86)\Debugging Tools for Windows (x86)\clr10\sos.dll") version is 6.12.2.633.
Issued the following commands:
sympath+ "...folder of exe and dll"
.loadby sos clr
.load sosex.dll
When running !mk I get the following:
Thread 0:
ESP EIP
00:U 0016ec6c 5f636578 0x5f636578
01:U 0016ec70 05f1380f SN!SN_MedistoreEngine::FetchNotes+0x13f [v:\mp\mp\src\sn\sn_medistoreengine.cpp # 59]
02:M 0016edbc 07dd0db1 SN_Bridge.SNB_bridge.fetch_notes(IOD_Msg*, System.String, System.String, Boolean)(+0x4a IL)(+0x131 Native) [v:\mp\portal\commonutils\src\snb\snb_bridge.cpp, # 142,0]
03:M 0016ef24 055bfde5 SN_Bridge.SNB_bridge.FetchNotesByStudyUId(System.String, System.String, System.Collections.Generic.List`1<SN_Bridge.SNB_StickyNote>)(+0x25 IL)(+0x55 Native) [v:\mp\portal\commonutils\src\snb\snb_bridge.cpp, # 296,0]
04:M 0016efb0 055bfd22
"xxx.Services.StickyNotes.dll" was not found in the image list.
Debugger will attempt to load "xxx.Services.StickyNotes.dll" at given base 00000000.
Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to add module at 00000000
"xxx.ni.Services.StickyNotes.dll" was not found in the image list.
Debugger will attempt to load "xxx.ni.Services.StickyNotes.dll" at given base 00000000.
Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to add module at 00000000
xxx.Services.StickyNotes.StickyNotesLogic.StickyNotesByStudyID(System.String, System.String, System.String)(+0x1d IL)(+0x52 Native)
05:M 0016efcc 055ba0ab SNConsole.Program.Main(System.String[])(+0x101 IL)(+0x24b Native) [D:\Documents and Settings\tamar\My Documents\Visual Studio 2010\Projects\SNConsole\Program.cs, # 48,17]
06:U 0016f02c 72da21db clr+0x21db
07:U 0016f034 72dae021 clr!DllUnregisterServerInternal+0x8025
08:U 0016f090 72dbc58d clr!DllUnregisterServerInternal+0x16591
When running !clrstack I get the following:
PDB symbol for clr.dll not loaded
OS Thread Id: 0x2984 (0)
Child SP IP Call Site
0016edc8 5f636578 [InlinedCallFrame: 0016edc8]
0016edc4 07dd0db1 SN_Bridge.SNB_bridge.fetch_notes(IOD_Msg*, System.String, System.String, Boolean) [v:\mp\portal\commonutils\src\snb\snb_bridge.cpp # 142]
0016ef30 055bfde5 SN_Bridge.SNB_bridge.FetchNotesByStudyUId(System.String, System.String, System.Collections.Generic.List`1<SN_Bridge.SNB_StickyNote>) [v:\mp\portal\commonutils\src\snb\snb_bridge.cpp # 296]
0016efb0 055bfd22 xxx.Services.StickyNotes.StickyNotesLogic.StickyNotesByStudyID(System.String, System.String, System.String)
0016efcc 055ba0ab SNConsole.Program.Main(System.String[]) [D:\Documents and Settings\tamar\My Documents\Visual Studio 2010\Projects\SNConsole\Program.cs # 48]
0016f25c 72da21db [GCFrame: 0016f25c]
As you can see, in both cases I get the file name + row number in the stack, except for the xxx.Services.StickyNotes.dll line.
I've tried:
.realod /f ""c:...\xxx.Services.StickyNotes.dll" - same errors
and
ld "c:...\xxx.Services.StickyNotes.dll" which resulted in
No modules matched
'c:...\Bin\xxx.Services.StickyNotes.dll'
using !sym noisy didn't help, I think it doesn't even try to load the pdb since the module itself isn't loaded.
I can't figure out why this specific dll won't load. SN_Console.exe and SNB_Bridge.dll load without a problem from the same directory.
(I am not concerned with the exception itself, I planted the code that creates is. The issue is about setting up a good debugging environment).
Thanks in advance,
Tamar

I happened to answer this question in person, but here's the documented story in case anyone else stumbles upon it: http://blogs.microsoft.co.il/blogs/sasha/archive/2011/01/16/clr-4-does-not-use-loadlibrary-to-load-assemblies.aspx
The big picture is that CLR 4 doesn't use LoadLibrary to load assemblies, so the debugger can't pick up the DLL and therefore its symbols.

You can hint the debugger the location of the DLL in its address space using:
.reload /f "c:...\xxx.Services.StickyNotes.dll=image_base_address"
image_base_address is the start of the address range which this DLL is mapped in the process. You can find this value in the DLL Pane of Process Explorer.
I wrote about it with a little more detail here.

Related

debugging vxworks loadModule failure

I have a VxWorks Image Project project without a File-System on MPC5200B, using DIAB tool-chain.
I need to dynamically load a module from flash.
I allocated memory on my stack char myTemporaryModuleData[MAX_MODULE_SIZE]
and filled it with the module data from Flash.
(checked that the binary data is intact)
then i create a memDevice('/mem/mem01', myTemporaryModuleData, moduleReadLength)
open the psuedo-stream int fdModuleData = open("/mem/mem01", O_RDONLY, 777);
when i run int mId = loadModule(fdModuleData, LOAD_ALL_SYMBOLS);
did not see anything in the console after running loadModule();
but mId = 0 which indicates failure :(.
getErrno() returned 0x3D0004 (S_objLib_OBJ_TIMEOUT)
NOTE: it didn't take long at all to fail => timeout?
i tried replacing the module with a simple void foo() { printf(...); } module but still failes with same issue.
tried loading an .out instead of .o
unfortunately, nothing got me nowhere,
How can i know what caused it to fail? (log, last_error, anything i should check?)
FOUND IT.
Apparently, it was a mistake in the data read from the flash.
What I can contribute is that 'loadModule()' from memDrv device is possible and working.

MQQueueManager Constructor throwing FileNotFoundException

I have the following vb.net code:
Imports IBM.WMQ
[...]
MQEnvironment.Hostname = hostName
MQEnvironment.Port = portNumber
MQEnvironment.Channel = channelName
queueManager = New MQQueueManager(queueManagerName) ' error here
which is throwing the following error:
System.IO.FileNotFoundException occurred
FileName=C:\Users\User\Documents\Visual Studio 2012\Projects\[...]\bin\Debug\mqclient.ini
HResult=-2147024894
Message=Could not find file 'C:\Users\User\Documents\Visual Studio 2012\Projects\[...]\bin\Debug\mqclient.ini'.
Source=mscorlib
StackTrace:
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
I am not using any ini files in the construction of my queue manager, so does anyone have any idea what's going on - why is it even looking for one, and why in the same directory as the program? I have installed the MQ client, and afaik I have all the environmental variables, etc. set up properly.
Thanks for any help you can give
Is that an unhandled or a first chance exception? Internally, the MQ .net layer will try to read a MQClient.ini but should function quite happily without it. It reads the file for compatibility with the C client, and can handle some of the MQClient.ini stanzas. I would not have expected an absence of such a file to cause problems, but it will try to open it internally. Was that the full callstack, as I'd have expected some MQ libraries on the stack otherwise.

Debugging a CLR20r3 System.InvalidOperationException in production

We have a production VS2008 VB.Net application that's installed on many (hundreds) of client computers. A customer recently installed the application on his Win 7 desktop. The installation completed without errors. However, when he tries to run the application, he receives the following error "Application Name has stopped working". I've included the contents of the event log file below. In my research, it seems this type of error can be caused by a number of things - bad or missing dependency, missing .net framework, permissions, a faulty icon, missing font.
My question is this: Is there a way we can effectively troubleshoot this problem in a production environment? I know the application installation is good; even this customer can run the application on another computer. The appropriate .net framework shows as being installed. I could have him reinstall the .net framework but would like to get a better handle on what's going on.
Here is the contents of the event log file:
Version=1
EventType=CLR20r3
EventTime=130163188478462012
ReportType=2
Consent=1
ReportIdentifier=31c14aab-daae-11e2-b34f-d48564179a03
WOW64=1
Response.type=4
Sig[0].Name=Problem Signature 01
Sig[0].Value=planguru2013.exe
Sig[1].Name=Problem Signature 02
Sig[1].Value=3.0.0.2
Sig[2].Name=Problem Signature 03
Sig[2].Value=51b6342b
Sig[3].Name=Problem Signature 04
Sig[3].Value=PlanGuru2013
Sig[4].Name=Problem Signature 05
Sig[4].Value=3.0.0.2
Sig[5].Name=Problem Signature 06
Sig[5].Value=51b6342b
Sig[6].Name=Problem Signature 07
Sig[6].Value=a0
Sig[7].Name=Problem Signature 08
Sig[7].Value=c6
Sig[8].Name=Problem Signature 09
Sig[8].Value=System.InvalidOperationException
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.1.7601.2.1.0.768.3
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1033
UI[2]=C:\Program Files (x86)\New Horizon\PlanGuru 2013\PlanGuru2013.exe
UI[3]=PlanGuru 2013 has stopped working
UI[4]=Windows can check online for a solution to the problem.
UI[5]=Check online for a solution and close the program
UI[6]=Check online for a solution later and close the program
UI[7]=Close the program
LoadedModule[0]=C:\Program Files (x86)\New Horizon\PlanGuru 2013\PlanGuru2013.exe
LoadedModule[1]=C:\Windows\SysWOW64\ntdll.dll
LoadedModule[2]=C:\Windows\SYSTEM32\MSCOREE.DLL
LoadedModule[3]=C:\Windows\syswow64\KERNEL32.dll
LoadedModule[4]=C:\Windows\syswow64\KERNELBASE.dll
LoadedModule[5]=C:\Windows\system32\apphelp.dll
LoadedModule[6]=C:\Windows\AppPatch\AcGenral.DLL
LoadedModule[7]=C:\Windows\SysWOW64\sechost.dll
LoadedModule[8]=C:\Windows\syswow64\msvcrt.dll
LoadedModule[9]=C:\Windows\syswow64\RPCRT4.dll
LoadedModule[10]=C:\Windows\syswow64\SspiCli.dll
LoadedModule[11]=C:\Windows\syswow64\CRYPTBASE.dll
LoadedModule[12]=C:\Windows\syswow64\SHLWAPI.dll
LoadedModule[13]=C:\Windows\syswow64\GDI32.dll
LoadedModule[14]=C:\Windows\syswow64\USER32.dll
LoadedModule[15]=C:\Windows\syswow64\ADVAPI32.dll
LoadedModule[16]=C:\Windows\syswow64\LPK.dll
LoadedModule[17]=C:\Windows\syswow64\USP10.dll
LoadedModule[18]=C:\Windows\system32\UxTheme.dll
LoadedModule[19]=C:\Windows\system32\WINMM.dll
LoadedModule[20]=C:\Windows\system32\samcli.dll
LoadedModule[21]=C:\Windows\syswow64\ole32.dll
LoadedModule[22]=C:\Windows\syswow64\OLEAUT32.dll
LoadedModule[23]=C:\Windows\system32\MSACM32.dll
LoadedModule[24]=C:\Windows\system32\VERSION.dll
LoadedModule[25]=C:\Windows\syswow64\SHELL32.dll
LoadedModule[26]=C:\Windows\system32\sfc.dll
LoadedModule[27]=C:\Windows\system32\sfc_os.DLL
LoadedModule[28]=C:\Windows\system32\USERENV.dll
LoadedModule[29]=C:\Windows\system32\profapi.dll
LoadedModule[30]=C:\Windows\system32\dwmapi.dll
LoadedModule[31]=C:\Windows\syswow64\SETUPAPI.dll
LoadedModule[32]=C:\Windows\syswow64\CFGMGR32.dll
LoadedModule[33]=C:\Windows\syswow64\DEVOBJ.dll
LoadedModule[34]=C:\Windows\syswow64\urlmon.dll
LoadedModule[35]=C:\Windows\syswow64\WININET.dll
LoadedModule[36]=C:\Windows\syswow64\iertutil.dll
LoadedModule[37]=C:\Windows\syswow64\CRYPT32.dll
LoadedModule[38]=C:\Windows\syswow64\MSASN1.dll
LoadedModule[39]=C:\Windows\system32\MPR.dll
LoadedModule[40]=C:\Windows\system32\IMM32.DLL
LoadedModule[41]=C:\Windows\syswow64\MSCTF.dll
LoadedModule[42]=C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscoreei.dll
LoadedModule[43]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
LoadedModule[44]=C:\Windows\WinSxS \x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.6195_none_d09154e044272b9a\MSVCR80.dll
LoadedModule[45]=C:\Windows\assembly\NativeImages_v2.0.50727_32\mscorlib\7150b9136fad5b79e88f6c7f9d3d2c39\mscorlib.ni.dll
LoadedModule[46]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorsec.dll
LoadedModule[47]=C:\Windows\syswow64\WINTRUST.dll
LoadedModule[48]=C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.7601.17514_none_ec83dffa859149af\COMCTL32.dll
LoadedModule[49]=C:\Windows\system32\CRYPTSP.dll
LoadedModule[50]=C:\Windows\system32\rsaenh.dll
LoadedModule[51]=C:\Windows\syswow64\imagehlp.dll
LoadedModule[52]=C:\Windows\system32\ncrypt.dll
LoadedModule[53]=C:\Windows\system32\bcrypt.dll
LoadedModule[54]=C:\Windows\SysWOW64\bcryptprimitives.dll
LoadedModule[55]=C:\Windows\system32\GPAPI.dll
LoadedModule[56]=C:\Windows\system32\cryptnet.dll
LoadedModule[57]=C:\Windows\syswow64\WLDAP32.dll
LoadedModule[58]=C:\Windows\system32\SensApi.dll
LoadedModule[59]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System \369f8bdca364e2b4936d18dea582912c\System.ni.dll
LoadedModule[60]=C:\Windows\assembly\NativeImages_v2.0.50727_32\Microsoft.VisualBas#\7366a39c36523a084bc11c230929ff92\Microsoft.VisualBasic.ni.dll
LoadedModule[61]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorjit.dll
LoadedModule[62]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Drawing\eead6629e384a5b69f9ae35284b7eeed\System.Drawing.ni.dll
LoadedModule[63]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Windows.Forms\30e3a21202000677d0a9270572251477\System.Windows.Forms.ni.dll
LoadedModule[64]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Runtime.Remo#\90b89f6e8032310e9ac72a309fd49e83\System.Runtime.Remoting.ni.dll
LoadedModule[65]=C:\Windows\syswow64\ws2_32.dll
LoadedModule[66]=C:\Windows\syswow64\NSI.dll
LoadedModule[67]=C:\Windows\system32\mswsock.dll
LoadedModule[68]=C:\Windows\System32\wshtcpip.dll
LoadedModule[69]=C:\Windows\System32\wship6.dll
LoadedModule[70]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Configuration\764f15e86c82662e977bd418bd6318c1\System.Configuration.ni.dll
LoadedModule[71]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Xml\f687c43e9fdec031988b33ae722c4613\System.Xml.ni.dll
LoadedModule[72]=C:\Windows\system32\RpcRtRemote.dll
LoadedModule[73]=C:\Program Files (x86)\New Horizon\PlanGuru 2013\FarPoint.Win.Spread.dll
LoadedModule[74]=C:\Program Files (x86)\New Horizon\PlanGuru 2013\FarPoint.Win.dll
LoadedModule[75]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Data\dd20416f723ee13ffb4173ec1afc4ec4\System.Data.ni.dll
LoadedModule[76]=C:\Windows\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll
LoadedModule[77]=C:\Windows\WinSxS\x86_microsoft.windows.gdiplus_6595b64144ccf1df_1.1.7601.17825_none_72d273598668a06b\gdiplus.dll
LoadedModule[78]=C:\Windows\system32\WindowsCodecs.dll
LoadedModule[79]=C:\Program Files (x86)\New Horizon\PlanGuru 2013\FarPoint.Win.TextRenderer.dll
LoadedModule[80]=C:\Program Files (x86)\New Horizon\PlanGuru 2013\FarPoint.CalcEngine.dll
FriendlyEventName=Stopped working
ConsentKey=CLR20r3
AppName=PlanGuru 2013
AppPath=C:\Program Files (x86)\New Horizon\PlanGuru 2013\PlanGuru2013.exe
ReportDescription=Stopped working
To get the memory crash dump of the exception you can set the following registry keys to tell windows error reporting to keep the crash dump on the pc.
HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\ForceQueue = 0x1
HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\Consent\DefaultConsent = 0x1
Now, look in:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue
Your crash should be there with a managed dump. Visual Studio should be able to open the crash dump. If not use WinDbg + SOS.dll (which is located in the .net framework installation folder). This will give you the callstack of the exception.
Sig[6].Name=Problem Signature 07
Sig[6].Value=a0
ildasm /tokens "PlanGuru2013.exe" /out=libcode.il
lookup failing method in IL 060000+ val above
.method /060000a0/ private hidebysig

Interpreting data from crash dump using winDbg

I am trying to debug a WCF service which is crashing from time to time. I have created a crash dump file using adplus, that was the easy part, I used this command.
adplus.exe -crash -pmn myservicehost.exe -o c:\dump
I am opening the file .dmp file with winDbg x64 version 6.2.9200. When I look at the threads !Threads I see there are a bunch of threads that that are waiting for a callback to complete (I think).
0:031> ~~[1b00]s
ntdll!NtWaitForMultipleObjects+0xa:
00000000`778d18ca c3 ret
0:029> ~~[1b00]s
ntdll!NtWaitForMultipleObjects+0xa:
00000000`778d18ca c3 ret
Eventually it crashes. Here is the output when I look at the call stack from that exception using !PrintException /d -nested 00000002814ad6d0
Exception object: 00000002814af4c8
Exception type: System.Runtime.CallbackException
Message: A user callback threw an exception. Check the exception stack and inner exception to determine the callback that failed.
InnerException: System.ServiceModel.CommunicationObjectAbortedException, Use !PrintException 00000002814ad6d0 to see more.
StackTrace (generated):
SP IP Function
000000000FA89D40 000007FEDD9AD3E2 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnClosed()+0x262
000000000FA8C240 000007FEDD93759D System_ServiceModel_ni!System.ServiceModel.ServiceHostBase.OnClosed()+0x6d
000000000FA8C290 000007FEDD9433D0 System_ServiceModel_ni!System.ServiceModel.ServiceHost.OnClosed()+0x10
000000000FA8C2C0 000007FEDE185B86 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Abort()+0x2b6
000000000FA8C3C0 000007FE8ABAC89C MyCompany_WcfApp_WcfAppServiceHost!MyCompany.WcfApp.WcfAppServiceHost.WcfAppServiceHost.FaultedServiceHandler(System.Object, System.EventArgs)+0x26c
000000000FA8C5F0 0000000000000000 mscorlib_ni!System.EventHandler.Invoke(System.Object, System.EventArgs)+0x1
000000000FA8C5F0 000007FEDE184E9A System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnFaulted()+0x1ca
000000000FA8C670 000007FEDE184784 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Fault()+0x94
000000000FA8C6E0 000007FEDE184E9A System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnFaulted()+0x1ca
000000000FA8C760 000007FEDE184784 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Fault()+0x94
000000000FA8C7D0 000007FEDE184E9A System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.OnFaulted()+0x1ca
000000000FA8C850 000007FEDE184784 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Fault()+0x94
000000000FA8C8C0 000007FEDE475407 System_ServiceModel_ni!System.ServiceModel.Channels.MsmqInputChannelBase.TryReceive(System.TimeSpan, System.ServiceModel.Channels.Message ByRef)+0x4f7
000000000FA8EBF0 000007FEDE5409AE System_ServiceModel_ni!System.ServiceModel.Dispatcher.InputChannelBinder.TryReceive(System.TimeSpan, System.ServiceModel.Channels.RequestContext ByRef)+0x2e
000000000FA8EC50 000007FEDEAC29E2 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ErrorHandlingReceiver.TryReceive(System.TimeSpan, System.ServiceModel.Channels.RequestContext ByRef)+0x646022
000000000FA8ECB0 000007FEDE47C8D6 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.TryTransactionalReceive(System.Transactions.Transaction, System.ServiceModel.Channels.RequestContext ByRef)+0x396
000000000FA8ED70 000007FEDE47BE07 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.TransactedLoop()+0xb7
000000000FA8EDF0 000007FEDE47BD31 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.SyncTransactionalMessagePump()+0x21
000000000FA8EE20 000007FEDE47A829 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.OnStartSyncMessagePump(System.Object)+0x209
000000000FA8EED0 000007FEDB6DE651 System_ServiceModel_Internals_ni!System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x71
000000000FA8EF30 000007FEDB77A260 System_ServiceModel_Internals_ni!System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x9bcd0
000000000FA8EF90 000007FEDF225C26 mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x96
Problem is, this isn't really helping me. I need to find out what is creating all these locks. Does anyone have any suggestions on how I might do that? I have never had an issue that required this level of debugging and to be honest I am not 100% sure what I am doing. If I haven't given enough info please let me know I will happily provide you with whatever else I can.
Thanks
Try using !SyncBlk command of SOS to determine deadlocks if any.

COM exception on "custom component" - how to identify DLL?

We have a large legacy VB app made up of a number of DLLs (a couple of dozen or so), all installed into a single COM+ Server Application. Every now and then, something happens that causes dllhost.exe to keel over (and automatically restart), leaving this message in the Windows Application Event log...
The system has called a custom component and that component has
failed and generated an exception. This indicates a problem with the
custom component. Notify the developer of this component that a failure has
occurred and provide them with the information below.
Server Application ID: {8CC02F18-2733-4A17-9E5C-1A70CB6B6977}
Server Application Instance ID: {1940A147-8A5E-45FA-86FE-DAF92A822597}
Server Application Name: MyTestApp
The serious nature of this error has caused the process to terminate.
Exception: C0000005
Address: 0x758DA3DA
Source: Complus
Event ID: 4786
Level: Error
Along side this is another log, specifically on dllhost.exe...
Faulting application name: dllhost.exe, version: 6.0.6000.16386, time stamp: 0x4549b14e
Faulting module name: msvcrt.dll, version: 7.0.6002.18005, time stamp: 0x49e0379e
Exception code: 0xc0000005
Fault offset: 0x0000a3da
Faulting process id: 0x83c
Faulting application start time: 0x01cb50c507ee0166
Faulting application path: %11
Faulting module path: %12
Report Id: %13
I know it's flagging a failure in the C runtime (msvcrt), but ideally I need to trace this back into the DLL that's called into msvcrt (probably with bad data/parameters). So without installing a debugger, is there any way to identify the DLL that causes this? I'm trying to see if there's a memory dump anywhere I can use to analyse offline - and thus tie the Address to something specific. But without that, I'm not sure that's possible. Can the COM subsystem be told to generate a minidump when a hosted application crashes? (yes it can [probably] - there's a checkbox on the 'Dump' tab).
This is on Windows Server 2008 R1 32-bit (but also be interested for Server 2003 as well).
It doesn't affect availability of the app -- COM+ simply restarts dllhost and the application continues, but it is an inconvienience that would be useful to fix.
Edit Okay, I've got a crash dump, I've got windbg, but it's not helping. Not sure if I'm being thick (a possibility) or something else :-) Output of !analyze -v is below , but it's not showing me anything in our DLLs, although it looks like it hasn't been able to resolve FAULTING_IP? I'm not sure where to turn next.
I'm wondering if any of my pdb's are dodgy and be worth generating new ones -- hooked into Microsoft's symbol server, so they shouldn't be, but not sure for what module it's (apparently) reporting wrong symbols for (BUGCHECK_STR and PRIMARY_PROBLEM_CLASS) (or are these symbols on the server the code was originally running on?). Would it be better to put the PDBs on the server itself?
If not, any other ideas? I've used windbg briefly before, but I'm no regular user of it, so maybe there's some more incantations I need to type to dig deeper? Guidance welcome :-)
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
+5c112faf02e0d82c
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 00000f1c
DEFAULT_BUCKET_ID: WRONG_SYMBOLS
PROCESS_NAME: dllhost.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
MOD_LIST: <ANALYSIS/>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0xf1c (0)
Current frame:
ChildEBP RetAddr Caller,Callee
LAST_CONTROL_TRANSFER: from 77b15620 to 77b15e74
PRIMARY_PROBLEM_CLASS: WRONG_SYMBOLS
BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS
STACK_TEXT:
0022fa68 77b15620 77429884 00000064 00000000 ntdll!KiFastSystemCallRet
0022fa6c 77429884 00000064 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
0022fadc 774297f2 00000064 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xbe
0022faf0 778e2c44 00000064 ffffffff 00e42374 kernel32!WaitForSingleObject+0x12
0022fb0c 778e2e32 00060848 0022fb5b 00000000 ole32!CSurrogateProcessActivator::WaitForSurrogateTimeout+0x55
0022fb24 00e413a4 0022fb40 00000000 00061d98 ole32!CoRegisterSurrogateEx+0x1e9
0022fcb0 00e41570 00e40000 00000000 00061d98 dllhost!WinMain+0xf2
0022fd40 7742d0e9 7ffde000 0022fd8c 77af19bb dllhost!_initterm_e+0x1a1
0022fd4c 77af19bb 7ffde000 dc2ccd29 00000000 kernel32!BaseThreadInitThunk+0xe
0022fd8c 77af198e 00e416e6 7ffde000 ffffffff ntdll!__RtlUserThreadStart+0x23
0022fda4 00000000 00e416e6 7ffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
STACK_COMMAND: .cxr 00000000 ; kb ; dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~0s; .ecxr ; kb
FOLLOWUP_IP:
dllhost!WinMain+f2
00e413a4 ff15a410e400 call dword ptr [dllhost!_imp__CoUninitialize (00e410a4)]
SYMBOL_STACK_INDEX: 6
SYMBOL_NAME: dllhost!WinMain+f2
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: dllhost
IMAGE_NAME: dllhost.exe
DEBUG_FLR_IMAGE_TIMESTAMP: 4549b14e
FAILURE_BUCKET_ID: WRONG_SYMBOLS_80000003_dllhost.exe!WinMain
BUCKET_ID: APPLICATION_FAULT_WRONG_SYMBOLS_dllhost!WinMain+f2
Do you have symbols for the VB dlls? Symbols are important to get the call-stack. I hope you have correct symbols. You can use ld * and then lme which should get you list of symbols that did not match within windbg. Also set the symbol path for MS symbols and as well as for your custom code using _NT_SYMBOL_PATH
One of the easiest option is to load the dump within DebugDiag which should give you reason for the failure along with call-stack. DebugDiag has debugger extensions for Complus.
And here is a command to native call stack for all the threads
~*ek
and this one switch to the current exception
.ecxr
Debug Mon / WinDbg is the best way to troubleshoot this issue.
you should be able to use the modules list in winDbg, or the lm command to list the loaded modules. The stack trace should then tell you which DLLs are involved. This should be possible even without the symbols for the process/dll.