Debugging unmanaged callback problems - com

Question

Apologies in advance here for the length of the question.
I have a closed source and undocumented COM object - an unmanaged DLL - that I'm attempting to integrate into a Windows service written in C#. The COM object wraps access to some hardware that the service needs to interact with.
I'm not able to get interface documentation or source for the object. All I have to go on is the object itself, three [closed source undocumented] clients that interact with the COM object, and a fair amount of domain specific knowledge.
So far this has been a very tough nut to crack - one week and counting.
I was able to obtain the object's CLSID from the registry - this allowed me to instantiate it in the service.
The next step was to find the IIDs for the interface(s) that I need to use. The particular methods that I was looking for are not exported. I don't have PDBs. There doesn't appear to be any typelib info and the OLE-COM Object Viewer refuses to open the COM object. IDispatch is not implemented either, so it has been a matter of digging. I eventually succeeded in identifying two IIDs by manually searching the binaries for GUIDs and eliminating unique and/or known GUIDs. At this point I'm confident that the IIDs are correct.
The IIDs are obviously useless without corresponding method info. For that I was forced to resort to reversing with IDA. Correlating references to the GUIDs with my knowledge of the hardware functions and the rough disassembly allowed me to make some educated guesses about the structure and purpose of the interfaces.
Now I'm at the point where I need to attempt to use the interfaces to interact with the hardware... and this is where I'm stuck.
From the disassembly, I know that the first method I have to call looks like this:
HRESULT __stdcall SetStateChangeCallback(LPVOID callback);
The callback signature looks something like this:
HRESULT (__stdcall *callbackType)(LPVOID data1, LPVOID data2)
Here is my service code:
[ComImport, System.Security.SuppressUnmanagedCodeSecurity,
Guid(...),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
private interface AccessInterface
{
[PreserveSig]
int SetStateChangeCallback(IntPtr callbackPtr);
...
}
[UnmanagedFunctionPointerAttribute(CallingConvention.StdCall)]
private delegate int OnStateChangeDelegate(IntPtr a, IntPtr b);
private int OnStateChange(IntPtr a, IntPtr b)
{
Debug("***** State change triggered! *****");
}
private Guid _typeClsid = new Guid(...);
private Guid _interfaceIid = new Guid(...);
private object _comObj = null;
private AccessInterface _interface = null;
private OnStateChangeDelegate _stateChangeDelegate = null;
private IntPtr _functionPtr = IntPtr.Zero;
private void InitHardware()
{
Type t = Type.GetTypeFromCLSID(_typeClsid);
_comObj = Activator.CreateInstance(t);
if (_comObj == null)
{
throw new NullReferenceException();
}
_interface = _comObj as AccessInterface;
if (_interface == null)
{
throw new NullReferenceException();
}
_stateChangeDelegate = new OnStateChangeDelegate(OnStateChange);
_functionPtr = Marshal.GetFunctionPointerForDelegate(_stateChangeDelegate);
int hr = _interface.SetStateChangeCallBack(_functionPtr);
// hr (HRESULT) == 0, indicating success
}
Now, I can run this code successfully but only if I pass IntPtr.Zero to SetStateChangeCallBack(). If I pass a real reference, the service crashes within a matter of seconds after calling SetStateChangeCallBack() - presumably when the COM object tries to invoke the callback for the first time - with exception code 0xc0000005.
The fault offset is consistent. With the aid of IDA and the previously generated disassembly I was able to identify the area where the problem occurs:
06B04EF7 loc_6B04EF7: ; CODE XREF: 06B04F49j
06B04EF7 lea eax, [esp+0Ch]
06B04EFB push eax
06B04EFC mov ecx, ebx
06B04EFE call near ptr unk_6B06660
06B04F03 test eax, eax
06B04F05 jl short loc_6B04F4B
06B04F07 mov esi, [esp+0Ch]
06B04F0B test esi, esi
06B04F0D jz short loc_6B04F45
06B04F0F push 36h
06B04F11 lea ecx, [esp+18h]
06B04F15 push 0
06B04F17 push ecx
06B04F18 call near ptr unk_6B0F960
06B04F1D mov edx, [esp+1Ch]
06B04F21 push edx
06B04F22 lea eax, [esp+24h]
06B04F26 push esi
06B04F27 push eax
06B04F28 call near ptr unk_6B0F9E0
06B04F2D push esi
06B04F2E call near ptr unk_6B0C8D2
06B04F33 mov eax, [edi+4]
06B04F36 mov ecx, [eax]
06B04F38 add esp, 1Ch
06B04F3B lea edx, [esp+14h]
06B04F3F push edx
06B04F40 push eax
06B04F41 mov eax, [ecx] ; Crash here!
06B04F43 call eax
06B04F45
06B04F45 loc_6B04F45: ; CODE XREF: 06B04F0Dj
06B04F45 cmp dword ptr [edi+28h], 0
06B04F49 jnz short loc_6B04EF7
06B04F4B
06B04F4B loc_6B04F4B: ; CODE XREF: 06B04F05j
06B04F4B pop esi
06B04F4C pop ebx
06B04F4D pop edi
06B04F4E add esp, 40h
06B04F51 retn
The crash is at offset 0x06B04F41 (ie. "mov eax, [ecx]").
Corresponding pseudo code function from the disassembly (note assembler above starts at the do loop):
void __thiscall sub_10004EE0(int this)
{
int v1; // edi#1
void *v2; // esi#4
void *v3; // [sp+4h] [bp-40h]#3
int v4; // [sp+8h] [bp-3Ch]#5
char v5; // [sp+Ch] [bp-38h]#5
v1 = this;
if ( *(_DWORD *)(this + 4) )
{
if ( *(_DWORD *)(this + 40) )
{
do
{
if ( sub_10006660(v1 + 12, (int)&v3) < 0 )
break;
v2 = v3;
if ( v3 )
{
memset(&v5, 0, 0x36u);
unknown_libname_44(&v5, v2, v4);
j_j__free(v2);
// Crash on this statement!
(*(void (__stdcall **)(_DWORD, char *))**(void (__stdcall ****)(_DWORD, _DWORD))(v1 + 4))(
*(_DWORD *)(v1 + 4),
&v5);
}
}
while ( *(_DWORD *)(v1 + 40) );
}
}
}
I'm convinced that I am not passing the function pointer to the COM object correctly, but I'm stuffed if I can figure out how to do it properly. I've tried [in order of desperation!]:
_functionPtr
_functionPtr.ToPointer() [as void* param]
_functionPtr.ToInt32() [as int param]
_stateChangeDelegate [as OnStateChangeDelegate param]
OnStateChange [as OnStateChangeDelegate param]
using CallingConvention.Cdecl for the delegate
adding static qualifier to variables and functions
changing signature of the callback (including removing the return value, changing the parameters to ints, modifying the number of parameters)
adding a level of indirection [by storing _functionPtr.ToInt32() in a block of memory allocated with Marshal.AllocCoTaskMem()]
In some cases the changes triggered different crash locations... like crashes in ntdll, or at 06B04F36. In most cases the crash is as described above - at 06B04F41.
When I attach IDA Pro to the process it looks like the address of my callback is going into EAX at 06B04F40, and the address that the COM object attempts to use has a fixed offset from that. For example:
EAX (correct address) = 000A1392
ECX (used address) = 0A1378B8
The last 4 digits of ECX are always 78B8.
So again, I think I'm not passing the delegate or function pointer correctly but I'm not sure how to do it. I guess the fact that the service is running in a WOW64 environment could also be having an impact.
My question: what would you suggest I do to (1) get more information about the problem and/or (2) solve the problem?
Keep in mind I don't have access to any source code except the full code for the C# service. I'm using the free version of IDA Pro so I don't seem to be able to do anything more useful than reverse to pseudo code or attach to the process and catch the crash exception. It is not possible to run the service from VS in debug mode so I really only have logging on that side... not that I think it would be much good as the problem is triggering in the unmanaged code where I don't have compilable/easily-readable source. Maybe I'm wrong?
Thank you sincerely for your advice!
Edit:
Well, after another day bashing my head against the problem I figured if I couldn't succeed from C# I would try and create a minimal C++ test application to do what the service has to do... and I was successful!
IAccessInterface : public IUnknown
{
public:
virtual HRESULT STDMETHODCALLTYPE SetCallback(
/* [in] */ LPVOID pCallBack) = 0;
virtual HRESULT STDMETHODCALLTYPE SetDevice(
/* [in] */ char* context1,
/* [in] */ LPVOID context2,
/* [in] */ LPVOID context3) = 0;
virtual HRESULT STDMETHODCALLTYPE CloseDevice() = 0;
};
IAccessInterface* pInterface;
int __stdcall CallbackImpl(char* context, char* data)
{
printf("Callback succeeded!\r\n");
return 0;
}
void CleanUp(bool deviceOpen)
{
if (pInterface != NULL)
{
if (deviceOpen)
{
pInterface->SetCallback(NULL);
pInterface->CloseDevice();
}
pInterface->Release();
pInterface = NULL;
}
CoUninitialize();
}
int _tmain(int argc, _TCHAR* argv[])
{
GUID objClsid = GUID();
GUID interfaceIid = GUID();
CoInitialize(NULL);
int hr = CoCreateInstance(objClsid, 0, 1, interfaceIid, (void**)&pInterface);
if (!pInterface || !SUCCEEDED(hr))
{
CleanUp(false);
return 1;
}
LPVOID ptr = &callbackImpl;
LPVOID ptr2 = &ptr;
hr = pInterface->SetCallback(&ptr2);
if (!SUCCEEDED(hr))
{
CleanUp(false);
return 1;
}
char* context1 = "a_device_identifier";
hr = pInterface->SetDevice(context1, NULL, NULL);
if (!SUCCEEDED(hr))
{
CleanUp(false);
}
Sleep(30000); // give time for device to initialise and trigger callbacks (testing only)
// clean up
CleanUp(true);
return 0;
}
So now I just need to find a way to replicate the following three lines with equivalent C#:
LPVOID ptr = &CallbackImpl;
LPVOID ptr2 = &ptr;
hr = pInterface->SetCallback(&ptr2);
It seems unnecessary (even suspicious) that so many levels of indirection would be required. Maybe I haven't fully understood the disassembly. At this point the most important thing is that it works.
So any comments about how to achieve this from C# would be welcome!