Bizarre hook behavior 32/64 bit - dll

Question

I'm using a local hook (WH_KEYBOARD) with ms word (OpusApp). Well, as far as I know a 32bit app with a 32bit DLL must work only with 32bit target applications. The weird thing is that the program only works with 64bits apps!!! That is it, only with 64bits APPS! For example, it works with IE 64 but not with IE 32!
The app and dll are 32bit compiled with radstudio XE2, I confirmed the version into PE header.
In 32bit OSs, the app and dll doesn´t work.
I found no solutions on net and see no starting point to solve this weird problem.
The DLL code:
// Exported functions
extern "C" __declspec(dllexport)bool __stdcall InstallMouseHook(unsigned long, void *);
extern "C" __declspec(dllexport)bool __stdcall RemoveMouseHook();
// Callback Procedure Declaration
LRESULT CALLBACK HookProc(int code, WPARAM wParam, LPARAM lParam);
// Global variables
HHOOK HookHandle;
HINSTANCE DllInstance;
typedef void (__stdcall *CALLIT)(int,WPARAM,LPARAM);
CALLIT callIt = NULL;
int WINAPI DllEntryPoint(HINSTANCE hinst, unsigned long reason, void*)
{
DllInstance=hinst;
return 1;
}
bool __stdcall InstallMouseHook(unsigned long pid, void *function)
{
callIt = ( CALLIT ) function;
if (function == NULL) {
ShowMessage("function is null!");
} else if (callIt == NULL) {
ShowMessage("callIt is null!");
}
HookHandle=SetWindowsHookEx(WH_KEYBOARD ,reinterpret_cast<HOOKPROC> (HookProc),DllInstance,pid);
if (HookHandle==NULL)return false;
else return true;
}
bool __stdcall RemoveMouseHook()
{
if(UnhookWindowsHookEx(HookHandle)==0)
{
return false;
}
else return true;
}
LRESULT CALLBACK HookProc(int code, WPARAM wParam, LPARAM lParam)
{
if (code<0) {
return CallNextHookEx(HookHandle,code,wParam,lParam);
}
if (callIt != NULL) {
callIt(code,wParam,lParam);
} else {
ShowMessage("HookProc - no function to execute OR 32/64 bits problem!");
}
//Call the next hook in the chain
return CallNextHookEx(HookHandle,code,wParam,lParam);
}
The EXE calling code:
void __fastcall TfrmMouseHook::btnHookAppDllClick(TObject *Sender)
{
HWND hWindow;
unsigned long pid;
String s = "MouseHookDLL.dll";
DllHandle=LoadLibrary(s.w_str());
MOUSEHOOKFCT_2 InstHook=reinterpret_cast<MOUSEHOOKFCT_2> (GetProcAddress(DllHandle,"InstallMouseHook"));
hWindow = FindWindow(ComboBox1->Text.w_str(),NULL);
if (!hWindow) {
msg("hWindow fail");
return;
}
pid = GetWindowThreadProcessId(hWindow ,0);
if (!pid) {
msg("pid fail");
return;
}
if(!InstHook(pid, (void *) callIt )) {
msg("Unable to install hook!");
} else {
msg(" #### hook INSTALLED! ####");
}
}
CALLIT callIt(code,wParam,lParam) {
frmMouseHook->msg("hook callit: code="+IntToStr(code) +" wparam="+IntToStr(wParam)+" lparam="+IntToStr(lParam) );
}
Call IT is a function pointer to a hooker app function.
Any ideas will be very wellcome!

Answer 1

It is physically impossible for a 32-bit app to install a 32-bit hook DLL and have it executed in 64-bit processes. A 32-bit DLL simply cannot be injected into a 64-bit process. Period. MSDN says this in multiple places, including in the SetWindowsHookEx() documentation:
SetWindowsHookEx can be used to inject a DLL into another process. A
32-bit DLL cannot be injected into a 64-bit process, and a 64-bit DLL
cannot be injected into a 32-bit process. If an application requires
the use of hooks in other processes, it is required that a 32-bit
application call SetWindowsHookEx to inject a 32-bit DLL into 32-bit
processes, and a 64-bit application call SetWindowsHookEx to inject a
64-bit DLL into 64-bit processes. The 32-bit and 64-bit DLLs must have
different names.
Because hooks run in the context of an application, they must match
the "bitness" of the application. If a 32-bit application installs a
global hook on 64-bit Windows, the 32-bit hook is injected into each
32-bit process (the usual security boundaries apply). In a 64-bit
process, the threads are still marked as "hooked." However, because a
32-bit application must run the hook code, the system executes the
hook in the hooking app's context; specifically, on the thread that
called SetWindowsHookEx. This means that the hooking application must
continue to pump messages or it might block the normal functioning of
the 64-bit processes.
If a 64-bit application installs a global hook on 64-bit Windows, the
64-bit hook is injected into each 64-bit process, while all 32-bit
processes use a callback to the hooking application.
The fact that you say your app and DLL do not work on 32-bit OS versions suggests your hooking code is flawed to begin with. But you have not shown enough code to diagnose that one way or the other.

Answer 2

What happens? Besides MSDN or anyone else says, there are some bug in XE6, compiling DLL in newer versions of IDE make this behavior disappears, in fact, the new DLL crash and hook nothing.
As Remy noted, by test I passed a function pointer to the DLL, one wrong thing to do but, when added with the wrong thing done by Embarcadero, sort of functioned.
By now, and I know people will get mad, I put both methods (the wrong and the correct hooks) in the same DLL and in my application and... get crazy... could hook into 32 and 64 bits app with only one DLL.
Don't believe? Install XE6 and try!
And Works in windows 10 as well in windows 7.