How to low-level debug a COM DLL (Firefox plugin) method via C++ code without debug info? - dll

Given the following:
the 32-bit DLL code file of some old Firefox plugin (i.e. a DLL containing among other a Typelib, XSD and XSL entries), without source code or debug info, originally coded in C++ and compiled with Visual Studio,
the name and parameters of an exported function/method in this DLL (a function of the Firefox plugin, accessable in JS code),
Visual Studio Community 2013 running on Windows 7,
experience in C++ development, but not with COM or Firefox,
experience with debugging Intel assembler code,
a code license which does not prohibit disassembling the DLL,
I would like to do this: Load the DLL into some C++ code, and step on CPU level into the code of the function to find out what it exactly does.
Can you give me any hint on where to start and how get this done? I guess the DLL may need some Firefox-specific initialization before I can call the function which I would like to debug. Could this be done with the Firefox SDK, without source code and debug info for the DLL? Or may I succeed in "nakedly" loading the DLL, finding the entry point of the - rather simple - function (how?) and calling it?
Thanks for any hints.

If no pdb file or source code, it is hard for you to debug the dll file, since the debugger loads debugging information from the PDB file and uses it to locate symbols or relate current execution state of a program source code. Visual Studio uses PDB files as its primary file format for debugging information during debugging. If no those files, you couldn't debug that library.
Update:
We are dynamically loading a dll to one project using LoadLibrary() function, but if you want to step into your dll file, it really require the pdb file. A simple sample is that you could create and place one pdb file in the same folder as one simple custom dll library project located. I think Visual Studio will automatically search the directory and load them, you could find the information in your Debug modules windows.
The following case is not the same issue as yours, but it also shared us that it would load the pdb file if the dll file was really called by one project/process:
Does winbase::LoadLibrary() load .pdbs?

Related

SDL2_ttf can't find entry point unless libfreetype-6.dll is in proj folder

I am working with SDL2 and its add-on libraries SDL2_image and SDL2_ttf, using Visual Studio 2017. The libraries, and the .dll files that come with them, are in another folder (C:\SDL2.0\lib\x86), which is in the system path.
When I run, it fails, with this error message:
The procedure entry point InterlockedCompareExchange#12 could not be located in the dynamic link library C:\SDL2.0\lib\x86\SDL2.ttf.dll.
I can fix this by putting libfreetype-6.dll, which comes with SDL2_ttf, in the same folder as the .vcxproj file; or in the Debug folder. I can also fix it by putting the .dll into c:\windows\SysWOW64. But I want to distribute my code, and I don't want to put that file in each folder or require users to have admin access (to access c:\windows\SysWOW64); I want Windows to find it in the PATH, as it does with the other .dll files it's using here.
A few things I tried as I looked around the web for solution (to no effect):
Recompiling libfreetype-6.dll
Downloading the latest versions of all associated libraries
Rearranging the .lib files in Project Properties, Linker, Input, Additional Dependencies. Admittedly I may not have tried all possible arrangements as there are several dependencies
regsvr32 libfreetype-6.dll. This led to a different error message:
The module "libfreetype-6.dll" may not be compatible with the version of Windows that you're running. Check if the module is compatible with an x86 (32-bit) or x64 (64-bit) version of regsvr32.exe.
I saw here that maybe I should use the version of regsvr32 in the system32 folder; when I do that, I get
The module "libfreetype-6.dll" was loaded but the entry-point DllRegisterServer was not found. Make sure that "libfreetype-6.dll" is a valid DLL or OCX file and then try again.
So: is there a way to get the program to find libfreetype-6.dll in another folder in the PATH, and eliminate the error message about the procedure entry point?
The program I'm testing on now is from the TrueType tutorial from the LazyFoo website (source).

Getting out of DLL Hell with Microsoft.VC90.CRT?

I've built a inproc com server dll which I can package as 1 file or many via the build utility py2exe. When I allow all the dependencies to remain external, I have no issues, but bundling as 1 file produces problems.
When the dll is utilized (either registering it or instantiating a com object from it), it immediately loads MSVCR90.DLL from the path c:\windows\winsxs\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.6871_none_50944e7cbcb706e5\MSVCR90.DLL no matter what I do, I can't change that. There is no information that I can find (using Dependency Walker) to indicate what is causing that to load. It just happens magically...
Then, later on it loads that dll again via an explicit call to LoadLibraryA("MSVCR90.dll") (part of some py2exe black box?), but this time it does not look into the winsxs manifests / directory. Instead it looks to the system path and/or will respect a dll redirection. That's when the problem occurs. If I set the system path to start with c:\windows\winsxs\x86_microsoft.vc90.crt...\ it will load the exact same dll and be happy - but if ANY other file is utilized - inclusive of a copy of the EXACT same dll - but at a different path - then the whole thing blows up. It can't handle using two different files.
How can I fix this? Ideally, I've love to make the initial magic loading of the dll draw upon a private assembly, but no matter what I do with manifests or .dll.local etc it will not respect that until this second dll loading takes place.
Note that with the non-bundled dll (external dependencies) it always uses the winsxs MSVCR90.DLL.
I can "fix" my failure to use the dll by forcing the system path to load the winsxs copy, but that is pretty useless for a deployable com server!
The reason is that you DLL has a manifest that tells the module loader to search also in the SxS storage.
You have several choices
Build your DLL using static linkage. Not using any of the MFC-DLLs (see project settings)
Don't use a side by side manifest for the DLL and still use the MFC DLLs. But beware you have to ship those DLL with your DLL in the local path (see DLL search sequence docs)
Use a later build of VS. Later versions of VS don't use the SxS storage any more and there are no manifests for those DLLs any more.
For the 2. see this article in code project. There is an update for VS-2008 [here].
2
Build your DLL

Why some programs require both .lib and .dll to work

When I was going to set up my developing environment for (SDLSimple DirectMediaLayer), I downloaded the package provided by the website, from the readme.txt file I found that I need both .lib and .dll...
My question is :
I am not sure if my understanding is correct : in my thought, .lib for windows is like .a for linux, is static library, and .dll for windows is like .so for linux, is shared library, is this correct ?
If the idea above is correct, I think we can only use .lib or .dll, since the work they do is the same ? So why bother to get both two in one program ?
And I do not quite understand .dll, if my memory servers me right, the one of the advantage for shared library is they can be updated while the program is running, so how can people do this, if we update the .dll file, how can an running program get to know the changes of the .dll and reload it to memory ?
In VC, there are two kinds of libs.
First type is a "real library", generated by the "static library project", which contains everything you can link and run, without dll.
The second type is "import library", generated by the "dll project", which just contains the symbols the link.exe needs. You can link to the library, but you need the dll to run.
There are two ways to use a dll. Link to the "import library" or "use LoadLibrary API".
You can not change the dll file when the dll is loaded.
But if you use LoadLibrary to load the dll, you can use FreeLibrary to unload the dll, change the dll and then use LoadLibrary to load the new dll. Of course, you need to monitor something to invoke this procedure.
Still the easier way is that, use a loader to do the update, then load the real exe.

Reference VB.NET DLL in Kofax Document Validation Script

We are working on a validation script for Kofax Capture 9.0 / 10.0 in VB.NET 3.5.
We know how to create a script using the Admin Module, and how to get it operational.
The problem is that we need to reference a dll, located on a remote machine. (GAC is no option) This dll holds abstract classes we need in each validation script.
Even when putting the dlls locally (copy local), the Validation Module (index.exe) immediately throws the "cannot find reference" exception, even though the project compiled perfectly.
I guess the basic question comes down to: where do we put the dlls, in order for the Validation Module to find them?
The simple answer is to put the dll in the same folder as the application because this is one of the places which .NET will probe when trying to find it. The Validation module is run from the Capture bin directory which will be something like "C:\Program Files (x86)\Kofax\CaptureSS\ServLib\Bin\". This would need to be done on each client using Validation.
If you have a more complicated scenario, you could look implementing the AppDomain.AssemblyResolve Event and using Assembly.LoadFile to get the assembly from a custom location, but the using the bin path is less complicated.
If you end up having further trouble, you can troubleshoot by using the Assembly Binding Log Viewer (Fuslogvw.exe) which can tell you more details about why the assembly failed to load and where .NET tried to search for it. Assembly loading can fail for reasons other than just the path.
For more detail on how .NET loads assemblies, see the following:
How the Runtime Locates Assemblies
Locating the Assembly through Codebases or Probing
We found a solution: add all library files as "links" to the project. (Add --> Existing File --> small arrow next to "Add" --> Add as Link)
This ensures the files are compiled when you build the project. The Kofax Validation Module can now find the files, whereas when referencing the file, it could not. Why it could not, remains a mystery...

Identify Assembly dll method invocations

I have a problematic win32 program that I only have the executable for. This program uses the ODBC32.dll. I am trying to figure out how this program sets up it's ODBC connection. More specifically if it is defining a timeout for command execution or not.
It seems reasonable to find this information by examining the assembly. Anyone know of any techniques for isolating calls to a dll within an assembly file?
I also have the pdb (debug symbols) for the exe if that helps.