Registering a DirectshowFilter links InprocServer32 registry entry to regsvr32.exe instead of host DLL, where did I go wrong? - com

I have a DirectShow filter (MonogramAAC encoder) successfully compiling. It also registers successfully, however, when I try to instantiate the filter in GrapheditPlus, it shows a CO_E_ERRORINDLL. After inspecting the filter properties under GraphEditPlus, the filter was actually linked to RegSvr32.exe instead of the actual host DLL!
Here's the evidence:
I'm sure the .def file in the linker is correct, I also tested that it IS being considered by the compiler and it is correctly pointing to the filter DLL.
A little bakground info:
I couldn't get the original project to compile under Visual Studio 2017 after automated conversion. (I couldn't resolve strange MFC and runtime library linker errors between the Monogram filter and the libaac.lib library, that I'm also able to compile successfully).
Finally I decided to recreate a brand-new VS2017 Project mimicking the original one and got it to compile and register (with the only warning being the mismatch between the Project name and the output library), but obviously I missed something important while recreating the project.
Any pointers?

Ok, I managed to find the reason for this unexpected behavior in this old post.
RegSvr32 just calls the DllRegisterServer entry point in your code.
What gets written to the registry is entirely up to your filter. If
you are using the standard baseclass call to AMovieDllRegisterServer2
as your DllRegisterServer, I would suspect that something went wrong
in setting up g_hInst, which should have been done by the call
DllEntryPoint in your DllMain.
…
The DllMain is not being called, so g_hInst stays 0 and
GetModuleFileNameA(...) returns the name of the currently executing
program instead of my filter.
The filter is using MFC (dynamically linked), and its DllMain() version is not calling the DllEntryPoint() function. I had to override the MFC DllMain() by adding extern "C" { int _afxForceUSRDLL; } as described in an answer to this question and call DllEntryPoint() myself to fix the registration problem. (I also found a missing #define was the reason for the property page not showing up).

Related

Invalid floating point operation while loading dll

I got a problem recently that when I added a code to a lib, which is used by a dll, the dll cannot be loaded. I get an exception "Floating point invalid operation at xxx". When I comment out the code it works. The code added is just straight forward like creating a TADOQuery object, add the sql statment and the parameters, then do an open, check value etc. Nothing suspicious. Now when I uncomment the code and comment another chuck of code in the unit elsewhere, it starts to work. So obviously the newly added code was just tripping the issue caused elsewhere.
Now this error happens when LoadLibrary() call is invoked. There is no attach code. The DllMain, just has a return 1. When this happens the assembly code is always at FSTP tbyte ptr doing a System::Variant::Clear().
Anyone has come across such issues? Any tips on how to debug is also appreciated.
PS: It could also be Embarcadero's compiler issue. I recently figured out that the assembly generated for a combination of assignment and comparison is incorrect. The only suspicion here is the lib that the dll uses is large. When I do a clean and build of the lib, the total lines shows 5.7m. Maybe there is some setting for large libs?
IDE-XE4, c++ builder
OS-Win 7
Thanks,
Mathew Joy
C++Builder and Delphi have different default FPU settings than, say, MSVC and the DLLs written in it.
It probably goes like this: After DllMain, certain exceptions are masked to be hidden. But before the first floating point operation in C++Builder code occurs, that exception is unmasked and the exception occurs.
This is a known problem with C++Builder/Delphi floating point and DLLs.

Why does a mingw-compiled Game Maker extension crash on exit when compiled without -static?

I compile a DLL with mingw 4.5.0 and use it as a Game Maker 8.0 extension. Game Maker dynamically loads the dll. Everything appears to work (the dll functions are called and provide correct return values), but when I close Game Maker a dialog pops up: "Microsoft Visual C++ Runtime Library", "This application has requested the Runtime to terminate in an unusual way." After that, the process continues to linger in the background for a few seconds and then disappears.
This also happens when none of the functions of the dll are actually called. There is no DllMain, and all static/global variables are basic data types or std::string (it actually also happens when I remove the std::strings).
The dll statically links with zlib and libpng. The problem seems to vanish when I pass -static to the linker which (I assume) also links the runtime statically. However, this considerably inflates the size of my DLL, and it is at best a workaround until I understand what is going on.
Any ideas on what might be the cause?
Update: Actually, it seems that the problem only happens when two extensons are loaded in Game Maker: One with a dll linked with -static, and the other without. Linking both without -static makes the problem disappear. However, I still don't understand the problem, because the dlls never directly interact or share data structures.
Update 2: I recently found out that this might be related to strange behavior of Game Maker itself. It seems that global variables aren't initialized correctly when the DLL is loaded, which might cause a crash on unloading if global objects try to free memory they don't own. That would mean the -static was just a random factor that changed the value of the uninitialized memory the globals were associated with.
Update 3: Modified the above to include the info that this is about Game Maker extensions, since this is likely relevant as per Update 2.
It's just a guess. Try to look into something like the "static initialization order fiasco" that is described here. It may be that your problem is more related to the destructors (since it happens on close).

Having trouble debugging class library plugin

I have a windows form application in which I'm attempting to utilize a plugin (class library). In the code I have it load the assembly from a dll file, which means I have not been able to debug. Furthermore I have not found out how to compile the library so I've had to use the debuged dll version for testing. I've run into a bug in which I create a new object and send that data through an interface to the plugin in an attempt to retrieve a blank slate group box from the plugin. However instead of reading the parameter as a new object i managed to step through the code once (don't ask me how, I don't know and I haven't been able to repeat it) and it appeared that the code was registering the parameter as "nothing" thus why I received a null reference error in the main program.
Is there a better way to debug this mechanism and find out where the problem is? Any ideas on what the problem could be. As I read over this is seems somewhat vague and I'm not sure how to describe it, but I'm willing to host a connect now meeting if anyone is willing to look at what is going on and I'm not making myself understood very well.
I'm not sure if I follow exactly what you're doing but I usually find that the best way to debug a class library is by in the solution for the class library I add a new project, either a WinForms one or a Console one, I then set this new project as the start up project and add a reference to the Class Library project (via the Project tab in the Add Reference dialog).
You'll then be able to call the methods in the class library from the other project and you can put breakpoints anywhere to see what's really going on easily.
Ok, so the problem was that any time you edit the class library you have to compile (and the only way I know how is debugging, I can't find a compile button and the publish button doesn't work and building doesn't appear to make a dll). But anyways you have to compile, transfer the file so you are reading the most recent one. If you edit the code during runtime it does NOT update the dll in use...which was my problem.

Access violation when running native C++ application that uses a /clr built DLL

I'm reorganzing a legacy mixed (managed and unmanaged DLLs) application so that the main application segment is unmanaged MFC and that will call a C++ DLL compiled with /clr flag that will bridge the communication between the managed (C# DLLs) and unmanaged code. Unfortuantely, my changed have resulted in an Access violation that occurs before the application InitInstance() is called. This makes it very difficult to debug. The only information I get is the following stack trace.
> 64006108()
ntdll.dll!_ZwCreateMutant#16() + 0xc bytes
kernel32.dll!_CreateMutexW#12() + 0x7a bytes
So, here are some sceanrios I've tried.
- Turned on Exceptions->Win32 Exceptions->c0000005 Access Violation to break when Thrown. Still the most detail I get is from the above stack trace. I've tried the application with F10, but it fails before any breakpoints are hit and fails with the above stack trace.
- I've stubbed out the bridge DLL so that it only has one method that returns a bool and that method is coded to just return false (no C# code called).
bool DllPassthrough::IsFailed() { return false; }
If the stubbed out DLL is compiled with the /clr flag, the application fails. If it is compiled without the /clr flag, the application runs.
- I've created a stub MFC application using the Visual Studio wizard for multidocument applications and call DllPassthrough::IsFailed(). This succeeds even with the /clr flag used to compile the DLL.
- I've tried doing a manual LoadLibrary on winmm.lib as outlined in the following note Access violation when using c++/cli. The application still fails.
So, my questions are how to solve the problem? Any hints, strategies, or previous incidents. And, failing that, how can I get more information on what code segment or library is causing the access exception? If I try more involved workarounds like doing LoadLibrary calls, I'd like to narrow it to the failing libraries.
Thanks. BTW, we are using Visual Studio 2008 and the project is being built against the .NET 2.0 framework for the managed sections.
I believe I solve my problem. By systematically removing each library reference and
commenting out the calls to that particular library in the application code (unmanaged), I eventually removed the problem library and got the program to run. It's a brute force way of diagnosing the problem, and fortunately I didn't have to remove too many libraries to solve it. I'd still be curious if anyone has a comment if the library could have been identified though the debugger.
So, the next step is to move these library calls to managed code and pass the information back to the unmanaged side via my bridge DLL. BTW, I reintegrated the winmm.lib into the project and it still works.

Bizarre VB6 Make Problem - Previously working identical code won't recompile

I've got a really strange error and any light that anyone can shed on this would be greatly appreciated.
I made some changes to some VB6 source which builds a COM object. The automated build which builds our app returned an error. No problem I thought--I'll just back out my changes. Well backing out my changes isn't making the problem go away.
Specifically when I attempt to build the app via a .vbg file, with a command line like path\to\vb6\vb6 ProjectFile.vbg /make
I get a message
"Compile Error in File '', Line : Object library
invalid or contains references to object definitions that could not be
found."
As I said, I reverted the source code so I'm really stumped as to why this error is still occurring. Any VB6 gurus around who might be able to point me at an answer?
I can post the exact code in question but the fact that it was building correctly, stopped building correctly and now refuses to build correctly makes me think this is not a problem with my code but rather some problem in the environment. Like something got put in the registry as a result of the previous build error.
Any tips, hints, or suggestions greatly welcome. I realize my question is a bit sketchy but I'm not even sure what's important to include and what isn't.
EDIT 1:
Thanks for the excellent suggestions guys. I think it is something to do with VB6 doing some sort of auto-registration.
Just to add a bit more detail: this problem does not occur when I build the referenced vbp file from the IDE. It only happens on the make on the .vbg which contains the vbp. Also the build tool in question automatically pulls latest source and the error happens on both my local box and the dedicated build box.
EDIT 2:
Hi again all,
The release engineering fellow figured out how to get this to build in his build environment so it's currently ok. Once we're past this crunch, I'll try to interrogate him about what he did and share the details with everyone.
Thanks again for all the great suggestions. This is what's so great about SO; that is, I asked about a 10-year-old technology and I got several great and on-point ideas.
Make sure that the VBG and all the VBP's got rolled back as well. That error is consistent with a project trying to reference a CLSID that is no longer valid for the dependency. Have you tried loading up the project group and building from the IDE, if that works and you save and check in all the changes to the group and project files, you might be fixed up.
I'm guessing the fact that you mention that it was a COM component might be the source of the problem. If any of the public method's or properties have changed then I seem to remember that VB6 will change the interface GUIDs and auto register the new ones.
My suggestion would be to check the registry to look for any mention of the component name, make a note of any associated CLSIDs, back up the registry, and then delete the references.
As cmsjr mentions it could also be a bad CLSID reference in your .vbp files.
The other option is that the failure has caused a problem with some .tlb (type library) or olb (object library) files.
The best thing to do is move all your compatibility DLL to a separate and combined directory. The reason for this is control over what VB6 is using to check for binary compatibility. In addition the Typelibs that are generated IMPORT the references. So if you using Binary DLL Ver 10 for compatibility however the import is pulling in Binary DLL Ver 9 you will have issues. By keeping all the libraries in a single folder and pointing your projects to the DLLs in that folder you ensure that the respective TypeLib Import the correct version.
Finally if you have multiple levels of DLL reference each other. You may run into mysterious error where the VB6 is unable to compile using binary compatibility. In such cases you need to compile the lowest DLL in the hierarchy (Utility DLL perhaps) copy it over into the compatibility folders. Work your way up the chain until everything compiles in one shot again.
This is because if have DLL A reference DLL B which Reference DLL C. VB6 will get sometimes get confused if you make a change to A and C. will compile fine but A will not until the compatibility libraries are updated.
Hunt down and delete any .obj and .exp files that may be lying around from the previous failed build.
You will have to open the project & re-type in the lines that you changed.
Save the project alongwith VBG and re-compile after that.
I think that will fix it.
EDIT: The idea is that the cls/bas file remember the class (CLSID) that you used. So, if you change the references but don't change the lines in the cls/bas - it is a mismatch of type (what was referenced vs what is typed in cls/bas file).