fixing libvlc release mode crash with VC2010 - crash

Question

I am using libVLC in one of my apps which I am compiling with VC2010 (also tried VC2008), the debug mode of my app works great but as soon as I compile to release mode and try to call into libVLC I get a crash. I asked for help on the vlc forums and someone mentioned this usually points to calling convention differences, however I am not sure what to check to see if this is the case or more importantly how to fix it.
some notes:
I am compiling libVLC using Ubuntu and following the how to guides on the libVLC wiki.
I'm using libVLC inside a C++ file.
I've tried compiling libVLC with and without debugging information.
I've tried calling libvlc_get_version and libvlc_new as my first call, both crash.
Even though I do not have symbols in my release version, I can see the call stack and it is definitely getting messed up as it is showing functions in the stack that are never-ever called which seems to indicate the wrong calling convention but again I'm not sure how to check/fix this.
I'm not sure if it is related but another issue I am having with libvlc is that I am trying to delay load the dll (have tried not doing this for the above problem but it didnt make a difference), i'm adding the linker flags: /DELAYLOAD:libvlc.dll /DELAYLOAD:libvlccore.dll
, but when the linking occurs I get these warnings:
LINK : warning LNK4199: /DELAYLOAD:libvlc.dll ignored; no imports found from libvlc.dll
LINK : warning LNK4199: /DELAYLOAD:libvlccore.dll ignored; no imports found from libvlccore.dll
However it is definitely linking to the lib and requiring the dll as seen with Dependency Walker (not to mention I am calling into it).. again not sure if this is related but wanted to throw it out there as well.
I appreciate any advice/help on this one. Thanks!

Answer 1

I just came to the same problem and after some digging up with IDA dissasembler I've found out that linker throws out all libvlc imports. And yes INCREMENTAL flags adds them back in but as you said it's not the explanation of the problem.
Now I had a similar occurrence when designing a driver where Release eliminated function pointers and strings. And the solution was to set Linker\Optimization\References to No (/OPT:NOREF). So then linker leaves in all references even if it thinks they are not used.
And of course that fixes the problem.
So another mystery solved. ,)
Best regards
Waldemar

Answer 2

Actually, adding '/OPT:NOREF' solves the problem as well, at least in my case. and i think the problem may result from an 'issue' with dlltool, as ffmpeg suffers the same problem (http://ffmpeg.org/platform.html#Linking-to-FFmpeg-with-Microsoft-Visual-C_002b_002b), and like ffmpeg, libvlc(i guess) may generate windows 'lib' files with 'dlltool' instead of 'lib.exe' from msvc. the related bug report with dlltool is here: https://sourceware.org/bugzilla/show_bug.cgi?id=12633#c1
as you claimed that you were "compiling libVLC using Ubuntu", i think you probably encountered the same problem. hope it'll help.
btw, the official distribution of ffmpeg provide '.def' files, so i could regenerate the 'correct' lib files with 'lib.exe' from msvc and problem solves. however, as the official windows distribution of vlc does not provide '.def' files, and i failed to reconstruct the lib files via the 'dumpbin and lib' approach(failed when dumpbin, there must be something strange with the dll), i cannot do further verification.