MSYS2+MinGW64: why do ldd and objdump find different DLL dependencies? - dll

I get different DLLs dependencies reported by different tools and like to deeply understand why. So answers like "tool xxx is crap, use cool yyy" are opinions and not welcome. Answers like "you have wrong path order" or "you missed to export XXX=yyyy" are not what I focus for, but if they are applicable, those are welcome too. What I exactly want to understand is for instance, which PE/ELF tables which of the tools use and which they can't see. All under the aspect of mingw / msys2 environment.
For completeness: I compiled my program inside Codelite IDE following exactly this Setup Instructions for Codelite and inserted /mingw64/bin:/clang64/bin: in front of PATH. The expectation is that this has nothing to do with my observation. Just for the case I couldn't see the wood because of all the trees. Inserted my code into the helloworld example.
For completeness II: the ucrt variant is installed for different projects. I did not use it here.
For completeness III: I give all my instructions here. Expecting, 80% have nothing to do with the question, but just in case you see things I missed.
Given: MSYS2 (Sept.2022, msys2-x86_64-20220904.exe) with:
pacman -Suy \
mingw-w64-x86_64-gcc \
base-devel development \
sys-utils \
mingw-w64-ucrt-x86_64-toolchain mingw-w64-i686-toolchain mingw-w64-x86_64-toolchain \
libmodbus \
mingw-w64-x86_64-codelite
Opened a MSYS terminal:
$ which gcc && which g++ && which clang
/mingw64/bin/gcc
/mingw64/bin/g++
/mingw64/bin/clang
codelite &
Inside Codelite I only changed Settings | Build Settings | Compilers to "MinGW 32bit (MSYS2 64bit)" and therein the two cfg lines:
make C:/c/msys64/mingw32/bin/mingw32-make.exe -j8 SHELL=sh.exe
makedir mkdir -p
Program was then built via codelite generated makefile:
C:\WINDOWS\system32\cmd.exe /C C:/c/msys64/mingw32/bin/mingw32-make.exe -j8 SHELL=sh.exe -e -f Makefile
...[cut]...
C:/c/msys64/mingw32/bin/g++.exe -c protocol_writer.cpp" -g -O0 -Wall -o Debug/protocol_writer.cpp.o -I. -I.
C:/c/msys64/mingw32/bin/g++.exe -c scan_dtsu666_main.cpp" -g -O0 -Wall -o Debug/scan_dtsu666_main.cpp.o -I. -I.
C:/c/msys64/mingw32/bin/g++.exe -o Debug/modbus-dtu-dump #"modbus-dtu-dump.txt" -L. -lmodbus
====0 errors, 0 warnings====
Program was not starting because of missing DLLs (no pop-up or error msg). Now we come to the core issue. I invoked:
$ which ldd
/usr/bin/ldd
$ ldd modbus-dtu-dump.exe
ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffdf2c70000)
ntdll.dll => /c/Windows/SysWOW64/ntdll.dll (0x77310000)
wow64.dll => /c/WINDOWS/System32/wow64.dll (0x7ffdf2510000)
wow64win.dll => /c/WINDOWS/System32/wow64win.dll (0x7ffdf1be0000)
$ which objdump
/mingw64/bin/objdump
$ objdump --private-headers modbus-dtu-dump.exe | grep 'DLL'
DLL Name: libgcc_s_dw2-1.dll
DLL Name: KERNEL32.dll
DLL Name: libmodbus-5.dll
DLL Name: msvcrt.dll
DLL Name: libwinpthread-1.dll
DLL Name: libstdc++-6.dll
... and wonder, why ldd doesn't tell anything about the mingw runtimes, while objdump doesn't tell about the windows os core dependencies.
Edit:
As I wrote in a side note "Program was not starting because of missing DLLs (no pop-up or error msg).", there seems to be another miss-configuration. Once I changed the IDE settings to use the 64bit target, it was able to start and all-aspect debugging. I'll keep this up to date, for the case Google leads some people to here, searching for similar codelite or mingw problems. Goal is to have both architectures compile-able and debug-able.

Related

Cannot install regular files for Yocto using CMake

I have been using autotools for a few years, and I'm learning CMake now for new projects.
Here is what I have:
myfirstrecipe.bb:
inherit pkgconfig cmake
...
do_install() {
install -d ${D}${datadir}/folder1/folder2
install -m 0755 ${S}/files/file.txt ${D}${datadir}/folder1/folder2
}
mysecondrecipe.bb:
...
DEPENDS = "myfirstrecipe"
...
This works fine. The second recipe can find the file.txt installed by the first recipe, which I see it is installed in the secondrecipe sysroot:
build/tmp/work/armv7ahf-vfp-os-linux-musleabi/mysecondrecipe/510-r0/mysecondrecipe-sysroot/usr/share/folder1/folder2/file.txt
However I want CMake to install the file instead. So when I try this:
myfirstrecipe.bb:
inherit pkgconfig cmake
...
OECMAKE_TARGET_INSTALL = "file-install"
CMakeLists.txt:
add_custom_target(file-install)
add_custom_command(TARGET file-install POST_BUILD
COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_INSTALL_DATADIR}/folder1/folder2
COMMAND ${CMAKE_COMMAND} -E copy_if_different
${CMAKE_SOURCE_DIR}/files/file.txt
${CMAKE_INSTALL_DATADIR}/folder1/folder2/)
Then I get a build error from mysecondrecipe.bb saying it could not find the file since it is not installed. I see it installed here:
build/tmp/work/armv7ahf-vfp-os-linux-musleabi/myfirstrecipe/1.0-r0/myfirstrecipe-1.0/share/folder1/folder2/file.txt
But not in the path above. Anyone can see what I am missing? If I were to use Autotools I could easily get this working with this:
Automake.am:
file-install: $(shell find files/ -type f -name '*.txt')
mkdir -p ${DESTDIR}${datadir}/folder1/folder2
cp -u $^ -t ${DESTDIR}${datadir}/folder1/folder2
Basically you do not use the standard way of installing files.
CMake has an install directive install, wich is commonly used and powerfull.
Doing so, leads to the nice situation, that Within myfirstrecipe.bb an own do_install task is not necessary. The cmake.bbclass, you already inherit, is adding a do_install task and relies on the install directive within your CMakeLists.txt
You can take a look at the cmake.bbclass to see how it is implemented. It's at poky/meta/classes/cmake.bbclass
I guess that switching to install will make life easier

Use Fortify sourceanalyzer with CMake

I have a Makefile generated by CMake. The following path to CMake executable is set in the Makefile:
CMAKE_COMMAND = /home/xyz/opt/cmake/cmake-3.1.1/bin/cmake
How can I integrate Fortify sourceanalyzer with it and run scans?
I had the same challenge but solved it by running it like this:
sourceanalyzer -b project_ID -clean
Go to your build directory and perform make clean or remove all contents including the Makefile
Run cmake by changing CC and CXX variables:
CC="sourceanalyzer -b project_ID gcc" CXX="sourceanalyzer -b project_ID g++" cmake ..
Run make and fortify should be translating files while compilers do their job.
Run sourceanalyzer -b project_ID -scan -f results.fpr
Hope it helps.
I was tasked with integrating our CMake build system with HP Fortify SCA and came across this Thread that gave some insights but lacked specifics as related to HP Fortify so I thought I would share my implementation.
I created a fortify_tools directory at the same level as the source directory. Inside the fortify_tools are a toolchain file and fortify_cc, fortify_cxx, and fortify_ar scripts that will be set as the cmake_compilers via the toolchain file.
fortify_cc
#!/bin/bash
sourceanalyzer -b <PROJECT_ID> gcc $#
fortify_cxx
#!/bin/bash
sourceanalyzer -b <PROJECT_ID> g++ $#
fortify_ar
#!/bin/bash
sourceanalyzer -b <PROJECT_ID> ar $#
NOTE: insert your project name in place of PROJECT_ID
Setting cmake to use the scripts is accomplished in a toolchain file.
fortify_linux_toolchain.cmake
INCLUDE (CMakeForceCompiler)
SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)
#specify the compilers
SET(CMAKE_C_COMPILER ${CMAKE_SOURCE_DIR}/fortify_tools/fortify_cc)
SET(CMAKE_CXX_COMPILER ${CMAKE_SOURCE_DIR}/fortify_tools/fortify_cxx)
SET(CMAKE_AR_COMPILER ${CMAKE_SOURCE_DIR}/fortify_tools/fortify_ar)
To generate makefiles using the toolchain file
ccmake -DCMAKE_TOOLCHAIN_FILE=../fortify_tools/foritfy_linux_toolchain.cmake ../
configure and generate your makefiles and build your project.
Once the project is built from within the build directory generate a fortify report by
sourceanalyzer -Xmx2400M -debug -verbose -b <PROJECT_ID> -scan -f <PROJECT_ID>.fpr
I understand the last step is outside of CMake but I am pretty confident a cmake_custom_command can be created to perform the scan step as a post build action.
Finally, this is just the linux implementation but the concept scales well to Windows by creating the necessary batch files and windows specific toolchain file
Fortify doesn't support CMake, I received confirmation from Fortify support team.
This answer is late, but might help someone. This is actually easy to fix - you simply need to run cmake inside sourceanalyzer as well. Make a simple build script that calls cmake and then make, and use sourceanalyzer on that instead. I am using fortify 4.21.
Our old Fortify script for building hand-created Makefiles used a build command that looked like this:
$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID make -f Makefile -j12
I was able to get it working for a project that had been converted to CMake by replacing the above line with this, inspired by a couple of the other answers here:
CC="$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID gcc" \
CXX="$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID g++" \
AR="$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID ar" \
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug ..
make -f Makefile -j12 VERBOSE=1
This is with cmake 2.8.12.2 on Linux.
Below is the script i use for my example project to generate HP Fortify report for Android JNI C/C++ Code.
#!/bin/sh
# Configure NDK version and CMake version
NDK_VERSION=21.0.6113669
CMAKE_VERSION=3.10.2
CMAKE_VERSION_PATH=$CMAKE_VERSION.4988404
PROJECTID="JNI_EXAMPLE"
REPORT_NAME=$PROJECTID"_$(date +'%Y%m%d_%H:%M:%S')"
WORKING_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
BUILD_HOME=${WORKING_DIR}/../hpfortify_build
FPR="$BUILD_HOME/$REPORT_NAME.fpr"
# Following exports need to be configured according to host machine.
export ANDROID_SDK_HOME=/Library/Android/sdk
export ANDROID_CMAKE_HOME=$ANDROID_SDK_HOME/cmake/$CMAKE_VERSION_PATH/bin
export ANDROID_NDK_HOME=$ANDROID_SDK_HOME/ndk/$NDK_VERSION
# E.g. JniExample/app/hpfortify/build/CMakeFiles/3.10.2
export CMAKE_FILES_PATH=${BUILD_HOME}/CMakeFiles/$CMAKE_VERSION
export HPFORTIFY_HOME="/Applications/Fortify/Fortify_SCA_and_Apps_20.1.0/bin"
export PATH=$PATH:$ANDROID_SDK_HOME:$ANDROID_NDK_HOME:$ANDROID_CMAKE_HOME:$HPFORTIFY_HOME
echo "[========Start Android JNI C/C++ HP Fortify scanning========]"
echo "[========Build Dir: $BUILD_HOME========]"
echo "[========HP Fortify report path: $FPR========]"
function create_build_folder {
rm -rf $BUILD_HOME
mkdir $BUILD_HOME
}
# The standalone cmake build command can be found from below file.
# JniExample/app/.cxx/cmake/release/x86/build_command.txt
# This file is generated after running command
# `➜ JniExample git:(master) ✗ ./gradlew :app:externalNativeBuildRelease`
function configure_cmake_files {
cd $BUILD_HOME
$ANDROID_CMAKE_HOME/cmake -H$BUILD_HOME/. \
-DCMAKE_CXX_FLAGS=-std=c++11 -frtti -fexceptions \
-DCMAKE_FIND_ROOT_PATH=$BUILD_HOME/.cxx/cmake/release/prefab/x86/prefab \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_TOOLCHAIN_FILE=$ANDROID_SDK_HOME/ndk/$NDK_VERSION/build/cmake/android.toolchain.cmake \
-DANDROID_ABI=x86 \
-DANDROID_NDK=$ANDROID_SDK_HOME/ndk/$NDK_VERSION \
-DANDROID_PLATFORM=android-16 \
-DCMAKE_ANDROID_ARCH_ABI=x86 \
-DCMAKE_ANDROID_NDK=$ANDROID_SDK_HOME/ndk/$NDK_VERSION \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=$BUILD_HOME/intermediates/cmake/release/obj/x86 \
-DCMAKE_MAKE_PROGRAM=$ANDROID_SDK_HOME/cmake/$CMAKE_VERSION_PATH/bin/ninja \
-DCMAKE_SYSTEM_NAME=Android \
-DCMAKE_SYSTEM_VERSION=16 \
-B$BUILD_HOME/.cxx/cmake/release/x86 \
-GNinja ..
}
function build {
cmake --build .
}
function cleanup {
rm -rf $BUILD_HOME/CMakeFiles/native-lib.dir
rm -rf $FPR
$HPFORTIFY_HOME/sourceanalyzer -clean
}
function replace_compiler_paths {
FORTIFY_TOOLS_PATH="$WORKING_DIR"
CLANG_PATH="$ANDROID_SDK_HOME/ndk/$NDK_VERSION/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang"
CLANGXX_PATH="$ANDROID_SDK_HOME/ndk/$NDK_VERSION/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++"
HPFORTIFY_CCPATH="$FORTIFY_TOOLS_PATH/fortify_cc"
HPFORTIFY_CXXPATH="$FORTIFY_TOOLS_PATH/fortify_cxx"\"
sed -i '' 's+'$CLANG_PATH'+'$HPFORTIFY_CCPATH'+g' $CMAKE_FILES_PATH/CMakeCCompiler.cmake
sed -i '' 's+'$CLANG_PATH.*[^")"]'+'$HPFORTIFY_CXXPATH'+g' $CMAKE_FILES_PATH/CMakeCXXCompiler.cmake
}
function scan {
$HPFORTIFY_HOME/sourceanalyzer -b $PROJECTID -scan -f $FPR
# copy the file to $WORKING_DIR
cp $FPR $WORKING_DIR
}
create_build_folder
configure_cmake_files
echo "[========Compile C/C++ using normal compiler ========"]
build
echo "[========Replace the compiler with HP Fortify analyser wrapper compilers ========"]
replace_compiler_paths
echo "[========Clean up the build intermediates and the older build ID and fpr file ========"]
cleanup
echo "[========Recompile C/C++ using HP Fortify analyser wrapper compilers ========"]
build
echo "[========Scan the compiled files and generate final report ========"]
scan
echo "[========Change directory to original working dir ========"]
cd $WORKING_DIR
Need to configure below vars before using it. For my case, I use NDK 21 and CMake 3.10.2 and my project ID is "JNI_EXAMPLE"
# Configure NDK version and CMake version
NDK_VERSION=21.0.6113669
CMAKE_VERSION=3.10.2
CMAKE_VERSION_PATH=$CMAKE_VERSION.4988404
PROJECTID="JNI_EXAMPLE"
# Following exports need to be configured according to host machine.
export ANDROID_SDK_HOME=/Library/Android/sdk
export ANDROID_NDK_HOME=$ANDROID_SDK_HOME/ndk/$NDK_VERSION
export HPFORTIFY_HOME="/Applications/Fortify/Fortify_SCA_and_Apps_20.1.0/bin"
Here is a more detailed explanation: Using HP Fortify to Scan Android JNI C/C++ Code
On recent version of CMake one can use:
CMAKE_<LANG>_COMPILER_LAUNCHER='sourceanalyzer;-b;<PROJECT_ID>'
You can add other arguments (like -Xmx2G for instance), semicolon separated, as mentioned on cmake documentation
You need to check if you don't use the compiler launcher for another tool like ccache. We can probably use both with
CCACHE_PREFIX='.../sourceanalyzer -b ID'
Here is what I've used in CMake project:
project(myFortifiedProject LANGUAGES CXX)
set(CMAKE_CXX_COMPILER_LAUNCHER ${FORTIFY_TOOL} -b ${PROJECT_NAME})
So when running cmake (assuming sourceanalyzer is on the path):
cmake <other args> -DFORTIFY_TOOL=sourceanalyzer
So the normal build command works:
make myFortifiedProject
And you can finally collect results with:
sourceanalyzer -b myFortifiedProject -scan

Objective-C for Windows with Gitbash and GCC

When I compile things with gitbash and gcc, is there someway to shorten what I need to type?
In order to compile my helloworld program, I have to type the following in:
gcc -o helloworld.exe helloworld.m -I C:/GNUstep/GNUstep/System/Library/Headers -L C:/GNUstep/GNUstep/System/Library/Libraries -std=c99 -lobjc -lgnustep-base -fconstant-string-class=NSConstantString
Most people use Makefiles (or something related, e.g. CMake) in order to ease development. Makefiles are similar to bash scripts (the syntax is similar as well). After you created a Makefile you can run make and it does exactly what you specified.
If you want to know how to run Automake on Windows, read this: How to run a makefile in Windows?

MinGW linking dynamic library with bad name

I am using MinGW64 (Windows 7) without MSYS and I have the following problem:
I have one dll, written in C99, which has to have the .mexw64 suffix so it can be used by Matlab. I would like to be able to link this dll form another dll (mexw64) dynamically but gcc won't allow me to link directly. I cannot do static linking, because both dlls have many functions of the same name which can be hidden by not exporting their symbols when creating the shared library.
So far, I have tried:
To create a symbolic link (with correct suffix and preffix) using mklink. This works, but I was not able to run mklink from the makefile. Maybe it is due to the fact I am not using MSYS which could have ln -s (I havent checked).
To do the copy of the first dll and correcting the suffix and prefix. This worked better than I expected, because on runtime the second dll actually uses the original .mexw64 and not the dll copy. I guess it is just because the .mexw64 is found first, but why is that .mexw64 searched in the first place? How the system know it is actually a dll?
My question is, is this correct/safe enough? Are there any other options?
Thanks for comments.
You should build a proper implib, either as a linker output or from a .def.
Linker:
$ gcc -shared -o testimpl.mexw64 testimpl.c -Wl,--out-implib,libtestimpl.a
$ dlltool -I libtestimpl.a
testimpl.mexw64
Or create a .def file, specifying an explicit LIBRARY:
$ cat testimpl.def
LIBRARY testimpl.mexw64
EXPORTS
test #1
$ dlltool -d testimpl.def -l libtestimpl.a
$ dlltool -I libtestimpl.a
testimpl.mexw64
And finally, link stuff:
$ gcc -o test.exe test.c libtestimpl.a
# or
$ gcc -o test.exe test.c -L. -ltestimpl
$ grep testimpl.mexw64 test.exe
Binary file test.exe matches

Unbound modules in OCaml

My problem is that ocamlc and ocamlopt apear to be refusing to find third party libraries installed through apt-get. I first started having this problem when I tried to incorporate third-party modules into my own OCaml programs, and quickly wrote it off as a personal failing in understanding OCaml compilation. Soon-- however-- I found myself running into the same problem when trying to compile other peoples projects under their own instructions.
Here is the most straight-forward example. The others all use ocamlbuild, which obfuscates things a little bit.
The program: http://groups.google.com/group/fa.caml/msg/5aee553df34548e2
The compilation:
$ocamlc -g -dtypes -pp camlp4oof -I +camlp4 dynlink.cma camlp4lib.cma -cc g++ llvm.cma llvm_bitwriter.cma minml.ml -o minml
File "minml.ml", line 43, characters 0-9:
Error:Unbound module Llvm
Even when I provide ocamlc with the obsolute paths to the llvm files, like so...
$ ocamlc -g -dtypes -pp camlp4oof -I +camlp4 dynlink.cma camlp4lib.cma -cc g++ /usr/lib/ocaml/llvm-2.7/llvm.cma /usr/lib/ocaml/llvm-2.7/llvm_bitwriter.cma minml.ml -o minml
... to no avail.
What am I doing wrong?
Your command is doing two things: it's compiling minml.ml (into minml.cmo), then linking the resulting object into minml.
Compiling a module requires the interfaces of the dependencies. The interfaces contain typing information that is necessary to both the type checker and the code generator; this information is not repeated in the implementation (.cma here). So for the compilation stage, llvm.cmi must be available. The compiler looks for it in the include path, so you need an additional -I +llvm-2.7 (which is short for -I /usr/lib/ocaml/llvm-2.7).
The linking stage requires llvm.cma, which contains the bytecode implementation of the module. Here, you can either use -I or give a full path to let ocamlc know where to find the file.
ocamlc -g -dtypes -I +camlp4 -I +llvm-2.7 -pp camlp4oof -c minml.ml
ocamlc -g -cc g++ -I +camlp4 -I +llvm-2.7 dynlink.cma camlp4lib.cma llvm.cma llvm_bitwriter.cma minml.cmo -o minml
or if you want to do both stages in a single command:
ocamlc -g -dtypes -cc g++ -I +camlp4 -I +llvm-2.7 dynlink.cma camlp4lib.cma llvm.cma llvm_bitwriter.cma -pp camlp4oof minml.ml -o minml