Using CMake's fixup_bundle for bundling linux-only application - cmake

We developing some CV application, based of OpenCV, Boost, LibVLC and Caffe. Some of our customers want to deploy it on outdated(or unpopular) Linux distributions, so we must bundle all it's dependencies(and some vlc plugins), most of them can be found in any actual distro, but we have custom build of libcaffe vendored in our repo. So, now i solve it with this bash script:
#!/bin/bash
set -uex
export LD_LIBRARY_PATH=./contrib/caffe.arch32/lib/
function copy_deps {
libs=$(LD_TRACE_LOADED_OBJECTS=1 /lib/ld-linux.so.2 "$1" | cut -d" " -f 3 | sort |uniq | grep -v '^$')
for lib in $libs
do
cp -un "$lib" ./bundle
done
}
mkdir -p bundle
mkdir -p ./bundle/vlc/plugins
cp -r /usr/lib/vlc/plugins ./bundle/vlc
rm -rf ./bundle/vlc/plugins/lua
rm -rf ./bundle/vlc/plugins/gui
rm -rf ./bundle/vlc/plugins/visualization
for plugin in $(find ./bundle/vlc/plugins -name "*.so")
do
copy_deps "$plugin"
done
copy_deps ./detector
cp /lib/ld-linux.so.2 ./bundle
cp ./detector ./bundle
cp ./config.ini ./bundle
mkdir -p ./bundle/config
cp -r ./config/nn ./bundle/config
cp -r ./config/neuron ./bundle/config
echo "LD_LIBRARY_PATH=./ ./ld-linux.so.2 ./detector 2> /dev/null" > ./bundle/run.sh
chmod +x ./bundle/run.sh
zip -q -r bundle.zip bundle
It works fine, but only for executable build(we need shared lib too), only for x86_32 distros. We build our project with cmake, so after reading it's docs i noticed, that fixup_bundle is cmake-way for bundling. All examples and blogs about fixup_bundle is very simple, or related with OSX or Windows. So, i append my CMakeLists.txt
set(CMAKE_INSTALL_PREFIX ${CMAKE_BINARY_DIR})
set(BUNDLE_NAME ${PROJECT_NAME})
set(BUNDLE_PATH "${CMAKE_INSTALL_PREFIX}/${BUNDLE_NAME}")
set(APPS ${BUNDLE_PATH})
list(APPEND DIRS ${CMAKE_INSTALL_PREFIX}/${LIBDIR} ${CAFFE_LINK_PATH} /lib/ /usr/lib)
list(APPEND LIBS)
INSTALL(CODE "
include(BundleUtilities)
fixup_bundle(\"${APPS}\" \"${LIBS}\" \"${DIRS}\")
" COMPONENT Runtime)
And then try to run make install, I noticed that only our custom libcaffe is bundled, no boost, no opencv, no VLC. Why? How to bundle all dependencies?
OS: Arch Linux.

Related

Building a .deb package of Mono itself

I'm currently compiling the latest version of Mono from Github on an original version Raspberry Pi, on latest Raspbian.
This is a very time consuming process, which when it's complete I would not like to have to repeat.
Can the compiled Mono installation be packaged into a .deb to, for example, allow me to re-install latest Raspbian, then dpkg -i my-mono-build.deb?
Sure, and it's very easy to do if you choose the proper tool so that you don't need a master on debian packaging. As for me, I chose fpm to do exactly this. (Note: install via gem, not apt-get.)
And here you have an example of a script of how to build a Mono .deb with this, which I copy+paste here for posterity (just in case I delete the github repo by mistake, or github stops being a thing in the future):
#!/bin/bash
set -e
die () {
echo >&2 "$#"
exit 1
}
[ "$#" -eq 1 ] || die "Please specify the version of Mono you want to build as the argument. (Check the versions in the tarball list here: http://download.mono-project.com/sources/mono/)"
which fpm > /dev/null || (echo "Please install fpm (from gem, not apt-get)" && exit 1)
if mono --version > /dev/null 2>&1; then
echo "Mono is installed locally; please uninstall first" && exit 1
fi
WORK_DIR=/tmp/7digital-mono-work
rm -rf $WORK_DIR
mkdir $WORK_DIR
cd $WORK_DIR
MONO_VERSION=$1
MONO_DIR="mono-$MONO_VERSION"
SEVEND_VERSION="701"
MONO7D_VERSION=$MONO_VERSION'.'$SEVEND_VERSION
MONO7D_NAME="mono-7d"
echo "Downloading $MONO_VERSION"
wget http://download.mono-project.com/sources/mono/mono-$MONO_VERSION.tar.bz2
tar -jxf mono-$MONO_VERSION.tar.bz2
TARGET_DIR="$WORK_DIR/destdir"
mkdir $TARGET_DIR
cd "$WORK_DIR/$MONO_DIR"
./configure --prefix=/usr
make
make install DESTDIR="$TARGET_DIR"
cd $WORK_DIR
fpm -s dir \
-t deb \
-n $MONO7D_NAME \
-v $MONO7D_VERSION \
-C $TARGET_DIR \
-d "libglib2.0-dev (>= 0)" \
usr/bin usr/lib usr/share usr/include usr/etc
echo "Done. Your package should be ready in $WORK_DIR"

Use Fortify sourceanalyzer with CMake

I have a Makefile generated by CMake. The following path to CMake executable is set in the Makefile:
CMAKE_COMMAND = /home/xyz/opt/cmake/cmake-3.1.1/bin/cmake
How can I integrate Fortify sourceanalyzer with it and run scans?
I had the same challenge but solved it by running it like this:
sourceanalyzer -b project_ID -clean
Go to your build directory and perform make clean or remove all contents including the Makefile
Run cmake by changing CC and CXX variables:
CC="sourceanalyzer -b project_ID gcc" CXX="sourceanalyzer -b project_ID g++" cmake ..
Run make and fortify should be translating files while compilers do their job.
Run sourceanalyzer -b project_ID -scan -f results.fpr
Hope it helps.
I was tasked with integrating our CMake build system with HP Fortify SCA and came across this Thread that gave some insights but lacked specifics as related to HP Fortify so I thought I would share my implementation.
I created a fortify_tools directory at the same level as the source directory. Inside the fortify_tools are a toolchain file and fortify_cc, fortify_cxx, and fortify_ar scripts that will be set as the cmake_compilers via the toolchain file.
fortify_cc
#!/bin/bash
sourceanalyzer -b <PROJECT_ID> gcc $#
fortify_cxx
#!/bin/bash
sourceanalyzer -b <PROJECT_ID> g++ $#
fortify_ar
#!/bin/bash
sourceanalyzer -b <PROJECT_ID> ar $#
NOTE: insert your project name in place of PROJECT_ID
Setting cmake to use the scripts is accomplished in a toolchain file.
fortify_linux_toolchain.cmake
INCLUDE (CMakeForceCompiler)
SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)
#specify the compilers
SET(CMAKE_C_COMPILER ${CMAKE_SOURCE_DIR}/fortify_tools/fortify_cc)
SET(CMAKE_CXX_COMPILER ${CMAKE_SOURCE_DIR}/fortify_tools/fortify_cxx)
SET(CMAKE_AR_COMPILER ${CMAKE_SOURCE_DIR}/fortify_tools/fortify_ar)
To generate makefiles using the toolchain file
ccmake -DCMAKE_TOOLCHAIN_FILE=../fortify_tools/foritfy_linux_toolchain.cmake ../
configure and generate your makefiles and build your project.
Once the project is built from within the build directory generate a fortify report by
sourceanalyzer -Xmx2400M -debug -verbose -b <PROJECT_ID> -scan -f <PROJECT_ID>.fpr
I understand the last step is outside of CMake but I am pretty confident a cmake_custom_command can be created to perform the scan step as a post build action.
Finally, this is just the linux implementation but the concept scales well to Windows by creating the necessary batch files and windows specific toolchain file
Fortify doesn't support CMake, I received confirmation from Fortify support team.
This answer is late, but might help someone. This is actually easy to fix - you simply need to run cmake inside sourceanalyzer as well. Make a simple build script that calls cmake and then make, and use sourceanalyzer on that instead. I am using fortify 4.21.
Our old Fortify script for building hand-created Makefiles used a build command that looked like this:
$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID make -f Makefile -j12
I was able to get it working for a project that had been converted to CMake by replacing the above line with this, inspired by a couple of the other answers here:
CC="$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID gcc" \
CXX="$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID g++" \
AR="$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID ar" \
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug ..
make -f Makefile -j12 VERBOSE=1
This is with cmake 2.8.12.2 on Linux.
Below is the script i use for my example project to generate HP Fortify report for Android JNI C/C++ Code.
#!/bin/sh
# Configure NDK version and CMake version
NDK_VERSION=21.0.6113669
CMAKE_VERSION=3.10.2
CMAKE_VERSION_PATH=$CMAKE_VERSION.4988404
PROJECTID="JNI_EXAMPLE"
REPORT_NAME=$PROJECTID"_$(date +'%Y%m%d_%H:%M:%S')"
WORKING_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
BUILD_HOME=${WORKING_DIR}/../hpfortify_build
FPR="$BUILD_HOME/$REPORT_NAME.fpr"
# Following exports need to be configured according to host machine.
export ANDROID_SDK_HOME=/Library/Android/sdk
export ANDROID_CMAKE_HOME=$ANDROID_SDK_HOME/cmake/$CMAKE_VERSION_PATH/bin
export ANDROID_NDK_HOME=$ANDROID_SDK_HOME/ndk/$NDK_VERSION
# E.g. JniExample/app/hpfortify/build/CMakeFiles/3.10.2
export CMAKE_FILES_PATH=${BUILD_HOME}/CMakeFiles/$CMAKE_VERSION
export HPFORTIFY_HOME="/Applications/Fortify/Fortify_SCA_and_Apps_20.1.0/bin"
export PATH=$PATH:$ANDROID_SDK_HOME:$ANDROID_NDK_HOME:$ANDROID_CMAKE_HOME:$HPFORTIFY_HOME
echo "[========Start Android JNI C/C++ HP Fortify scanning========]"
echo "[========Build Dir: $BUILD_HOME========]"
echo "[========HP Fortify report path: $FPR========]"
function create_build_folder {
rm -rf $BUILD_HOME
mkdir $BUILD_HOME
}
# The standalone cmake build command can be found from below file.
# JniExample/app/.cxx/cmake/release/x86/build_command.txt
# This file is generated after running command
# `➜ JniExample git:(master) ✗ ./gradlew :app:externalNativeBuildRelease`
function configure_cmake_files {
cd $BUILD_HOME
$ANDROID_CMAKE_HOME/cmake -H$BUILD_HOME/. \
-DCMAKE_CXX_FLAGS=-std=c++11 -frtti -fexceptions \
-DCMAKE_FIND_ROOT_PATH=$BUILD_HOME/.cxx/cmake/release/prefab/x86/prefab \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_TOOLCHAIN_FILE=$ANDROID_SDK_HOME/ndk/$NDK_VERSION/build/cmake/android.toolchain.cmake \
-DANDROID_ABI=x86 \
-DANDROID_NDK=$ANDROID_SDK_HOME/ndk/$NDK_VERSION \
-DANDROID_PLATFORM=android-16 \
-DCMAKE_ANDROID_ARCH_ABI=x86 \
-DCMAKE_ANDROID_NDK=$ANDROID_SDK_HOME/ndk/$NDK_VERSION \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=$BUILD_HOME/intermediates/cmake/release/obj/x86 \
-DCMAKE_MAKE_PROGRAM=$ANDROID_SDK_HOME/cmake/$CMAKE_VERSION_PATH/bin/ninja \
-DCMAKE_SYSTEM_NAME=Android \
-DCMAKE_SYSTEM_VERSION=16 \
-B$BUILD_HOME/.cxx/cmake/release/x86 \
-GNinja ..
}
function build {
cmake --build .
}
function cleanup {
rm -rf $BUILD_HOME/CMakeFiles/native-lib.dir
rm -rf $FPR
$HPFORTIFY_HOME/sourceanalyzer -clean
}
function replace_compiler_paths {
FORTIFY_TOOLS_PATH="$WORKING_DIR"
CLANG_PATH="$ANDROID_SDK_HOME/ndk/$NDK_VERSION/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang"
CLANGXX_PATH="$ANDROID_SDK_HOME/ndk/$NDK_VERSION/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++"
HPFORTIFY_CCPATH="$FORTIFY_TOOLS_PATH/fortify_cc"
HPFORTIFY_CXXPATH="$FORTIFY_TOOLS_PATH/fortify_cxx"\"
sed -i '' 's+'$CLANG_PATH'+'$HPFORTIFY_CCPATH'+g' $CMAKE_FILES_PATH/CMakeCCompiler.cmake
sed -i '' 's+'$CLANG_PATH.*[^")"]'+'$HPFORTIFY_CXXPATH'+g' $CMAKE_FILES_PATH/CMakeCXXCompiler.cmake
}
function scan {
$HPFORTIFY_HOME/sourceanalyzer -b $PROJECTID -scan -f $FPR
# copy the file to $WORKING_DIR
cp $FPR $WORKING_DIR
}
create_build_folder
configure_cmake_files
echo "[========Compile C/C++ using normal compiler ========"]
build
echo "[========Replace the compiler with HP Fortify analyser wrapper compilers ========"]
replace_compiler_paths
echo "[========Clean up the build intermediates and the older build ID and fpr file ========"]
cleanup
echo "[========Recompile C/C++ using HP Fortify analyser wrapper compilers ========"]
build
echo "[========Scan the compiled files and generate final report ========"]
scan
echo "[========Change directory to original working dir ========"]
cd $WORKING_DIR
Need to configure below vars before using it. For my case, I use NDK 21 and CMake 3.10.2 and my project ID is "JNI_EXAMPLE"
# Configure NDK version and CMake version
NDK_VERSION=21.0.6113669
CMAKE_VERSION=3.10.2
CMAKE_VERSION_PATH=$CMAKE_VERSION.4988404
PROJECTID="JNI_EXAMPLE"
# Following exports need to be configured according to host machine.
export ANDROID_SDK_HOME=/Library/Android/sdk
export ANDROID_NDK_HOME=$ANDROID_SDK_HOME/ndk/$NDK_VERSION
export HPFORTIFY_HOME="/Applications/Fortify/Fortify_SCA_and_Apps_20.1.0/bin"
Here is a more detailed explanation: Using HP Fortify to Scan Android JNI C/C++ Code
On recent version of CMake one can use:
CMAKE_<LANG>_COMPILER_LAUNCHER='sourceanalyzer;-b;<PROJECT_ID>'
You can add other arguments (like -Xmx2G for instance), semicolon separated, as mentioned on cmake documentation
You need to check if you don't use the compiler launcher for another tool like ccache. We can probably use both with
CCACHE_PREFIX='.../sourceanalyzer -b ID'
Here is what I've used in CMake project:
project(myFortifiedProject LANGUAGES CXX)
set(CMAKE_CXX_COMPILER_LAUNCHER ${FORTIFY_TOOL} -b ${PROJECT_NAME})
So when running cmake (assuming sourceanalyzer is on the path):
cmake <other args> -DFORTIFY_TOOL=sourceanalyzer
So the normal build command works:
make myFortifiedProject
And you can finally collect results with:
sourceanalyzer -b myFortifiedProject -scan

Can I remove directory after $git clone and $make install

I wrote myself a litte script to install opencv under ubuntu14.04. Can I remove the directory 3party after the make install sorted the lib into system directories or are there dependencies? (Remove not only the MYBUILD but the complete 3party)
echo "\nInstall OpenCV?...<any key>\n"
read inp1; # $inp1
mkdir 3party;
cd 3party;
git clone https://github.com/Itseez/opencv.git
cd opencv;
mkdir MYBUILD;
cd MYBUILD;
#sudo mkdir -p /usr/local/lib/opencv;
cmake -L -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local .. ;
echo"check if path is ok?...<any key> or abort";
read inp1; # $inp1
make;
#sudo mkdir -p /usr/local/lib/opencv;
make install;
cd ../../..;
chmod -R 777 3party;
echo "\nDone.\nPlease exit...<any key>";
EDIT: I did tag it cmake because the configuration step is performed with this build tool. Also the tutorial on the OpenCV website stated it. Please correct me if wrong.
Building OpenCV from Source Using CMake, Using the Command Line
Normally, after installation of any package its source and binary directories can be safetly removed. OpenCV follows this convention too.

run a script after deb package is created with cpack

I am trying to create a deb package using cpack. But due to a bug in cpack it is creating file 'md5sums' with wrong permissions and i am getting a warning when installing the deb package using software center. I have a script which will change the permissions of the file from the deb package. But i am confused about how to automatically run the script once the package is made.
You may use post-install script like this:
set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${PROJECT_NAME}/contrib/postinst;")
I used the following method
used a script which has the following content
#!/bin/sh
set -e
mkdir fix_up_deb
dpkg-deb -x #CPACK_PACKAGE_FILE_NAME#.deb fix_up_deb
dpkg-deb --control #CPACK_PACKAGE_FILE_NAME#.deb fix_up_deb/DEBIAN
rm #CPACK_PACKAGE_FILE_NAME#.deb
chmod 0644 fix_up_deb/DEBIAN/md5sums
find -type d -print0 |xargs -0 chmod 755
fakeroot dpkg -b fix_up_deb #CPACK_PACKAGE_FILE_NAME#.deb
rm -rf fix_up_deb
Then configured it using
CONFIGURE_FILE("${PROJECT_SOURCE_DIR}/debian/fixup_deb.sh.in" "${CMAKE_CURRENT_BINARY_DIR}/fixup_deb.sh" #ONLY IMMEDIATE)
Then run it once the package is build using (I havnt tested this step)
add_custom_command(TARGET package POST_BUILD COMMAND bash fixup_deb.sh WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} )
Or run it manually as from the build directory
bash fixup_deb.sh

Let Xcode Project build static library

I have a Core Library for my ios projects. I use this library in several projects.
When building the library, I copied the .a -file and the header-files into several xcode projects. Everything worked fine.
Now I would like that each project, which uses the library, build the library on their own.
I tried to use this runscript:
# Build Core library first
CORE="${PWD}/../Core"
IDENT="PROJECTIDENT"
PROJ=$CORE/Core.xcodeproj
CORE_LIB="${PWD}/Classes/Core"
mkdir -p "${CORE_LIB}"
#rm -rf "${CORE_LIB}/"*
mkdir -p "/tmp/${IDENT}"
IS_EMPTY=`ls "${CORE_LIB}" | wc -l`
if [ "true" == ${ALREADYINVOKED:-false} ]; then
echo "ALREADY INVOKED"
else
# CRITICAL:
# Prevent infinite recursion (Xcode sucks)
export ALREADYINVOKED="true"
SDK_VERSION=$(echo ${SDK_NAME} | grep -o '.\{3\}$')
OUT_DIR="/tmp/${IDENT}"
EXEC_NAME="libCore.a"
UNIVERSAL_DIR="${OUT_DIR}/Universal"
rm -rf "${UNIVERSAL_DIR}"
mkdir -p "${UNIVERSAL_DIR}"
CONFIG="Debug"
if [ ${CONFIGURATION} != "Debug" ]; then
CONFIG="Release"
fi
echo "${CONFIG}"
# Create build
DID_COMPILE=`xcodebuild -project $PROJ -target Core -configuration "${CONFIG}" -sdk "${PLATFORM_NAME}${SDK_VERSION}" build RUN_CLANG_STATIC_ANALYZER=NO BUILD_DIR="${OUT_DIR}" BUILD_ROOT="${OUT_DIR}" EXECUTABLE_NAME="${EXEC_NAME}" | grep "CompileC"`
if [ ! -z "${DID_COMPILE}" ] || [ ${IS_EMPTY} -eq 0 ]; then
# Copy Headers
cp "${OUT_DIR}/${CONFIG}-${PLATFORM_NAME}/usr/local/include/"* "${UNIVERSAL_DIR}"
cp "${OUT_DIR}/${CONFIG}-${PLATFORM_NAME}/${EXEC_NAME}"* "${UNIVERSAL_DIR}"
# Finally we have a fat static library with .a and .h files
# now we copy them to the Project
rm -rf "${CORE_LIB}"
cp -Rf "${UNIVERSAL_DIR}" "${CORE_LIB}"
fi
fi
This script works but only every second time.
E.g. I make a change in a Library Class, and built, the main project is not compiled while the Library dependency get refreshed.
Then I make another change to the Library, build again, and all changes are usable in the main probject.
But always only each second build....
How can I fix this?