cmake version 3.20
For this zip file:
http://kakadusoftware.com/wp-content/uploads/KDU805_Demo_Apps_for_Win64_200602.msi_.zip
When running
cmake -E tar xvz KDU805_Demo_Apps_for_Win64_200602.msi_.zip
the msi is only 176 bytes in size : should be around 20 MB.
Is this a known issue for cmake tar ?
The z flag in your command specified that it should be extracted using gzip. Remove that flag.
cmake -E tar xv KDU805_Demo_Apps_for_Win64_200602.msi_.zip
Related
I'm trying to speed up build process using ramdisk for build directory.
I've created ramdisk:
sudo mount -t tmpfs -o size=1024m tmpfs /mnt/ramdisk
On ramdisk I've created build dir:
mkdir -p /mnt/ramdisk/rust/hello3/build/
Then I've symlinked ramdisk build dir to project in which I want to use this directory:
cd /home/wakatana/rust/hello3
ln -s /mnt/ramdisk/rust/hello3/build/ build
After this I did classic combo for building project:
cd /home/wakatana/rust/hello3/build
cmake ..
make
But above command does not worked because relative path (cmake ..) is translated to /mnt/ramdisk/rust/hello3 and not to /home/wakatana/rust/hello3/ (I suspect that this is whole problem)
So instead of classic combo I did a little bit modified combo (when build dir is not symlinked this works):
cd /home/wakatana/rust/hello3/build
cmake /home/wakatana/rust/hello3
make
But this ends up with errors during make phase:
-- Configuring done
-- Generating done
-- Build files have been written to: /home/wakatana/rust/hello3/build
make[2]: *** No rule to make target '../src/lib.rs', needed by 'src/x86_64-unknown-linux-gnu/debug/libtest_lib.a'. Stop.
CMakeFiles/Makefile2:122: recipe for target 'src/CMakeFiles/test-lib_target.dir/all' failed
make[1]: *** [src/CMakeFiles/test-lib_target.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
Is it possible to somehow tell to cmake/make to deal symlinks correctly?
Just save yourself the trouble of creating symlinks and work on the ram disc:
cmake -S /home/wakatana/rust/hello3 -B /mnt/ramdisk/rust/hello3/build/
cmake --build /mnt/ramdisk/rust/hello3/build/
You could create the symlink, and then work from parent dir:
ln -s /mnt/ramdisk/rust/hello3/build/
cd /home/wakatana/rust/hello3
cmake -S . -B build
cmake --build build
# or expand the symlink before cmake has to:
cmake -S . -B "$(readlink -f "./build")"
cmake --build "$(readlink -f "./build")"
The other way is to rebind your RAM disk into your project tree instead of symlinking:
$ cd /home/wakatana/rust/hello3
$ mkdir -p build
$ mount --bind /mnt/ramdisk/rust/hello3/build build
I have a Makefile generated by CMake. The following path to CMake executable is set in the Makefile:
CMAKE_COMMAND = /home/xyz/opt/cmake/cmake-3.1.1/bin/cmake
How can I integrate Fortify sourceanalyzer with it and run scans?
I had the same challenge but solved it by running it like this:
sourceanalyzer -b project_ID -clean
Go to your build directory and perform make clean or remove all contents including the Makefile
Run cmake by changing CC and CXX variables:
CC="sourceanalyzer -b project_ID gcc" CXX="sourceanalyzer -b project_ID g++" cmake ..
Run make and fortify should be translating files while compilers do their job.
Run sourceanalyzer -b project_ID -scan -f results.fpr
Hope it helps.
I was tasked with integrating our CMake build system with HP Fortify SCA and came across this Thread that gave some insights but lacked specifics as related to HP Fortify so I thought I would share my implementation.
I created a fortify_tools directory at the same level as the source directory. Inside the fortify_tools are a toolchain file and fortify_cc, fortify_cxx, and fortify_ar scripts that will be set as the cmake_compilers via the toolchain file.
fortify_cc
#!/bin/bash
sourceanalyzer -b <PROJECT_ID> gcc $#
fortify_cxx
#!/bin/bash
sourceanalyzer -b <PROJECT_ID> g++ $#
fortify_ar
#!/bin/bash
sourceanalyzer -b <PROJECT_ID> ar $#
NOTE: insert your project name in place of PROJECT_ID
Setting cmake to use the scripts is accomplished in a toolchain file.
fortify_linux_toolchain.cmake
INCLUDE (CMakeForceCompiler)
SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)
#specify the compilers
SET(CMAKE_C_COMPILER ${CMAKE_SOURCE_DIR}/fortify_tools/fortify_cc)
SET(CMAKE_CXX_COMPILER ${CMAKE_SOURCE_DIR}/fortify_tools/fortify_cxx)
SET(CMAKE_AR_COMPILER ${CMAKE_SOURCE_DIR}/fortify_tools/fortify_ar)
To generate makefiles using the toolchain file
ccmake -DCMAKE_TOOLCHAIN_FILE=../fortify_tools/foritfy_linux_toolchain.cmake ../
configure and generate your makefiles and build your project.
Once the project is built from within the build directory generate a fortify report by
sourceanalyzer -Xmx2400M -debug -verbose -b <PROJECT_ID> -scan -f <PROJECT_ID>.fpr
I understand the last step is outside of CMake but I am pretty confident a cmake_custom_command can be created to perform the scan step as a post build action.
Finally, this is just the linux implementation but the concept scales well to Windows by creating the necessary batch files and windows specific toolchain file
Fortify doesn't support CMake, I received confirmation from Fortify support team.
This answer is late, but might help someone. This is actually easy to fix - you simply need to run cmake inside sourceanalyzer as well. Make a simple build script that calls cmake and then make, and use sourceanalyzer on that instead. I am using fortify 4.21.
Our old Fortify script for building hand-created Makefiles used a build command that looked like this:
$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID make -f Makefile -j12
I was able to get it working for a project that had been converted to CMake by replacing the above line with this, inspired by a couple of the other answers here:
CC="$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID gcc" \
CXX="$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID g++" \
AR="$SOURCEANALYZER $MEMORY $LAUNCHERSWITCHES -b $BUILDID ar" \
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug ..
make -f Makefile -j12 VERBOSE=1
This is with cmake 2.8.12.2 on Linux.
Below is the script i use for my example project to generate HP Fortify report for Android JNI C/C++ Code.
#!/bin/sh
# Configure NDK version and CMake version
NDK_VERSION=21.0.6113669
CMAKE_VERSION=3.10.2
CMAKE_VERSION_PATH=$CMAKE_VERSION.4988404
PROJECTID="JNI_EXAMPLE"
REPORT_NAME=$PROJECTID"_$(date +'%Y%m%d_%H:%M:%S')"
WORKING_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
BUILD_HOME=${WORKING_DIR}/../hpfortify_build
FPR="$BUILD_HOME/$REPORT_NAME.fpr"
# Following exports need to be configured according to host machine.
export ANDROID_SDK_HOME=/Library/Android/sdk
export ANDROID_CMAKE_HOME=$ANDROID_SDK_HOME/cmake/$CMAKE_VERSION_PATH/bin
export ANDROID_NDK_HOME=$ANDROID_SDK_HOME/ndk/$NDK_VERSION
# E.g. JniExample/app/hpfortify/build/CMakeFiles/3.10.2
export CMAKE_FILES_PATH=${BUILD_HOME}/CMakeFiles/$CMAKE_VERSION
export HPFORTIFY_HOME="/Applications/Fortify/Fortify_SCA_and_Apps_20.1.0/bin"
export PATH=$PATH:$ANDROID_SDK_HOME:$ANDROID_NDK_HOME:$ANDROID_CMAKE_HOME:$HPFORTIFY_HOME
echo "[========Start Android JNI C/C++ HP Fortify scanning========]"
echo "[========Build Dir: $BUILD_HOME========]"
echo "[========HP Fortify report path: $FPR========]"
function create_build_folder {
rm -rf $BUILD_HOME
mkdir $BUILD_HOME
}
# The standalone cmake build command can be found from below file.
# JniExample/app/.cxx/cmake/release/x86/build_command.txt
# This file is generated after running command
# `➜ JniExample git:(master) ✗ ./gradlew :app:externalNativeBuildRelease`
function configure_cmake_files {
cd $BUILD_HOME
$ANDROID_CMAKE_HOME/cmake -H$BUILD_HOME/. \
-DCMAKE_CXX_FLAGS=-std=c++11 -frtti -fexceptions \
-DCMAKE_FIND_ROOT_PATH=$BUILD_HOME/.cxx/cmake/release/prefab/x86/prefab \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_TOOLCHAIN_FILE=$ANDROID_SDK_HOME/ndk/$NDK_VERSION/build/cmake/android.toolchain.cmake \
-DANDROID_ABI=x86 \
-DANDROID_NDK=$ANDROID_SDK_HOME/ndk/$NDK_VERSION \
-DANDROID_PLATFORM=android-16 \
-DCMAKE_ANDROID_ARCH_ABI=x86 \
-DCMAKE_ANDROID_NDK=$ANDROID_SDK_HOME/ndk/$NDK_VERSION \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=$BUILD_HOME/intermediates/cmake/release/obj/x86 \
-DCMAKE_MAKE_PROGRAM=$ANDROID_SDK_HOME/cmake/$CMAKE_VERSION_PATH/bin/ninja \
-DCMAKE_SYSTEM_NAME=Android \
-DCMAKE_SYSTEM_VERSION=16 \
-B$BUILD_HOME/.cxx/cmake/release/x86 \
-GNinja ..
}
function build {
cmake --build .
}
function cleanup {
rm -rf $BUILD_HOME/CMakeFiles/native-lib.dir
rm -rf $FPR
$HPFORTIFY_HOME/sourceanalyzer -clean
}
function replace_compiler_paths {
FORTIFY_TOOLS_PATH="$WORKING_DIR"
CLANG_PATH="$ANDROID_SDK_HOME/ndk/$NDK_VERSION/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang"
CLANGXX_PATH="$ANDROID_SDK_HOME/ndk/$NDK_VERSION/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++"
HPFORTIFY_CCPATH="$FORTIFY_TOOLS_PATH/fortify_cc"
HPFORTIFY_CXXPATH="$FORTIFY_TOOLS_PATH/fortify_cxx"\"
sed -i '' 's+'$CLANG_PATH'+'$HPFORTIFY_CCPATH'+g' $CMAKE_FILES_PATH/CMakeCCompiler.cmake
sed -i '' 's+'$CLANG_PATH.*[^")"]'+'$HPFORTIFY_CXXPATH'+g' $CMAKE_FILES_PATH/CMakeCXXCompiler.cmake
}
function scan {
$HPFORTIFY_HOME/sourceanalyzer -b $PROJECTID -scan -f $FPR
# copy the file to $WORKING_DIR
cp $FPR $WORKING_DIR
}
create_build_folder
configure_cmake_files
echo "[========Compile C/C++ using normal compiler ========"]
build
echo "[========Replace the compiler with HP Fortify analyser wrapper compilers ========"]
replace_compiler_paths
echo "[========Clean up the build intermediates and the older build ID and fpr file ========"]
cleanup
echo "[========Recompile C/C++ using HP Fortify analyser wrapper compilers ========"]
build
echo "[========Scan the compiled files and generate final report ========"]
scan
echo "[========Change directory to original working dir ========"]
cd $WORKING_DIR
Need to configure below vars before using it. For my case, I use NDK 21 and CMake 3.10.2 and my project ID is "JNI_EXAMPLE"
# Configure NDK version and CMake version
NDK_VERSION=21.0.6113669
CMAKE_VERSION=3.10.2
CMAKE_VERSION_PATH=$CMAKE_VERSION.4988404
PROJECTID="JNI_EXAMPLE"
# Following exports need to be configured according to host machine.
export ANDROID_SDK_HOME=/Library/Android/sdk
export ANDROID_NDK_HOME=$ANDROID_SDK_HOME/ndk/$NDK_VERSION
export HPFORTIFY_HOME="/Applications/Fortify/Fortify_SCA_and_Apps_20.1.0/bin"
Here is a more detailed explanation: Using HP Fortify to Scan Android JNI C/C++ Code
On recent version of CMake one can use:
CMAKE_<LANG>_COMPILER_LAUNCHER='sourceanalyzer;-b;<PROJECT_ID>'
You can add other arguments (like -Xmx2G for instance), semicolon separated, as mentioned on cmake documentation
You need to check if you don't use the compiler launcher for another tool like ccache. We can probably use both with
CCACHE_PREFIX='.../sourceanalyzer -b ID'
Here is what I've used in CMake project:
project(myFortifiedProject LANGUAGES CXX)
set(CMAKE_CXX_COMPILER_LAUNCHER ${FORTIFY_TOOL} -b ${PROJECT_NAME})
So when running cmake (assuming sourceanalyzer is on the path):
cmake <other args> -DFORTIFY_TOOL=sourceanalyzer
So the normal build command works:
make myFortifiedProject
And you can finally collect results with:
sourceanalyzer -b myFortifiedProject -scan
I wrote myself a litte script to install opencv under ubuntu14.04. Can I remove the directory 3party after the make install sorted the lib into system directories or are there dependencies? (Remove not only the MYBUILD but the complete 3party)
echo "\nInstall OpenCV?...<any key>\n"
read inp1; # $inp1
mkdir 3party;
cd 3party;
git clone https://github.com/Itseez/opencv.git
cd opencv;
mkdir MYBUILD;
cd MYBUILD;
#sudo mkdir -p /usr/local/lib/opencv;
cmake -L -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local .. ;
echo"check if path is ok?...<any key> or abort";
read inp1; # $inp1
make;
#sudo mkdir -p /usr/local/lib/opencv;
make install;
cd ../../..;
chmod -R 777 3party;
echo "\nDone.\nPlease exit...<any key>";
EDIT: I did tag it cmake because the configuration step is performed with this build tool. Also the tutorial on the OpenCV website stated it. Please correct me if wrong.
Building OpenCV from Source Using CMake, Using the Command Line
Normally, after installation of any package its source and binary directories can be safetly removed. OpenCV follows this convention too.
I am trying to create a deb package using cpack. But due to a bug in cpack it is creating file 'md5sums' with wrong permissions and i am getting a warning when installing the deb package using software center. I have a script which will change the permissions of the file from the deb package. But i am confused about how to automatically run the script once the package is made.
You may use post-install script like this:
set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${PROJECT_NAME}/contrib/postinst;")
I used the following method
used a script which has the following content
#!/bin/sh
set -e
mkdir fix_up_deb
dpkg-deb -x #CPACK_PACKAGE_FILE_NAME#.deb fix_up_deb
dpkg-deb --control #CPACK_PACKAGE_FILE_NAME#.deb fix_up_deb/DEBIAN
rm #CPACK_PACKAGE_FILE_NAME#.deb
chmod 0644 fix_up_deb/DEBIAN/md5sums
find -type d -print0 |xargs -0 chmod 755
fakeroot dpkg -b fix_up_deb #CPACK_PACKAGE_FILE_NAME#.deb
rm -rf fix_up_deb
Then configured it using
CONFIGURE_FILE("${PROJECT_SOURCE_DIR}/debian/fixup_deb.sh.in" "${CMAKE_CURRENT_BINARY_DIR}/fixup_deb.sh" #ONLY IMMEDIATE)
Then run it once the package is build using (I havnt tested this step)
add_custom_command(TARGET package POST_BUILD COMMAND bash fixup_deb.sh WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} )
Or run it manually as from the build directory
bash fixup_deb.sh
I normally compress using tar zcvf and decompress using tar zxvf (using gzip due to habit).
I've recently gotten a quad core CPU with hyperthreading, so I have 8 logical cores, and I notice that many of the cores are unused during compression/decompression.
Is there any way I can utilize the unused cores to make it faster?
You can also use the tar flag "--use-compress-program=" to tell tar what compression program to use.
For example use:
tar -c --use-compress-program=pigz -f tar.file dir_to_zip
You can use pigz instead of gzip, which does gzip compression on multiple cores. Instead of using the -z option, you would pipe it through pigz:
tar cf - paths-to-archive | pigz > archive.tar.gz
By default, pigz uses the number of available cores, or eight if it could not query that. You can ask for more with -p n, e.g. -p 32. pigz has the same options as gzip, so you can request better compression with -9. E.g.
tar cf - paths-to-archive | pigz -9 -p 32 > archive.tar.gz
Common approach
There is option for tar program:
-I, --use-compress-program PROG
filter through PROG (must accept -d)
You can use multithread version of archiver or compressor utility.
Most popular multithread archivers are pigz (instead of gzip) and pbzip2 (instead of bzip2). For instance:
$ tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 paths_to_archive
$ tar --use-compress-program=pigz -cf OUTPUT_FILE.tar.gz paths_to_archive
Archiver must accept -d. If your replacement utility hasn't this parameter and/or you need specify additional parameters, then use pipes (add parameters if necessary):
$ tar cf - paths_to_archive | pbzip2 > OUTPUT_FILE.tar.gz
$ tar cf - paths_to_archive | pigz > OUTPUT_FILE.tar.gz
Input and output of singlethread and multithread are compatible. You can compress using multithread version and decompress using singlethread version and vice versa.
p7zip
For p7zip for compression you need a small shell script like the following:
#!/bin/sh
case $1 in
-d) 7za -txz -si -so e;;
*) 7za -txz -si -so a .;;
esac 2>/dev/null
Save it as 7zhelper.sh. Here the example of usage:
$ tar -I 7zhelper.sh -cf OUTPUT_FILE.tar.7z paths_to_archive
$ tar -I 7zhelper.sh -xf OUTPUT_FILE.tar.7z
xz
Regarding multithreaded XZ support. If you are running version 5.2.0 or above of XZ Utils, you can utilize multiple cores for compression by setting -T or --threads to an appropriate value via the environmental variable XZ_DEFAULTS (e.g. XZ_DEFAULTS="-T 0").
This is a fragment of man for 5.1.0alpha version:
Multithreaded compression and decompression are not implemented yet, so this
option has no effect for now.
However this will not work for decompression of files that haven't also
been compressed with threading enabled. From man for version 5.2.2:
Threaded decompression hasn't been implemented yet. It will only work
on files that contain multiple blocks with size information in
block headers. All files compressed in multi-threaded mode meet this
condition, but files compressed in single-threaded mode don't even if
--block-size=size is used.
Recompiling with replacement
If you build tar from sources, then you can recompile with parameters
--with-gzip=pigz
--with-bzip2=lbzip2
--with-lzip=plzip
After recompiling tar with these options you can check the output of tar's help:
$ tar --help | grep "lbzip2\|plzip\|pigz"
-j, --bzip2 filter the archive through lbzip2
--lzip filter the archive through plzip
-z, --gzip, --gunzip, --ungzip filter the archive through pigz
You can use the shortcut -I for tar's --use-compress-program switch, and invoke pbzip2 for bzip2 compression on multiple cores:
tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 DIRECTORY_TO_COMPRESS/
If you want to have more flexibility with filenames and compression options, you can use:
find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec \
tar -P --transform='s#/my/path/##g' -cf - {} + | \
pigz -9 -p 4 > myarchive.tar.gz
Step 1: find
find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec
This command will look for the files you want to archive, in this case /my/path/*.sql and /my/path/*.log. Add as many -o -name "pattern" as you want.
-exec will execute the next command using the results of find: tar
Step 2: tar
tar -P --transform='s#/my/path/##g' -cf - {} +
--transform is a simple string replacement parameter. It will strip the path of the files from the archive so the tarball's root becomes the current directory when extracting. Note that you can't use -C option to change directory as you'll lose benefits of find: all files of the directory would be included.
-P tells tar to use absolute paths, so it doesn't trigger the warning "Removing leading `/' from member names". Leading '/' with be removed by --transform anyway.
-cf - tells tar to use the tarball name we'll specify later
{} + uses everyfiles that find found previously
Step 3: pigz
pigz -9 -p 4
Use as many parameters as you want.
In this case -9 is the compression level and -p 4 is the number of cores dedicated to compression.
If you run this on a heavy loaded webserver, you probably don't want to use all available cores.
Step 4: archive name
> myarchive.tar.gz
Finally.
A relatively newer (de)compression tool you might want to consider is zstandard. It does an excellent job of utilizing spare cores, and it has made some great trade-offs when it comes to compression ratio vs. (de)compression time. It is also highly tweak-able depending on your compression ratio needs.
Here is an example for tar with modern zstd compressor, as finding out good examples on this one was difficult:
apt poem to install zstd and pv utilities for Ubuntu
Compress multiple files and folders (zstd command alone can only do single files)
Display progress using pv - shows the total bytes compressed and compression speed GB/sec real-time
Use all physical cores with -T0
Set compression level higher than the default with -8
Display the resulting wall clock and CPU time used after the operation is finished using time
apt install zstd pv
DATA_DIR=/path/to/my/folder/to/compress
TARGET=/path/to/my/arcive.tar.zst
time (cd $DATA_DIR && tar -cf - * | pv | zstd -T0 -8 -o $TARGET)