py.test is know tool for testing Python scripts. I am missing, however, CMake macro for the detetion of py.test.
Is there anything of this kind within CMake pool of macros ?
Directly calling py.test from a CMakeLists.txt isn't very portable, because different systems name it quite differently, e.g.:
py.test # pip, as of early 2018
pytest # ditto
py.test-3 # Fedora 26 package
pytest-3 # ditto
Of course, the system might default to version 2 or 3, and might have both version 2 and 3 installed side by side.
Thus, a simple method for reliable executing right pytest version via cmake is to invoke pytest differently:
python3 -m pytest
(instead of py.test or pytest or ...)
If you want to test in cmake if the pytest package is available you can test this with execute_process(), e.g.:
execute_process(COMMAND python3 -m pytest --version
OUTPUT_VARIABLE PYTEST_output
ERROR_VARIABLE PYTEST_error
RESULT_VARIABLE PYTEST_result)
if(${PYTEST_result} UNEQUAL 0)
message(SEND_ERROR "Pytest package not available: ${PYTEST_error}")
endif()
Related
I find myself going against the grain configuring cmake paths with ccmake over and over again as with every change of for ex. compiler some of my library paths get lost.
In particular paths to (unlinked) lapack, lapacke, gsl get either lost or set to system defaults instead the ones I've installed with brew.
There has to be a way to tell cmake to "ignore" system libraries and instead look in homebrew paths (say. /opt/homebrew/lib, /opt/homebrew/include etc.).
I'd prefer not to link those libraries as this is not recommend and I'm not experienced in switching environments.
[EDIT] MRE:
git clone https://gitlab.physik.uni-muenchen.de/AG-Scrinzi/tRecX.git
cd tRecX
cmake . -DCMAKE_BUILD_TYPE=Parallel
make -j 8
I add the following to .bash_profile/.zshrc:
export LDFLAGS="-L/opt/homebrew/opt/lapack/lib -L/opt/homebrew/opt/lapack/lib"
export CPPFLAGS="-I/opt/homebrew/opt/lapack/include -I/opt/homebrew/opt/openblas/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/lapack/lib/pkgconfig /opt/homebrew/opt/openblas/lib/pkgconfig"
then I try:
cmake . -DCMAKE_PREFIX_PATH=/opt/homebrew -DCMAKE_FIND_FRAMEWORK=NEVER -DCMAKE_FIND_APPBUNDLE=NEVER -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE -DCMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH=FALSE -DMPI_CXX_COMPILER=/opt/homebrew/bin/mpicxx -DMPI_C_COMPILER=/opt/homebrew/bin/mpicc -DCMAKE_CXX_COMPILER=/opt/homebrew/bin/g++-11 -DCMAKE_C_COMPILER=/opt/homebrew/bin/gcc-11
The most common solution is to just set CMAKE_PREFIX_PATH to /opt/homebrew. CMake will then look preferentially in /opt/homebrew for everything. Since you're on Apple, you might need to set CMAKE_FIND_FRAMEWORK and CMAKE_FIND_APPBUNDLE to LAST or NEVER, too.
You can skip the standard platform search paths by setting CMAKE_FIND_USE_CMAKE_SYSTEM_PATH to FALSE at the command line, in a preset, or in a toolchain file. You might also wish to disable looking at the PATH environment variable by setting CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH to FALSE.
Finally, if you're in a cross-compiling scenario or toolchain file, you can change the definition of the system directories by setting CMAKE_SYSROOT. Note that the sysroot will have to contain the language runtime libraries (e.g. glibc) and will be passed to the --sysroot flag (or equivalent). Just be aware of those effects, too.
All of this is documented here:
https://cmake.org/cmake/help/latest/command/find_package.html#search-procedure
https://cmake.org/cmake/help/latest/variable/CMAKE_FIND_FRAMEWORK.html#variable:CMAKE_FIND_FRAMEWORK
https://cmake.org/cmake/help/latest/variable/CMAKE_FIND_APPBUNDLE.html#variable:CMAKE_FIND_APPBUNDLE
The following homebrew.cmake toolchain file worked for me:
set(HOMEBREW_PREFIX "/usr/local"
CACHE PATH "Path to Homebrew installation")
set(CMAKE_C_COMPILER "${HOMEBREW_PREFIX}/bin/gcc-11")
set(CMAKE_CXX_COMPILER "${HOMEBREW_PREFIX}/bin/g++-11")
set(CMAKE_PREFIX_PATH
"${HOMEBREW_PREFIX}"
# These libraries are keg-only and not loaded into
# the root prefix by default (to avoid clashes).
"${HOMEBREW_PREFIX}/opt/lapack"
"${HOMEBREW_PREFIX}/opt/openblas"
"${HOMEBREW_PREFIX}/opt/gcc/lib/gcc/11"
)
list(TRANSFORM CMAKE_PREFIX_PATH APPEND "/include"
OUTPUT_VARIABLE CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES)
set(CMAKE_C_STANDARD_INCLUDE_DIRECTORIES "${CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES}")
set(CMAKE_FIND_FRAMEWORK NEVER)
set(CMAKE_FIND_APPBUNDLE NEVER)
set(CMAKE_FIND_USE_CMAKE_SYSTEM_PATH FALSE)
set(CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH FALSE)
I built with the following commands:
$ ls
tRecX homebrew.cmake
$ cmake -G Ninja -S tRecX -B tRecX-build \
-DCMAKE_TOOLCHAIN_FILE=$PWD/homebrew.cmake \
-DCBLAS=/usr/local/opt/openblas/lib/libblas.dylib \
-DCMAKE_EXE_LINKER_FLAGS="-Wl,-undefined,dynamic_lookup" \
-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-undefined,dynamic_lookup" \
-DCMAKE_BUILD_TYPE=Parallel
[ ... output clipped ... ]
Boost found -- full functionality
Build "Parallel" with C++ flags -D_USE_BOOST_ -O3 -pthread -D_USE_FFTW_, return to default by -UCMAKE_BUILD_TYPE
Compiler: /usr/local/bin/g++-11, change by -DCMAKE_CXX_COMPILER=[path_to_complier]
-- Linking to libraries Boost::system;Boost::filesystem;/usr/local/lib/libfftw3.dylib;/usr/local/opt/gcc/lib/gcc/11/libgfortran.dylib;alglib;/usr/local/lib/libarpack.dylib;Boost::system;Boost::filesystem;/usr/local/opt/lapack/lib/liblapacke.dylib;/usr/local/opt/openblas/lib/libblas.dylib;/usr/local/opt/lapack/lib/liblapack.dylib;/usr/local/opt/lapack/lib/libblas.dylib;m
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/alexreinking/Development/tRecX-build
$ cmake --build tRecX-build
I had to set CBLAS manually because libblas.dylib provides the OpenBLAS CBLAS interface, but the build system specifically looks for a library named libcblas. There's no other option in this case.
The code and build have issues with its linking model and dependencies. I was able to paper over these by setting -Wl,-undefined,dynamic_lookup. However, note that this will just defer linker errors to runtime and might impose a large startup cost.
If you can make commits to the project, I would store these settings in a preset, maybe name it homebrew-parallel or something:
-DCMAKE_TOOLCHAIN_FILE=$PWD/homebrew.cmake \
-DCBLAS=/usr/local/opt/openblas/lib/libblas.dylib \
-DCMAKE_EXE_LINKER_FLAGS="-Wl,-undefined,dynamic_lookup" \
-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-undefined,dynamic_lookup" \
-DCMAKE_BUILD_TYPE=Parallel
Then you could just run cmake --preset=homebrew-parallel
I am trying to compile Trilinos with MPI capabilities. But to specify the cmake command, i need to also specify the MPI base directory:
cmake \
-DTPL_ENABLE_MPI=ON \
-DMPI_BASE_DIR:FILEPATH="" \
-DTrilinos_ENABLE_PyTrilinos:BOOL=ON \
-DTrilinos_ENABLE_ALL_PACKAGES=ON \
-DTrilinos_ENABLE_TESTS:BOOL=ON \
-DBUILD_SHARED_LIBS:BOOL=ON \
-DCMAKE_INSTALL_PREFIX:STRING="$HOME/trilinos-install" \
$SOURCE_DIR
However, I am unable to find any base directory even though MPI is installed on my machine. When i enter commands like mpirun --version, I get:
mpirun (Open MPI) 2.1.1
or ompi_info:
Package: Open MPI buildd#lcy01-amd64-009 Distribution
Open MPI: 2.1.1
Open MPI repo revision: v2.1.0-100-ga2fdb5b
Open MPI release date: May 10, 2017
Open RTE: 2.1.1
...
I am running Ubuntu 18.04 LTS on WSL if that is useful info.
The command "which mpich" can give you the default directory of mpi installation.
Another way is to use mpic++ as the compiler in the CMake.
I would like to run program to perform do extra installation tasks from CMake. My attempted solution, based on INSTALL(CODE ...) is (this is a real MWE):
macro(MY_EXTRA_STUFF ARG)
execute_process(...)
endmacro()
install(CODE "MY_EXTRA_STUFF(${SOME_ARG})")
but CMake complains when I run ninja install (or make install, depending on generator in use):
[0/1] Install the project...
-- Install configuration: ""
CMake Error at cmake_install.cmake:41 (MY_EXTRA_STUFF):
Unknown CMake command "MY_EXTRA_STUFF".
FAILED: CMakeFiles/install.util
cd /tmp && /usr/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.
Is there a way to smuggle my own code into the install stage? The code is too long to fit inside install(CODE "...") nicely. A bonus to do it without an external file. Thanks!
The code passed to install(CODE) is executed as standalone CMake code, thus it shouldn't use definitions (functions,macros, variables) from the rest of CMakeLists.txt.
That is, install(CODE) behaves similar as install(SCRIPT) with a standalone script containing given code.
The thing is that configuration stage (when you call cmake to configure your project) and installation stage, which, as you can see, calls /usr/bin/cmake -P cmake_install.cmake, are separate cmake invocations. These invocations parse different files, so they unaware about context of each other.
For a project I build a C library and implict Python bindings (via GObject introspection) with CMake. I also want to distribute some Python helper modules using distutils. I am able to build and install the module with this CMakeLists.txt
find_program(PYTHON "python")
if (PYTHON)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(DEPS "${CMAKE_CURRENT_SOURCE_DIR}/module/__init__.py")
set(OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(OUTPUT ${OUTPUT}
COMMAND ${PYTHON}
ARGS setup.py build
DEPENDS ${DEPS})
add_custom_target(target ALL DEPENDS ${OUTPUT})
install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install)")
endif()
and the following setup.py.in:
from distutils.core import setup, Extension
if __name__ == '__main__':
setup(name='foo',
version='${PACKAGE_VERSION}',
package_dir={ '': '${CMAKE_CURRENT_SOURCE_DIR}' },
packages=['module'])
Unfortunately, the build step is executed each time I run make. I guess, the problem is related to the output of the custom command which is a directory rather than a file. Now, is there any way to tell CMake to run python setup.py build only when setup.py.in or one of the sources changed?
Only files, not directories, can be reliably used as OUTPUT and DEPENDS. You could modify your custom command to also produce a timestamp file, something like this:
add_custom_command(
OUTPUT ${OUTPUT}/timestamp
COMMAND ${PYTHON} setup.py build
COMMAND ${CMAKE_COMMAND} -E touch ${OUTPUT}/timestamp
DEPENDS ${DEPS}
)
add_custom_target(target ALL DEPENDS ${OUTPUT}/timestamp)
Is it somehow possible to be able to have a parallel build no matter which build tool is used?
Under Unix we can add make -jN where N are the number of threads, and under Windows I added to the CXX_FLAG "/MP" which is then used in Visual Studio to parallel build...(?) How can I make my version such that CMAKE_MAKE_PROGRAM is not always extended when I run CMake?
What is a general solution?
I came up with this:
# Add some multithreaded build support
MARK_AS_ADVANCED(MULTITHREADED_BUILD)
set(MULTITHREADED_BUILD 12 CACHE STRING "How many threads are used to build the project")
if(MULTITHREADED_BUILD)
if(${CMAKE_GENERATOR} MATCHES "Unix Makefiles")
message(STATUS ${CMAKE_BUILD_TOOL})
set(CMAKE_MAKE_PROGRAM "${CMAKE_MAKE_PROGRAM} -j${MULTITHREADED_BUILD}")
message(STATUS "Added arguments to CMAKE_BUILD_TOOL: ${CMAKE_MAKE_PROGRAM}")
elseif(MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /MP")
message(STATUS "Added parallel build arguments to CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
endif()
endif()
With CMake 3.12 this is possible. From the release notes:
The cmake(1) Build a Project (cmake --build) gained --parallel [<jobs>] and -j [<jobs>] options to specify a parallel build level. They map to corresponding options of the native build tool.
As mentioned by dkg, you can also set the environment variable CMAKE_BUILD_PARALLEL_LEVEL.
Links to CMake's documentation:
Build a Project
CMAKE_BUILD_PARALLEL_LEVEL
If you have CMake v2.8.8 or higher, you may use Ninja as an alternative of GNU make:
mkdir build
cd build
cmake -G Ninja ..
ninja # Parallel build (no need -j12)
or
mkdir build
cd build
cmake -G Ninja ..
cmake --build . # Parallel build using Ninja
As you can see, no need to use CMAKE_MAKE_PROGRAM, the build is run in parallel by default, optimizing the number of jobs depending on available CPU cores.
Ninja is based on a low-level JSON configuration to speed up the startup phase. Therefore its JSON configuration is not easy to write by hand, and I always generate it using a high-level tool/IDE:
CMake v2.8.8 (2012)
Qt Creator v2.6 (2012)
KDevelop v4.6 (2013)
Meson on Linux (2013)
... see the generators of Ninja configuration at https://github.com/ninja-build/ninja/wiki/List-of-generators-producing-ninja-build-files
As a C++ build often requires lots of memory, your computer must provide as much memory as the number of CPU cores.
As pointed out by Ruslan, CMake 3.12 (2018) has a new option cmake --build -j <N> to limit build to <N> cores (jobs) thus limiting the memory consumption (see also the documentation). If you use an older CMake version, you can still use cmake --build -- -j <N>. The option -- tells to CMake to pass the rest directly to the underlying builder tool, here it is Ninja.
Now it is very simple to parallel build with cmake. You can add "-j jobs_number" when using "cmake --build". For example:
cmake --build . -j 24
More details can be found in CMAKE manual: https://cmake.org/cmake/help/latest/manual/cmake.1.html#build-tool-mode
--parallel [], -j [] The maximum number of concurrent processes to use when building. If is omitted the native build
tool’s default number is used.
The CMAKE_BUILD_PARALLEL_LEVEL environment variable, if set, specifies
a default parallel level when this option is not given.
Some native build tools always build in parallel. The use of
value of 1 can be used to limit to a single job.
As already mention above one can use --parallel [<jobs>] (or -j [<jobs>]) option of CMake to build solution in parallel.
But I would like to note that in Windows a command
cmake.exe --build --parallel <jobs>
starts a root MSBuild process with the following arguments:
msbuild.exe /m:<jobs> /p:CL_MPCount=1 <another-useful-arguments...>
This means the root MSBuild process will start multiple child MSBuild processes due to /m:<jobs> (or /maxcpucount:<jobs>). But each of this child MSBuild runs at most one compiler process due to /p:CL_MPCount=1 argument (for more details find this link). In other words each MSBuild can compile at most one source file at the same time.
To overcome this limitation one may call CMake in the following way:
cmake.exe --build --parallel <n_msbuild> -- /p:CL_MPcount=<n_cl>
This approach sets /MP option thereby allows multiple compiler processes under each MSBuild instance.
And for the record. To configure builds ran form Visual Studio GUI one need to go to Settings:
Option "Projects and Solutions → Build And Run → maximum number of parallel projects build" affects on maxcpucount.
Option "Projects and Solutions → VC++ Project Settings → Build → Maximum Concurrent C++ Compilations" represents the default boundary to /MP.
You can't do this cross-platform. The -jN option is a parameter to make, and not part of the generated Makefile. However, you could have CMake generate a Bash script that runs make for your project using -jN (where the script looks up the number of cores you have).
I have settled down to writing a parallelmake.sh script for Unix Makefiles-based generators. This is done here: https://github.com/gabyx/ApproxMVBB
And the relevant parts in the the CMake file:
https://github.com/gabyx/ApproxMVBB/blob/master/CMakeLists.txt#L89
# Add some multithreaded build support =====================================================================================================
MARK_AS_ADVANCED(MULTITHREADED_BUILD)
SET(MULTITHREADED_BUILD ON CACHE BOOL "Parallel build with as many threads as possible!")
if(MULTITHREADED_BUILD)
if(${CMAKE_GENERATOR} MATCHES "Unix Makefiles")
file(COPY ${ApproxMVBB_ROOT_DIR}/cmake/parallelmake.sh DESTINATION ${PROJECT_BINARY_DIR}
FILE_PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE
NO_SOURCE_PERMISSIONS
)
SET(CMAKE_MAKE_PROGRAM "${PROJECT_BINARY_DIR}/parallelmake.sh")
MESSAGE(STATUS "Set make program to ${PROJECT_BINARY_DIR}/parallelmake.sh")
elseif(MSVC)
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}" "/MP")
MESSAGE(STATUS "Added parallel build arguments to CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
endif()
endif()
# ========================================================================================================================================