I'm have dependency on Xerces-c++ and for instal target it always builds example executables and documentation. That consumes ~50% of build time (6 mins on my build VM).
My current build command is cmake --build ${proj}\build_${platform} --config RelWithDebInfo --target install --parallel.
How can I skip non essential targets?
It's third party dependency, so I don't want to modify CMakeLists.txt.
According to the source there is no option to control the build of samples
add_subdirectory(samples)
ref: https://github.com/apache/xerces-c/blob/master/CMakeLists.txt#L172
But you can patch it !
option(BUILD_SAMPLES "Build samples" ON)
message(STATUS "Build samples: ${BUILD_SAMPLES}")
...
if(BUILD_SAMPLES)
add_subdirectory(samples)
endif()
note: you can also enable CMP0077 if you have a super build...
Related
I've worked in c++ for many years but I am new to CMake.
I build my app with
cmake --build build_dir --config Debug --target all -- -j 1
This works fine and builds the Debug version.
If I change the --config to anything else, for example Release with the following command:
cmake --build build_dir --config Release --target all -- -j 1
ninja says "no work to do" and exits. Running the compiled app it is clearly not optimised. There are two other options in this Cmake Project, RelWithDebug and MinSizeRel, and they act the same.
In CmakeLists.txt there is:
set(CMAKE_CONFIGURATION_TYPES "Release;Debug;MinSizeRel;RelWithDebInfo")
There are a lot of other mentions of the various types, but it's all slightly greek to me at this time.
What can I do to work out where the issue is?
I'm going to give you the full in depth answer. Just in case someone else has this confusion.
What is a generator?
Essentially a generator is the build system CMake creates. CMake doesn't directly build your project. Because CMake is a meta-build system. IE CMake is a build system that 'generates' your true build system. That's why it is called a 'generator'.
"Visual Studio 16 2019" was designed to handles multiple configurations in 1 build.
Which is why when you open a Visual Studio project created by CMake you can easily change between Debug, Release, etc.
Where as "Unix Makefiles" and "Ninja" can only handle 1 config type at a time.
This difference in build system abilities leads to slightly different CLI when running CMake.
Visual Studio 16 2019 (Multi-Config)
As mentioned before Visual Studio supports multiple config types in the build.
cmake -S . -B build/vs -G "Visual Studio 16 2019"
cmake --build build/vs --config Debug
cmake --build build/vs --config Release
In the first command you are creating the Visual Studio 2019 project.
In the second command you are actually building the binaries. And since Visual Studio projects are multi-config you don't need a different build folder for each type. Since Visual Studio handles it for you!
Ninja (Single Config)
cmake -S . -B build/ninja/ -G "Ninja" -D CMAKE_BUILD_TYPE=Debug
cmake --build build/ninja/
# Now I've updated the project to make Release binaries
cmake -S . -B build/ninja/ -D CMAKE_BUILD_TYPE=Release
cmake --build build/ninja/
In the example above you create a Ninja project for a debug build. Then you build it.
Then you create a Ninja project for a release build. Then you build it.
Notice how you have to manually specify 2 build folders for each type yourself.
Ninja Multi-Config (new in CMake 3.17)
Thanks to advances in Ninja and CMake you can create avoid the hassle of specifying the build type at project creation time. So now it's just like Visual Studio.
So now you can create a "Ninja Multi-Config" project instead of just a "Ninja" project.
cmake -S . -B build/nin -G "Ninja Multi-Config"
cmake --build build/nin --config Debug
cmake --build build/nin --config Release
Further Elaboration
Single Config Vs Multi-Config
How can you tell if your generator is Multi vs Single?
Read the docs. Or if you need to in your scripts you can check it programmatically.
You can query the global property GENERATOR_IS_MULTI_CONFIG
CMAKE_BUILD_TYPE
A minor thing to mention is that CMAKE_BUILD_TYPE doesn't do anything on multi-config generators. So avoid using it in your CMake code.
See this answer: CMAKE_BUILD_TYPE is not being used in CMakeLists.txt
My project depends on mariadb-connector-c and I'm trying to automate the download, build and link process with cmake.
I currently download the project into a directory, I then try to execute generate ninja files and run them but I cannot run cmake at all:
execute_process(COMMAND "cmake -GNinja ." WORKING_DIRECTORY ${mariadb-connector-c_SOURCE_DIR})
I know this doesn't work because the next step, running ninja, fails:
execute_process(COMMAND "ninja" WORKING_DIRECTORY ${mariadb-connector-c_SOURCE_DIR})
cmake runs fine in CLI, I've tried using the full path to the cmake executable and replacing the dot with the variable with the full directory (which is also a valid variable, if you're wondering.)
How can I tell cmake to run cmake on that external project?
You can organize your project to a top-level CMakeLists.txt build your subprojects as ExternalProject.
This approach requires more work and maintenance of more CMake modules but it has its own benefits. I download Google Test as follows:
# Create download URL derived from version number.
set(GTEST_HOME https://github.com/google/googletest/archive)
set(GTEST_DOWNLOAD_URL ${GTEST_HOME}/release-${GTEST_VERSION}.tar.gz)
unset(GTEST_HOME)
# Download and build the Google Test library and add its properties to the third party arguments.
set(GTEST_ROOT ${THIRDPARTY_INSTALL_PATH}/gtest CACHE INTERNAL "")
ExternalProject_Add(gtest
URL ${GTEST_DOWNLOAD_URL}
CMAKE_ARGS -DBUILD_GTEST=ON -DBUILD_GMOCK=ON -DCMAKE_INSTALL_PREFIX=${GTEST_ROOT}
INSTALL_COMMAND make install
)
list(APPEND GLOBAL_THIRDPARTY_LIB_ARGS "-DGTEST_ROOT:PATH=${GTEST_ROOT}")
unset(GTEST_DOWNLOAD_URL)
unset(GTEST_ROOT)
The code abowe is inside my ExternalGoogleTest.cmake module which is included by CMakeLists.txt of third-party libraries:
set_directory_properties(PROPERTIES EP_BASE ${CMAKE_BINARY_DIR}/ThirdParty)
get_directory_property(THIRDPARTY_BASE_PATH EP_BASE)
set(THIRDPARTY_INSTALL_PATH ${THIRDPARTY_BASE_PATH}/Install)
set(GTEST_VERSION 1.8.0)
include(ExternalProject)
include(ExternalGoogleTest)
Your own project which depends on an external library will need a CMake module to build it as ExternalProject too. It can looks like:
ExternalProject_Add(my_project
DEPENDS gtest whatever
SOURCE_DIR ${CMAKE_SOURCE_DIR}/lib
CMAKE_ARGS
${GLOBAL_DEFAULT_ARGS}
${GLOBAL_THIRDPARTY_LIB_ARGS}
-DCMAKE_INSTALL_PREFIX=${DESIRED_INSTALL_PATH}/my_project
BUILD_COMMAND make
)
You can found more tips about this pattern here.
I would like to setup commands like make debug, make test, etc... What is the best way to do this with cmake, so that I run cmake .. one time (it takes a while) and then be able to choose the build type with make?
I couldn't find any resources on this.
As #Tsyvarev has commented this needs a little extra work for single-configuration environments (respectively CMake's Makefile generators) since the build type is chosen during CMake's configuration and finalized during the build environment generation step.
So here is what I've done:
First you run - e.g. in a script - CMake for all configurations you want to support and choose respective sub-folders for the output:
> cmake -H"." -B"Debug" -DCMAKE_BUILD_TYPE=Debug
> cmake -H"." -B"Release" -DCMAKE_BUILD_TYPE=Release
Note: -H (for "home directory") and -B (for "binary output directory") are undocumented options, but very useful in those cases. And they work with all CMake releases so far. Just be careful not to put spaces between the option and the their values.
Then you can use again CMake to build from/in those sub-folders:
> cmake --build "Debug"
> cmake --build "Release"
And if you want to run the tests you can:
> cmake --build "Debug" --target "test"
> cmake --build "Release" --target "test"
References
Changing CMake files standard location
Does CMake always generate configurations for all possible project configurations?
cmake build multiple targets in different build directories
CMAKE_BUILD_TYPE not being used in CMakeLists.txt
I would like to build a third-party project that already has CMake as part of my project's CMake strips. ExternalProject_Add is for this purpose, but I have found it can only be made to work with a specific generator, and I wanted it to work on many platforms easily.
For example, here is my external project with an added script for zlib, which has its own CMakeLists.txt:
set(USE_PROJECT_CMAKE_MODULE_PATH "-DCMAKE_MODULE_PATH=${MAKE_MODULE_PATH}")
ExternalProject_Add(ZLIB
SOURCE_DIR ${CMAKE_SOURCE_DIR}/external/zlib
DOWNLOAD_COMMAND ""
UPDATE_COMMAND ""
CMAKE_ARGS
-DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
-DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}
-DCMAKE_C_FLAGS=${CMAKE_C_FLAGS}
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
${USE_PROJECT_CMAKE_MODULE_PATH}
INSTALL_COMMAND "")
ExternalProject_Add_Step(ZLIB installInternally
COMMAND cd <BINARY_DIR> && make install
DEPENDEES install
ALWAYS 1)
ExternalProject_Get_Property(ZLIB install_dir)
if(UNIX)
set(ZLIB_NAME libz)
else(UNIX)
set(ZLIB_NAME zlib)
endif(UNIX)
add_library(zlib UNKNOWN IMPORTED)
set_property(TARGET zlib PROPERTY IMPORTED_LOCATION ${install_dir}/lib/${ZLIB_NAME}.a)
set(ZLIB_LIBRARIES zlib)
set(ZLIB_LIBRARIES_OPTIONAL ${ZLIB_LIBRARIES})
set(ZLIB_DIR ${install_dir} CACHE INTERNAL "zlib ROOT dir")
set(ZLIB_INCLUDE_DIRS ${install_dir}/include CACHE INTERNAL "zlib include dirs")
set(ZLIB_DEFINES "-msse2 -mfpmath=sse" CACHE INTERNAL "zlib defines")
The problem with this is that it works with make, but not with Xcode or Visual Studio. Perhaps there is some way to take the CMake build commands passed to my project and forward them to ExternalProject_Add.
How can I write ExternalProject_Add calls in a cross-platform way with minimal code complexity, or is there a better alternative?
Problems
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
This is enough for single-configuration projects. But for Xcode and Visual Studio, you need to set CMAKE_CONFIGURATION_TYPES plus call build . --config at the build stage. See my answer.
COMMAND cd <BINARY_DIR> && make install
This will work only for Makefile generators of course. To be cross-platform you can use:
--build . --target install --config inside INSTALL_COMMAND of ExternalProject_Add.
Take a look at this template file, and in particular the following lines:
ExternalProject_Add(
"${current_project}"
URL
#HUNTER_PACKAGE_URL#
URL_HASH
SHA1=#HUNTER_PACKAGE_SHA1#
DOWNLOAD_DIR
"#HUNTER_PACKAGE_DOWNLOAD_DIR#"
SOURCE_DIR
"#HUNTER_PACKAGE_SOURCE_DIR#"
INSTALL_DIR
"#HUNTER_PACKAGE_INSTALL_PREFIX#"
# Not used, just avoid creating Install/<name> empty directory
BUILD_COMMAND ""
# This command is empty because all necessary targets will
# be built on install stage
CMAKE_ARGS
"-G#CMAKE_GENERATOR#"
"-C#HUNTER_CACHE_FILE#"
"-C#HUNTER_ARGS_FILE#"
"-D${postfix_name}=${${postfix_name}}"
"-DCMAKE_BUILD_TYPE=${configuration}"
"-DCMAKE_CONFIGURATION_TYPES=${configuration}"
"-DCMAKE_INSTALL_PREFIX=#HUNTER_PACKAGE_INSTALL_PREFIX#"
"-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}"
INSTALL_COMMAND
"#CMAKE_COMMAND#"
--build .
--target install
--config ${configuration}
--
${jobs_option}
)
Alternative
or is there a better alternative?
Have you seen Hunter?
You can add zlib just like this:
hunter_add_package(ZLIB)
find_package(ZLIB CONFIG REQUIRED)
target_link_libraries(... ZLIB::zlib)
This code works everywhere. Third party dependencies will be downloaded automatically in the configuration step. Example of building with different generator/toolchains (build.py is just a CMake wrapper that sets CMAKE_TOOLCHAIN_FILE and -G/-B):
build.py --toolchain mingw --config Release # MinGW Makefiles
build.py --toolchain vs-12-2013 --config Debug # Visual Studio 12 2013
build.py --toolchain xcode --config Release # Xcode
build.py --toolchain libcxx --config Release # Makefile with -stdlib=libc++ toolchain
build.py --toolchain ios-8-2 --config Release # Xcode with iOS SDK 8.2 toolchain
You got full control what options, build types or number of jobs you want to have while building third-party packages. For instance, this is how you can build four types, Debug, Release, MinSizeRel, and RelWithDebInfo for zlib and link MinSizeRel to the current project:
> build.py --toolchain xcode --verbose --config MinSizeRel --fwd "HUNTER_CONFIGURATION_TYPES=Release;Debug;MinSizeRel;RelWithDebInfo"
/.../clang /.../lib/libz-MinSizeRel.a ... -o /.../_builds/xcode/MinSizeRel/foo
> ls -la /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz*
99056 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-MinSizeRel.a
307872 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-RelWithDebInfo.a
109536 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz.a
258904 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libzd.a
CMake ExternalProject_Add calls work cross-platform by default and will only fail to do so if one uses particular commands that are only available on a subset of operating systems.
Typically, CMAKE_ARGS is used to pass information to each superbuild unit within an external project build. The CMakeLists.txt files that control each miniature part of the overall build use CMake's declarative syntax (e.g., "add_library(library_name SHARED filename1.hpp filename1.cpp). CMake will convert such syntax to the commands that are specific to the particular build system you wish to use (e.g., make and Ninja).
The sample above re: zlib fails to be cross-platform in part because the ExternalProject_Add_Step contains "COMMAND cd && make install", which necessarily only works in situations where invoking "cd" is actually the correct way to change directories, and where invoking "make" is actually the correct way to build software.
CMake's -E option provides a way to invoke basic operations like changing/copying/making/removing directories without making such assumptions.
(By the way, if you're using IDEs such as Visual Studio or Xcode, you'll likely want to invoke one or more IDE generators when using CMake. For instance, setting
-G "Eclipse CDT4 - Unix Makefiles" -DCMAKE_ECLIPSE_GENERATE_SOURCE_PROJECT=TRUE
will cause Eclipse projects to be generated in each build area, and also in the source code area that is shared for all builds. Of course, if you are using Xcode or Visual Studio, you'll have to substitute the appropriate flag for those IDEs. Alternatively, you could consider using Eclipse with Ninja on all platforms, though at the time of writing, I am not completely certain that Ninja is ready for prime-time on non-Linux, non-Windows operating systems.)
Is it somehow possible to be able to have a parallel build no matter which build tool is used?
Under Unix we can add make -jN where N are the number of threads, and under Windows I added to the CXX_FLAG "/MP" which is then used in Visual Studio to parallel build...(?) How can I make my version such that CMAKE_MAKE_PROGRAM is not always extended when I run CMake?
What is a general solution?
I came up with this:
# Add some multithreaded build support
MARK_AS_ADVANCED(MULTITHREADED_BUILD)
set(MULTITHREADED_BUILD 12 CACHE STRING "How many threads are used to build the project")
if(MULTITHREADED_BUILD)
if(${CMAKE_GENERATOR} MATCHES "Unix Makefiles")
message(STATUS ${CMAKE_BUILD_TOOL})
set(CMAKE_MAKE_PROGRAM "${CMAKE_MAKE_PROGRAM} -j${MULTITHREADED_BUILD}")
message(STATUS "Added arguments to CMAKE_BUILD_TOOL: ${CMAKE_MAKE_PROGRAM}")
elseif(MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /MP")
message(STATUS "Added parallel build arguments to CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
endif()
endif()
With CMake 3.12 this is possible. From the release notes:
The cmake(1) Build a Project (cmake --build) gained --parallel [<jobs>] and -j [<jobs>] options to specify a parallel build level. They map to corresponding options of the native build tool.
As mentioned by dkg, you can also set the environment variable CMAKE_BUILD_PARALLEL_LEVEL.
Links to CMake's documentation:
Build a Project
CMAKE_BUILD_PARALLEL_LEVEL
If you have CMake v2.8.8 or higher, you may use Ninja as an alternative of GNU make:
mkdir build
cd build
cmake -G Ninja ..
ninja # Parallel build (no need -j12)
or
mkdir build
cd build
cmake -G Ninja ..
cmake --build . # Parallel build using Ninja
As you can see, no need to use CMAKE_MAKE_PROGRAM, the build is run in parallel by default, optimizing the number of jobs depending on available CPU cores.
Ninja is based on a low-level JSON configuration to speed up the startup phase. Therefore its JSON configuration is not easy to write by hand, and I always generate it using a high-level tool/IDE:
CMake v2.8.8 (2012)
Qt Creator v2.6 (2012)
KDevelop v4.6 (2013)
Meson on Linux (2013)
... see the generators of Ninja configuration at https://github.com/ninja-build/ninja/wiki/List-of-generators-producing-ninja-build-files
As a C++ build often requires lots of memory, your computer must provide as much memory as the number of CPU cores.
As pointed out by Ruslan, CMake 3.12 (2018) has a new option cmake --build -j <N> to limit build to <N> cores (jobs) thus limiting the memory consumption (see also the documentation). If you use an older CMake version, you can still use cmake --build -- -j <N>. The option -- tells to CMake to pass the rest directly to the underlying builder tool, here it is Ninja.
Now it is very simple to parallel build with cmake. You can add "-j jobs_number" when using "cmake --build". For example:
cmake --build . -j 24
More details can be found in CMAKE manual: https://cmake.org/cmake/help/latest/manual/cmake.1.html#build-tool-mode
--parallel [], -j [] The maximum number of concurrent processes to use when building. If is omitted the native build
tool’s default number is used.
The CMAKE_BUILD_PARALLEL_LEVEL environment variable, if set, specifies
a default parallel level when this option is not given.
Some native build tools always build in parallel. The use of
value of 1 can be used to limit to a single job.
As already mention above one can use --parallel [<jobs>] (or -j [<jobs>]) option of CMake to build solution in parallel.
But I would like to note that in Windows a command
cmake.exe --build --parallel <jobs>
starts a root MSBuild process with the following arguments:
msbuild.exe /m:<jobs> /p:CL_MPCount=1 <another-useful-arguments...>
This means the root MSBuild process will start multiple child MSBuild processes due to /m:<jobs> (or /maxcpucount:<jobs>). But each of this child MSBuild runs at most one compiler process due to /p:CL_MPCount=1 argument (for more details find this link). In other words each MSBuild can compile at most one source file at the same time.
To overcome this limitation one may call CMake in the following way:
cmake.exe --build --parallel <n_msbuild> -- /p:CL_MPcount=<n_cl>
This approach sets /MP option thereby allows multiple compiler processes under each MSBuild instance.
And for the record. To configure builds ran form Visual Studio GUI one need to go to Settings:
Option "Projects and Solutions → Build And Run → maximum number of parallel projects build" affects on maxcpucount.
Option "Projects and Solutions → VC++ Project Settings → Build → Maximum Concurrent C++ Compilations" represents the default boundary to /MP.
You can't do this cross-platform. The -jN option is a parameter to make, and not part of the generated Makefile. However, you could have CMake generate a Bash script that runs make for your project using -jN (where the script looks up the number of cores you have).
I have settled down to writing a parallelmake.sh script for Unix Makefiles-based generators. This is done here: https://github.com/gabyx/ApproxMVBB
And the relevant parts in the the CMake file:
https://github.com/gabyx/ApproxMVBB/blob/master/CMakeLists.txt#L89
# Add some multithreaded build support =====================================================================================================
MARK_AS_ADVANCED(MULTITHREADED_BUILD)
SET(MULTITHREADED_BUILD ON CACHE BOOL "Parallel build with as many threads as possible!")
if(MULTITHREADED_BUILD)
if(${CMAKE_GENERATOR} MATCHES "Unix Makefiles")
file(COPY ${ApproxMVBB_ROOT_DIR}/cmake/parallelmake.sh DESTINATION ${PROJECT_BINARY_DIR}
FILE_PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE
NO_SOURCE_PERMISSIONS
)
SET(CMAKE_MAKE_PROGRAM "${PROJECT_BINARY_DIR}/parallelmake.sh")
MESSAGE(STATUS "Set make program to ${PROJECT_BINARY_DIR}/parallelmake.sh")
elseif(MSVC)
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}" "/MP")
MESSAGE(STATUS "Added parallel build arguments to CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
endif()
endif()
# ========================================================================================================================================