Invoking make install on a subdirectory before make all - cmake

I am working on migrating a C++ project whose configuration currently is a combination of cmake (CMakeLists.txt) and Make (GNUmakefile), to using only cmake. The way it's currently set is:
myproject
| CMakeLists.txt (add_subdirectory(src))
| src/
| apps/
| myapp/
| [myapp source files]
| GNUmakefile
| ...
| libs/
| mylib/
| [mylib source files]
| GNUmakefile
| ...
| common/
| CMakeLists.txt (add_subdirectory(dir1, dir2, dir3...))
| ...
| include/
There are more apps and libs outside of myapp and mylib, but they all build and link in similar ways, so if I have one, I can get all.
The way the project builds right now, there is a script that essentially has 4 lines:
cmake path/to/myproject // with the necessary flags and variables passed
make all DESTDIR=$INSTALL_DIR install
cd src/libs ; make all
cd src/apps ; make all
This works because common, which has no dependencies, builds and installs first (include also gets installed here), then libs - which depends on common - builds, and then apps - which depends on both libs and common - builds.
My problem is, if I replace the GNUmakefiles in libs and apps, I don't know how to change the commands I run in the build script so that cmake and make are interwoven like so:
cmake configures common
make compiles & installs all the cmake targets from common (let's say those are libcommon.so and libcommonutils.so)
cmake configures libs
make compiles & installs all the cmake targets from libs (which uses the libraries from common to produce mylib.a)
cmake configures apps
make compiles & installs all the cmake targets from apps (which uses mylib.a and the common shared libraries)
Installing include will also have to happen at some point, early on, since common depends on the header files inside it.
An alternative solution might also work, of course, but this (interweaving cmake and make calls) is the only one I could think of.

You can use ExternalProject_add for common, libs and apps.
src/CMakeLists.txt:
ExternalProject_add(common
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/common
BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/common
# Forward CMAKE_INSTALL_PREFIX to external project.
CMAKE_CACHE_ARGS -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_INSTALL_PREFIX}
)
ExternalProject_add(libs
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libs
BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/libs
# Disable *install* step
INSTALL_COMMAND ""
)
ExternalProject_add(apps
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/apps
BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/apps
# Disable *install* step
INSTALL_COMMAND ""
)
# Adjust dependencies between components
add_dependencies(libs common)
add_dependencies(apps libs common)
Usage of this script(out-of-source build; in-source build is also supported)
cmake -DCMAKE_INSTALL_PREFIX=<install-dir> <source-dir>
make
is equvalent to the following sequence of commands:
mkdir src/common && cd src/common \
&& cmake -DCMAKE_INSTALL_PREFIX=<install-dir> <source-dir>/src/common && make install
mkdir src/libs && cd src/libs && cmake <source-dir>/src/libs && make all
mkdir src/apps && cd src/apps && cmake <source-dir>/src/apps && make all
Processing of include is not implemented, but you may do that in similar way.

Related

cmake add_custom_command introduces false dependency unless add_custom_target also used

I'm using cmake to build some libraries, all of which generate some of their files.
I've created the generated files using add_custom_command(), but I've discovered something which seems like a false dependency. If a downstream library has a generated file and links to an upstream library, the downstream library sources will not start to compile until the upstream library is completely built. With many libraries (more than 50) in my project, this false dependency causes serialization in the build.
What's curious is that I also noticed that if an explicit add_custom_target() for the generated file is used with add_dependencies(), the false dependency no longer exists, and the files in the downstream library will compile concurrently with the ones in the upstream library. So I have a workaround, but is this expected behavior?
Using Cmake 1.19, Ninja 1.10.2.
The following is a minimal CMakeLists.txt file that shows what happens. The WORKS option conditionally adds the add_custom_target() and add_dependencies() clauses which cause it to work (quickly). The files foo.c and bar.c are empty, and I'm building in a subdirectory of the CMakeLists.txt directory.
I put a file called /tmp/x which is a wrapper around cc that sleeps to show the serialization occur:
#!/bin/bash -e
echo "mycc" $(date)
sleep 4
exec /usr/bin/cc $*
Here is the CMakeLists.txt:
cmake_minimum_required(VERSION 3.19)
project(test)
option(WORKS "false dependency gone if WORKS set to ON" OFF)
add_library(foo
foo.c
)
add_library(bar
bar.c
bargen.h
)
target_include_directories(bar PRIVATE ${CMAKE_CURRENT_BINARY_DIR})
add_custom_command(OUTPUT bargen.h
COMMAND touch bargen.h
)
if (${WORKS})
add_custom_target(generate_file
DEPENDS bargen.h
)
add_dependencies(bar generate_file)
endif()
target_link_libraries(bar PUBLIC foo)
Build it like this and you will see the serialization: bar.c will not start to compile until after foo.c has finished compiling (in fact, not until after the foo library is built). (I'm explicitly choosing '-j 4' to ensure Ninja will try to build in parallel).
cmake -DWORKS=OFF -G Ninja -DCMAKE_C_COMPILER=/tmp/x .. && ninja clean && ninja -j 4 -v | grep mycc
This shows the following output:
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/rhb/Downloads/cmake-anomaly/build
[1/1] Cleaning all built files...
Cleaning... 5 files.
mycc Sat Jan 2 15:15:25 EST 2021
mycc Sat Jan 2 15:15:29 EST 2021
Now build it like this and you'll see that bar.c compiles concurrently with foo.c:
cmake -DWORKS=ON -G Ninja -DCMAKE_C_COMPILER=/tmp/x .. && ninja clean && ninja -j 4 -v | grep mycc
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/rhb/Downloads/cmake-anomaly/build
[1/1] Cleaning all built files...
Cleaning... 5 files.
mycc Sat Jan 2 15:15:37 EST 2021
mycc Sat Jan 2 15:15:37 EST 2021
This was reported as a cmake issue and recently fixed with a new argument to add_custom_command. This fix will be released with the next cmake version (3.27).
DEPENDS_EXPLICIT_ONLY
.. versionadded:: 3.27
Indicate that the command's DEPENDS argument represents all files
required by the command and implicit dependencies are not required.
Without this option, if any target uses the output of the custom command,
CMake will consider that target's dependencies as implicit dependencies for
the custom command in case this custom command requires files implicitly
created by those targets.
Only the Ninja Generators actually use this information to remove
unnecessary implicit dependencies.

ExternalProject_Add autogen project prevent configure on rebuild

I have a CMake project on linux and I'm using ExternalProject to build Google Protobuf. It works great, however any subsequent builds still call the configure step in the external project (which is annoying because protobuf is an autogen project with a rather long step). I used the UPDATE_DISCONNECTED argument so it wouldn't re-clone which helps some, but you'd think if it didn't re-clone, it wouldn't need to re-configure or re-build/install. How can I get CMake to just build it the one time and skip subsequent builds (i.e. my next make from the build directory)?
Here's my CMakeLists.txt
cmake_minimum_required(VERSION 2.8.11)
project(pbuf_test)
include(${PROJECT_SOURCE_DIR}/cmake/protogen.cmake)
set(PBUF_DIR ${PROJECT_BINARY_DIR}/protobuf)
include(ExternalProject)
ExternalProject_Add(protobuf
PREFIX ${PROTOBUF_DIR}
GIT_REPOSITORY https://github.com/google/protobuf.git
GIT_TAG v3.4.1
UPDATE_DISCONNECTED 1
BUILD_IN_SOURCE 1
CONFIGURE_COMMAND ./autogen.sh COMMAND ./configure --prefix=${PBUF_DIR}
)
set(PBUF_INCLUDE_DIR ${PBUF_DIR}/include)
set(PBUF_LIBRARY ${PBUF_DIR}/lib/libprotobuf.so)
set(PBUF_PROTOC ${PBUF_DIR}/bin/protoc)
file(GLOB PBUF_FILES "${PROJECT_SOURCE_DIR}/msg/*.proto")
custom_protobuf_generate_cpp(PBUF_SRCS PBUF_HDRS ${PBUF_FILES})
include_directories(
${PROJECT_BINARY_DIR}
${PBUF_INCLUDE_DIR}
)
add_executable(${PROJECT_NAME} main.cpp ${PBUF_SRCS} ${PBUF_HDRS})
add_dependencies(${PROJECT_NAME} protobuf)
target_link_libraries(${PROJECT_NAME}
${PBUF_LIBRARY}
)
Full example project here
The only way I know is to move the steps to build the external project into a script called by add_command. And then add an dependency between your project and the external project's output/lib/binary.
Pro: Avoids unnecessary rebuilds
Pro: Avoids unexpected updates of the external lib
Con: Does not rebuild in case the external repository was updated (and this is required)
Con: Additional work to write such a script
BTW: Within my project, we do it the hard way. But we do not want any unexpected updates of external libs.
It seems the solution to this for now that still uses ExternalProject is to move the configure step into the download step.
Something like this. Note the name of the target is the directory it expects the download to create. This might only work on certain operating systems because of the use of the && joiner: I'm a little new to cmake.
ExternalProject_Add(protobuf
PREFIX ${PROTOBUF_DIR}
DOWNLOAD_COMMAND git clone https://github.com/google/protobuf.git protobuf &&
cd protobuf &&
git checkout v3.4.1 &&
./autogen.sh &&
./configure --prefix=${PBUF_DIR}
UPDATE_COMMAND git pull
UPDATE_DISCONNECTED 1
BUILD_IN_SOURCE 1
CONFIGURE_COMMAND ""
)

How can I run cmake from within cmake?

My project depends on mariadb-connector-c and I'm trying to automate the download, build and link process with cmake.
I currently download the project into a directory, I then try to execute generate ninja files and run them but I cannot run cmake at all:
execute_process(COMMAND "cmake -GNinja ." WORKING_DIRECTORY ${mariadb-connector-c_SOURCE_DIR})
I know this doesn't work because the next step, running ninja, fails:
execute_process(COMMAND "ninja" WORKING_DIRECTORY ${mariadb-connector-c_SOURCE_DIR})
cmake runs fine in CLI, I've tried using the full path to the cmake executable and replacing the dot with the variable with the full directory (which is also a valid variable, if you're wondering.)
How can I tell cmake to run cmake on that external project?
You can organize your project to a top-level CMakeLists.txt build your subprojects as ExternalProject.
This approach requires more work and maintenance of more CMake modules but it has its own benefits. I download Google Test as follows:
# Create download URL derived from version number.
set(GTEST_HOME https://github.com/google/googletest/archive)
set(GTEST_DOWNLOAD_URL ${GTEST_HOME}/release-${GTEST_VERSION}.tar.gz)
unset(GTEST_HOME)
# Download and build the Google Test library and add its properties to the third party arguments.
set(GTEST_ROOT ${THIRDPARTY_INSTALL_PATH}/gtest CACHE INTERNAL "")
ExternalProject_Add(gtest
URL ${GTEST_DOWNLOAD_URL}
CMAKE_ARGS -DBUILD_GTEST=ON -DBUILD_GMOCK=ON -DCMAKE_INSTALL_PREFIX=${GTEST_ROOT}
INSTALL_COMMAND make install
)
list(APPEND GLOBAL_THIRDPARTY_LIB_ARGS "-DGTEST_ROOT:PATH=${GTEST_ROOT}")
unset(GTEST_DOWNLOAD_URL)
unset(GTEST_ROOT)
The code abowe is inside my ExternalGoogleTest.cmake module which is included by CMakeLists.txt of third-party libraries:
set_directory_properties(PROPERTIES EP_BASE ${CMAKE_BINARY_DIR}/ThirdParty)
get_directory_property(THIRDPARTY_BASE_PATH EP_BASE)
set(THIRDPARTY_INSTALL_PATH ${THIRDPARTY_BASE_PATH}/Install)
set(GTEST_VERSION 1.8.0)
include(ExternalProject)
include(ExternalGoogleTest)
Your own project which depends on an external library will need a CMake module to build it as ExternalProject too. It can looks like:
ExternalProject_Add(my_project
DEPENDS gtest whatever
SOURCE_DIR ${CMAKE_SOURCE_DIR}/lib
CMAKE_ARGS
${GLOBAL_DEFAULT_ARGS}
${GLOBAL_THIRDPARTY_LIB_ARGS}
-DCMAKE_INSTALL_PREFIX=${DESIRED_INSTALL_PATH}/my_project
BUILD_COMMAND make
)
You can found more tips about this pattern here.

How can I distribute/install 3rd-party libraries with CMake?

I'm building and distributing an executable which relies on several 3rd-party libraries, most of which are built outside of CMake's buildsystem (though, if it helps, I can add and build them as custom targets in my CMake file).
Here's how I include the libs:
target_link_libraries(MyApp
Lib1_x64
Lib2_x64
etc.}
I'd like to include a CMake directive which installs my executable along with its dependencies. Is there a better way to do this other than calling the install command for every single dependency?
install(DIRECTORY ${DEV_PATH}/Release/ DESTINATION lib FILES_MATCHING PATTERN "libLib1*" PATTERN "*.svn" EXCLUDE PATTERN "*-obj" EXCLUDE)
Is there maybe a way to do this via the add_library command?
This is possible in nowadays with CMake 3.21, which allows you to get a set of dependent libraries via parameter RUNTIME_DEPENDENCY_SET in the install command. Keep in mind, that it will gather all dependent libraries including system, you will need to filter them.
cmake_minimum_required(VERSION 3.21)
# Install your app or library
install(
TARGETS ${PROJECT_NAME}
RUNTIME_DEPENDENCY_SET runtime_deps
)
# Install third-party libraries (exclude system libraries Windows/Unix)
LIST(APPEND pre_exclude_regexes "api-ms-.*")
LIST(APPEND pre_exclude_regexes "ext-ms-.*")
LIST(APPEND post_exclude_regexes ".*WINDOWS[\\/]system32.*")
LIST(APPEND post_exclude_regexes "^/lib" "^/usr" "^/bin")
install(RUNTIME_DEPENDENCY_SET runtime_deps
PRE_EXCLUDE_REGEXES ${pre_exclude_regexes}
POST_EXCLUDE_REGEXES ${post_exclude_regexes}
)
Sorry, no. The add_library command serves the purpose of creating a target in your makefile to build a library. target_link_dependencies just passes the various -L ... -l ... flags to the compiler for a target.
The normal CMake approach is to test for the presence of other libraries or executables on which yours depends, and fail if they are not available.
Handling dependencies is normally left to a package manage such as apt, rpm, etc. It is unwise to build your dependent components as part of your own makefile for a number of reasons:
Your clients might want to use a slightly different version
Any change in how those dependencies build requires you to modify your own build scripts
Your program may work with multiple versions of a dependency, all of which build in different ways.
As someone who uses CMake to build large parts of an entire embedded Linux distribution (see Open webOS), I advise you to keep your scripts focused very tightly on building just one thing.
I am also finding a way to deal this problem today.
and I found :
include(ExternalProject)
ExternalProject_Add()
does it!
here is my sample CMakeLists.txt:
cmake_minimum_required(VERSION 3.13)
project(third-party-build)
include(ExternalProject)
ExternalProject_Add(libuuid
PREFIX libuuid-prefix
# DOWNLOAD_DIR libuuid-download
# SOURCE_DIR libuuid-source
# DOWNLOAD_COMMAND wget xxx.com/xxx.tar.gz && tar zxf xxx.tar.gz
DOWNLOAD_COMMAND tar zxf ${PROJECT_SOURCE_DIR}/libuuid-1.0.3.tar.gz
CONFIGURE_COMMAND <DOWNLOAD_DIR>/libuuid-1.0.3/configure
BUILD_COMMAND ${MAKE}
BUILD_IN_SOURCE 1
INSTALL_COMMAND mkdir -p ${PROJECT_SOURCE_DIR}/lib/ && cp <SOURCE_DIR>/.libs/libuuid.a <SOURCE_DIR>/.libs/libuuid.so <SOURCE_DIR>/.libs/libuuid.so.1 <SOURCE_DIR>/.libs/libuuid.so.1.0.0 ${PROJECT_SOURCE_DIR}/lib/
LOG_DOWNLOAD 1
LOG_BUILD 1
# LOG_INSTALL 1
)

How do I use CMake ExternalProject_Add or alternatives in a cross-platform way?

I would like to build a third-party project that already has CMake as part of my project's CMake strips. ExternalProject_Add is for this purpose, but I have found it can only be made to work with a specific generator, and I wanted it to work on many platforms easily.
For example, here is my external project with an added script for zlib, which has its own CMakeLists.txt:
set(USE_PROJECT_CMAKE_MODULE_PATH "-DCMAKE_MODULE_PATH=${MAKE_MODULE_PATH}")
ExternalProject_Add(ZLIB
SOURCE_DIR ${CMAKE_SOURCE_DIR}/external/zlib
DOWNLOAD_COMMAND ""
UPDATE_COMMAND ""
CMAKE_ARGS
-DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
-DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}
-DCMAKE_C_FLAGS=${CMAKE_C_FLAGS}
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
${USE_PROJECT_CMAKE_MODULE_PATH}
INSTALL_COMMAND "")
ExternalProject_Add_Step(ZLIB installInternally
COMMAND cd <BINARY_DIR> && make install
DEPENDEES install
ALWAYS 1)
ExternalProject_Get_Property(ZLIB install_dir)
if(UNIX)
set(ZLIB_NAME libz)
else(UNIX)
set(ZLIB_NAME zlib)
endif(UNIX)
add_library(zlib UNKNOWN IMPORTED)
set_property(TARGET zlib PROPERTY IMPORTED_LOCATION ${install_dir}/lib/${ZLIB_NAME}.a)
set(ZLIB_LIBRARIES zlib)
set(ZLIB_LIBRARIES_OPTIONAL ${ZLIB_LIBRARIES})
set(ZLIB_DIR ${install_dir} CACHE INTERNAL "zlib ROOT dir")
set(ZLIB_INCLUDE_DIRS ${install_dir}/include CACHE INTERNAL "zlib include dirs")
set(ZLIB_DEFINES "-msse2 -mfpmath=sse" CACHE INTERNAL "zlib defines")
The problem with this is that it works with make, but not with Xcode or Visual Studio. Perhaps there is some way to take the CMake build commands passed to my project and forward them to ExternalProject_Add.
How can I write ExternalProject_Add calls in a cross-platform way with minimal code complexity, or is there a better alternative?
Problems
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
This is enough for single-configuration projects. But for Xcode and Visual Studio, you need to set CMAKE_CONFIGURATION_TYPES plus call build . --config at the build stage. See my answer.
COMMAND cd <BINARY_DIR> && make install
This will work only for Makefile generators of course. To be cross-platform you can use:
--build . --target install --config inside INSTALL_COMMAND of ExternalProject_Add.
Take a look at this template file, and in particular the following lines:
ExternalProject_Add(
"${current_project}"
URL
#HUNTER_PACKAGE_URL#
URL_HASH
SHA1=#HUNTER_PACKAGE_SHA1#
DOWNLOAD_DIR
"#HUNTER_PACKAGE_DOWNLOAD_DIR#"
SOURCE_DIR
"#HUNTER_PACKAGE_SOURCE_DIR#"
INSTALL_DIR
"#HUNTER_PACKAGE_INSTALL_PREFIX#"
# Not used, just avoid creating Install/<name> empty directory
BUILD_COMMAND ""
# This command is empty because all necessary targets will
# be built on install stage
CMAKE_ARGS
"-G#CMAKE_GENERATOR#"
"-C#HUNTER_CACHE_FILE#"
"-C#HUNTER_ARGS_FILE#"
"-D${postfix_name}=${${postfix_name}}"
"-DCMAKE_BUILD_TYPE=${configuration}"
"-DCMAKE_CONFIGURATION_TYPES=${configuration}"
"-DCMAKE_INSTALL_PREFIX=#HUNTER_PACKAGE_INSTALL_PREFIX#"
"-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}"
INSTALL_COMMAND
"#CMAKE_COMMAND#"
--build .
--target install
--config ${configuration}
--
${jobs_option}
)
Alternative
or is there a better alternative?
Have you seen Hunter?
You can add zlib just like this:
hunter_add_package(ZLIB)
find_package(ZLIB CONFIG REQUIRED)
target_link_libraries(... ZLIB::zlib)
This code works everywhere. Third party dependencies will be downloaded automatically in the configuration step. Example of building with different generator/toolchains (build.py is just a CMake wrapper that sets CMAKE_TOOLCHAIN_FILE and -G/-B):
build.py --toolchain mingw --config Release # MinGW Makefiles
build.py --toolchain vs-12-2013 --config Debug # Visual Studio 12 2013
build.py --toolchain xcode --config Release # Xcode
build.py --toolchain libcxx --config Release # Makefile with -stdlib=libc++ toolchain
build.py --toolchain ios-8-2 --config Release # Xcode with iOS SDK 8.2 toolchain
You got full control what options, build types or number of jobs you want to have while building third-party packages. For instance, this is how you can build four types, Debug, Release, MinSizeRel, and RelWithDebInfo for zlib and link MinSizeRel to the current project:
> build.py --toolchain xcode --verbose --config MinSizeRel --fwd "HUNTER_CONFIGURATION_TYPES=Release;Debug;MinSizeRel;RelWithDebInfo"
/.../clang /.../lib/libz-MinSizeRel.a ... -o /.../_builds/xcode/MinSizeRel/foo
> ls -la /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz*
99056 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-MinSizeRel.a
307872 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-RelWithDebInfo.a
109536 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz.a
258904 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libzd.a
CMake ExternalProject_Add calls work cross-platform by default and will only fail to do so if one uses particular commands that are only available on a subset of operating systems.
Typically, CMAKE_ARGS is used to pass information to each superbuild unit within an external project build. The CMakeLists.txt files that control each miniature part of the overall build use CMake's declarative syntax (e.g., "add_library(library_name SHARED filename1.hpp filename1.cpp). CMake will convert such syntax to the commands that are specific to the particular build system you wish to use (e.g., make and Ninja).
The sample above re: zlib fails to be cross-platform in part because the ExternalProject_Add_Step contains "COMMAND cd && make install", which necessarily only works in situations where invoking "cd" is actually the correct way to change directories, and where invoking "make" is actually the correct way to build software.
CMake's -E option provides a way to invoke basic operations like changing/copying/making/removing directories without making such assumptions.
(By the way, if you're using IDEs such as Visual Studio or Xcode, you'll likely want to invoke one or more IDE generators when using CMake. For instance, setting
-G "Eclipse CDT4 - Unix Makefiles" -DCMAKE_ECLIPSE_GENERATE_SOURCE_PROJECT=TRUE
will cause Eclipse projects to be generated in each build area, and also in the source code area that is shared for all builds. Of course, if you are using Xcode or Visual Studio, you'll have to substitute the appropriate flag for those IDEs. Alternatively, you could consider using Eclipse with Ninja on all platforms, though at the time of writing, I am not completely certain that Ninja is ready for prime-time on non-Linux, non-Windows operating systems.)