ExternalProject_Add autogen project prevent configure on rebuild - cmake

I have a CMake project on linux and I'm using ExternalProject to build Google Protobuf. It works great, however any subsequent builds still call the configure step in the external project (which is annoying because protobuf is an autogen project with a rather long step). I used the UPDATE_DISCONNECTED argument so it wouldn't re-clone which helps some, but you'd think if it didn't re-clone, it wouldn't need to re-configure or re-build/install. How can I get CMake to just build it the one time and skip subsequent builds (i.e. my next make from the build directory)?
Here's my CMakeLists.txt
cmake_minimum_required(VERSION 2.8.11)
project(pbuf_test)
include(${PROJECT_SOURCE_DIR}/cmake/protogen.cmake)
set(PBUF_DIR ${PROJECT_BINARY_DIR}/protobuf)
include(ExternalProject)
ExternalProject_Add(protobuf
PREFIX ${PROTOBUF_DIR}
GIT_REPOSITORY https://github.com/google/protobuf.git
GIT_TAG v3.4.1
UPDATE_DISCONNECTED 1
BUILD_IN_SOURCE 1
CONFIGURE_COMMAND ./autogen.sh COMMAND ./configure --prefix=${PBUF_DIR}
)
set(PBUF_INCLUDE_DIR ${PBUF_DIR}/include)
set(PBUF_LIBRARY ${PBUF_DIR}/lib/libprotobuf.so)
set(PBUF_PROTOC ${PBUF_DIR}/bin/protoc)
file(GLOB PBUF_FILES "${PROJECT_SOURCE_DIR}/msg/*.proto")
custom_protobuf_generate_cpp(PBUF_SRCS PBUF_HDRS ${PBUF_FILES})
include_directories(
${PROJECT_BINARY_DIR}
${PBUF_INCLUDE_DIR}
)
add_executable(${PROJECT_NAME} main.cpp ${PBUF_SRCS} ${PBUF_HDRS})
add_dependencies(${PROJECT_NAME} protobuf)
target_link_libraries(${PROJECT_NAME}
${PBUF_LIBRARY}
)
Full example project here

The only way I know is to move the steps to build the external project into a script called by add_command. And then add an dependency between your project and the external project's output/lib/binary.
Pro: Avoids unnecessary rebuilds
Pro: Avoids unexpected updates of the external lib
Con: Does not rebuild in case the external repository was updated (and this is required)
Con: Additional work to write such a script
BTW: Within my project, we do it the hard way. But we do not want any unexpected updates of external libs.

It seems the solution to this for now that still uses ExternalProject is to move the configure step into the download step.
Something like this. Note the name of the target is the directory it expects the download to create. This might only work on certain operating systems because of the use of the && joiner: I'm a little new to cmake.
ExternalProject_Add(protobuf
PREFIX ${PROTOBUF_DIR}
DOWNLOAD_COMMAND git clone https://github.com/google/protobuf.git protobuf &&
cd protobuf &&
git checkout v3.4.1 &&
./autogen.sh &&
./configure --prefix=${PBUF_DIR}
UPDATE_COMMAND git pull
UPDATE_DISCONNECTED 1
BUILD_IN_SOURCE 1
CONFIGURE_COMMAND ""
)

Related

Cmake run a ExternalProject if find_package fails

I am writing an application that requires ncurses, and I have it setup to build ncurses (among others) as external projects:
ExternalProject_Add(ncurses-extern
INSTALL_DIR ${CMAKE_CURRENT_SOURCE_DIR}/ncurses
GIT_REPOSITORY https://github.com/mirror/ncurses.git
CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix <INSTALL_DIR> --with-normal --without-debug --without-manpages
BUILD_COMMAND make -C <BINARY_DIR>
INSTALL_COMMAND make -C <BINARY_DIR> install
)
add_library(ncurses-lib INTERFACE)
target_link_libraries(ncurses-lib INTERFACE -L${CMAKE_CURRENT_SOURCE_DIR}/ncurses/lib -lncurses)
target_include_directories(ncurses-lib INTERFACE ncurses/include)
However for this to work, however I either have to make it so ncurses-lib depends on ncurses extern which will make it so it runs the external project every time (even if its already installed), or I have to manually run it.
What I would like it to do is only run the external project to run, if A: ncurses can't be found on the system with find_package, or B: if ncurses doesn't already exist in the lib folder (where the external project puts it).
However I am not sure how I would achieve this, any help would be great, thanks.

How can I run cmake from within cmake?

My project depends on mariadb-connector-c and I'm trying to automate the download, build and link process with cmake.
I currently download the project into a directory, I then try to execute generate ninja files and run them but I cannot run cmake at all:
execute_process(COMMAND "cmake -GNinja ." WORKING_DIRECTORY ${mariadb-connector-c_SOURCE_DIR})
I know this doesn't work because the next step, running ninja, fails:
execute_process(COMMAND "ninja" WORKING_DIRECTORY ${mariadb-connector-c_SOURCE_DIR})
cmake runs fine in CLI, I've tried using the full path to the cmake executable and replacing the dot with the variable with the full directory (which is also a valid variable, if you're wondering.)
How can I tell cmake to run cmake on that external project?
You can organize your project to a top-level CMakeLists.txt build your subprojects as ExternalProject.
This approach requires more work and maintenance of more CMake modules but it has its own benefits. I download Google Test as follows:
# Create download URL derived from version number.
set(GTEST_HOME https://github.com/google/googletest/archive)
set(GTEST_DOWNLOAD_URL ${GTEST_HOME}/release-${GTEST_VERSION}.tar.gz)
unset(GTEST_HOME)
# Download and build the Google Test library and add its properties to the third party arguments.
set(GTEST_ROOT ${THIRDPARTY_INSTALL_PATH}/gtest CACHE INTERNAL "")
ExternalProject_Add(gtest
URL ${GTEST_DOWNLOAD_URL}
CMAKE_ARGS -DBUILD_GTEST=ON -DBUILD_GMOCK=ON -DCMAKE_INSTALL_PREFIX=${GTEST_ROOT}
INSTALL_COMMAND make install
)
list(APPEND GLOBAL_THIRDPARTY_LIB_ARGS "-DGTEST_ROOT:PATH=${GTEST_ROOT}")
unset(GTEST_DOWNLOAD_URL)
unset(GTEST_ROOT)
The code abowe is inside my ExternalGoogleTest.cmake module which is included by CMakeLists.txt of third-party libraries:
set_directory_properties(PROPERTIES EP_BASE ${CMAKE_BINARY_DIR}/ThirdParty)
get_directory_property(THIRDPARTY_BASE_PATH EP_BASE)
set(THIRDPARTY_INSTALL_PATH ${THIRDPARTY_BASE_PATH}/Install)
set(GTEST_VERSION 1.8.0)
include(ExternalProject)
include(ExternalGoogleTest)
Your own project which depends on an external library will need a CMake module to build it as ExternalProject too. It can looks like:
ExternalProject_Add(my_project
DEPENDS gtest whatever
SOURCE_DIR ${CMAKE_SOURCE_DIR}/lib
CMAKE_ARGS
${GLOBAL_DEFAULT_ARGS}
${GLOBAL_THIRDPARTY_LIB_ARGS}
-DCMAKE_INSTALL_PREFIX=${DESIRED_INSTALL_PATH}/my_project
BUILD_COMMAND make
)
You can found more tips about this pattern here.

Using cmake as the configure tool (CONFIGURE_COMMAND) in CMakes "Add_External_Project" function

I am having an issue with CMakes Add_External_Project functionality (more of an annoyance than anything else). Specifically, I do not understand the keys CONFIGURE_COMMAND, BUILD_COMMAND and INSTALL_COMMAND.
In the following (working) example, which downloads Google's test library, the two files at the end of the question will ensure that the third party libraries are downloaded and built (not installed).
However, when I tried to add configure and build commands as "CONFIGURE_COMMAND" and "BUILD_COMMAND" (cmake . and cmake --build) instead of having to do execute_process CMake craps out with the error message:
[ 55%] Performing configure step for 'googletest'
/bin/sh: 1: cmake .: not found
Am I trying to do something that is obviously not within the scope of the Add_External_Project functionality?
Example Files:
CMakeLists.txt
cmake_minimum_required (VERSION 3.0)
project (Test VERSION 0.1.0.0 LANGUAGES CXX)
# Download and unpack googletest at configure time
configure_file("${CMAKE_CURRENT_SOURCE_DIR}/CMakeLists.txt.in" "${CMAKE_BINARY_DIR}/googletest-download/CMakeLists.txt" #ONLY)
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" . WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/googletest-download" )
execute_process(COMMAND ${CMAKE_COMMAND} --build . WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/googletest-download")
add_subdirectory("${CMAKE_BINARY_DIR}/googletest-src" "${CMAKE_BINARY_DIR}/googletest-build")
CMakeLists.txt.in
cmake_minimum_required(VERSION 3.0)
project(third-party NONE)
include(ExternalProject)
ExternalProject_Add(googletest
GIT_REPOSITORY https://github.com/google/googletest.git
GIT_TAG master
SOURCE_DIR "#CMAKE_BINARY_DIR#/googletest-src"
BINARY_DIR "#CMAKE_BINARY_DIR#/googletest-build"
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
INSTALL_COMMAND ""
TEST_COMMAND ""
)
If you don't specify CONFIGURE_COMMAND at all, it will assume a CMake project and run the appropriate cmake command for you (by appropriate, I mean it will use the same CMake generator as your main build, etc.). Similarly, if you leave out BUILD_COMMAND, it will also assume a CMake project and do cmake --build for you. So in your case, just leave out those two lines and ExternalProject_Add() should do exactly what you want.
The main reason you might specify these two options as empty strings is to prevent those steps from doing anything at all. This can be useful, for example, to use ExternalProject_Add() simply for its download and unpacking functionality. This exact situation is used in a technique described here for downloading the source of GoogleTest so it can be added to your project via add_subdirectory(), making it part of your build (see also this answer and other answers to that question for some related material). I suspect this might be where your code is derived from, as the structure looks similar.
For completeness, if you find yourself in a situation where you do need to specify a CMake command, don't use a bare cmake to refer to the command to run. Instead, always use ${CMAKE_COMMAND}, which is provided by CMake as the location of the CMake executable currently being used to process the file. Using this variable means cmake doesn't have to be on the user's PATH and also ensures that if the developer chooses to run a different version of CMake other than the one on the PATH, that same cmake will still be used for the command you are adding.
You can use PATCH_COMMAND like this:
option(WITH_MBEDTLS "Build with mbedtls" OFF)
if(WITH_MBEDTLS)
ExternalProject_Add(external-mbedtls
URL https://github.com/ARMmbed/mbedtls/archive/mbedtls-2.16.1.tar.gz
UPDATE_COMMAND ""
PATCH_COMMAND ./scripts/config.pl set MBEDTLS_THREADING_C &&
./scripts/config.pl set MBEDTLS_THREADING_PTHREAD
CMAKE_ARGS
-DCMAKE_INSTALL_PREFIX:PATH=${PROJECT_BINARY_DIR}/third_party/mbedtls
-DCMAKE_TOOLCHAIN_FILE:PATH=${TOOLCHAIN_FILE}
-DCMAKE_BUILD_TYPE:STRING=Debug
-DENABLE_TESTING:BOOL=OFF
-DENABLE_PROGRAMS:BOOL=ON
TEST_COMMAND ""
)
set(MBEDTLS_PREFIX ${PROJECT_BINARY_DIR}/third_party/mbedtls PARENT_SCOPE)
endif(WITH_MBEDTLS)

How can I distribute/install 3rd-party libraries with CMake?

I'm building and distributing an executable which relies on several 3rd-party libraries, most of which are built outside of CMake's buildsystem (though, if it helps, I can add and build them as custom targets in my CMake file).
Here's how I include the libs:
target_link_libraries(MyApp
Lib1_x64
Lib2_x64
etc.}
I'd like to include a CMake directive which installs my executable along with its dependencies. Is there a better way to do this other than calling the install command for every single dependency?
install(DIRECTORY ${DEV_PATH}/Release/ DESTINATION lib FILES_MATCHING PATTERN "libLib1*" PATTERN "*.svn" EXCLUDE PATTERN "*-obj" EXCLUDE)
Is there maybe a way to do this via the add_library command?
This is possible in nowadays with CMake 3.21, which allows you to get a set of dependent libraries via parameter RUNTIME_DEPENDENCY_SET in the install command. Keep in mind, that it will gather all dependent libraries including system, you will need to filter them.
cmake_minimum_required(VERSION 3.21)
# Install your app or library
install(
TARGETS ${PROJECT_NAME}
RUNTIME_DEPENDENCY_SET runtime_deps
)
# Install third-party libraries (exclude system libraries Windows/Unix)
LIST(APPEND pre_exclude_regexes "api-ms-.*")
LIST(APPEND pre_exclude_regexes "ext-ms-.*")
LIST(APPEND post_exclude_regexes ".*WINDOWS[\\/]system32.*")
LIST(APPEND post_exclude_regexes "^/lib" "^/usr" "^/bin")
install(RUNTIME_DEPENDENCY_SET runtime_deps
PRE_EXCLUDE_REGEXES ${pre_exclude_regexes}
POST_EXCLUDE_REGEXES ${post_exclude_regexes}
)
Sorry, no. The add_library command serves the purpose of creating a target in your makefile to build a library. target_link_dependencies just passes the various -L ... -l ... flags to the compiler for a target.
The normal CMake approach is to test for the presence of other libraries or executables on which yours depends, and fail if they are not available.
Handling dependencies is normally left to a package manage such as apt, rpm, etc. It is unwise to build your dependent components as part of your own makefile for a number of reasons:
Your clients might want to use a slightly different version
Any change in how those dependencies build requires you to modify your own build scripts
Your program may work with multiple versions of a dependency, all of which build in different ways.
As someone who uses CMake to build large parts of an entire embedded Linux distribution (see Open webOS), I advise you to keep your scripts focused very tightly on building just one thing.
I am also finding a way to deal this problem today.
and I found :
include(ExternalProject)
ExternalProject_Add()
does it!
here is my sample CMakeLists.txt:
cmake_minimum_required(VERSION 3.13)
project(third-party-build)
include(ExternalProject)
ExternalProject_Add(libuuid
PREFIX libuuid-prefix
# DOWNLOAD_DIR libuuid-download
# SOURCE_DIR libuuid-source
# DOWNLOAD_COMMAND wget xxx.com/xxx.tar.gz && tar zxf xxx.tar.gz
DOWNLOAD_COMMAND tar zxf ${PROJECT_SOURCE_DIR}/libuuid-1.0.3.tar.gz
CONFIGURE_COMMAND <DOWNLOAD_DIR>/libuuid-1.0.3/configure
BUILD_COMMAND ${MAKE}
BUILD_IN_SOURCE 1
INSTALL_COMMAND mkdir -p ${PROJECT_SOURCE_DIR}/lib/ && cp <SOURCE_DIR>/.libs/libuuid.a <SOURCE_DIR>/.libs/libuuid.so <SOURCE_DIR>/.libs/libuuid.so.1 <SOURCE_DIR>/.libs/libuuid.so.1.0.0 ${PROJECT_SOURCE_DIR}/lib/
LOG_DOWNLOAD 1
LOG_BUILD 1
# LOG_INSTALL 1
)

How do I use CMake ExternalProject_Add or alternatives in a cross-platform way?

I would like to build a third-party project that already has CMake as part of my project's CMake strips. ExternalProject_Add is for this purpose, but I have found it can only be made to work with a specific generator, and I wanted it to work on many platforms easily.
For example, here is my external project with an added script for zlib, which has its own CMakeLists.txt:
set(USE_PROJECT_CMAKE_MODULE_PATH "-DCMAKE_MODULE_PATH=${MAKE_MODULE_PATH}")
ExternalProject_Add(ZLIB
SOURCE_DIR ${CMAKE_SOURCE_DIR}/external/zlib
DOWNLOAD_COMMAND ""
UPDATE_COMMAND ""
CMAKE_ARGS
-DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
-DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}
-DCMAKE_C_FLAGS=${CMAKE_C_FLAGS}
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
${USE_PROJECT_CMAKE_MODULE_PATH}
INSTALL_COMMAND "")
ExternalProject_Add_Step(ZLIB installInternally
COMMAND cd <BINARY_DIR> && make install
DEPENDEES install
ALWAYS 1)
ExternalProject_Get_Property(ZLIB install_dir)
if(UNIX)
set(ZLIB_NAME libz)
else(UNIX)
set(ZLIB_NAME zlib)
endif(UNIX)
add_library(zlib UNKNOWN IMPORTED)
set_property(TARGET zlib PROPERTY IMPORTED_LOCATION ${install_dir}/lib/${ZLIB_NAME}.a)
set(ZLIB_LIBRARIES zlib)
set(ZLIB_LIBRARIES_OPTIONAL ${ZLIB_LIBRARIES})
set(ZLIB_DIR ${install_dir} CACHE INTERNAL "zlib ROOT dir")
set(ZLIB_INCLUDE_DIRS ${install_dir}/include CACHE INTERNAL "zlib include dirs")
set(ZLIB_DEFINES "-msse2 -mfpmath=sse" CACHE INTERNAL "zlib defines")
The problem with this is that it works with make, but not with Xcode or Visual Studio. Perhaps there is some way to take the CMake build commands passed to my project and forward them to ExternalProject_Add.
How can I write ExternalProject_Add calls in a cross-platform way with minimal code complexity, or is there a better alternative?
Problems
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
This is enough for single-configuration projects. But for Xcode and Visual Studio, you need to set CMAKE_CONFIGURATION_TYPES plus call build . --config at the build stage. See my answer.
COMMAND cd <BINARY_DIR> && make install
This will work only for Makefile generators of course. To be cross-platform you can use:
--build . --target install --config inside INSTALL_COMMAND of ExternalProject_Add.
Take a look at this template file, and in particular the following lines:
ExternalProject_Add(
"${current_project}"
URL
#HUNTER_PACKAGE_URL#
URL_HASH
SHA1=#HUNTER_PACKAGE_SHA1#
DOWNLOAD_DIR
"#HUNTER_PACKAGE_DOWNLOAD_DIR#"
SOURCE_DIR
"#HUNTER_PACKAGE_SOURCE_DIR#"
INSTALL_DIR
"#HUNTER_PACKAGE_INSTALL_PREFIX#"
# Not used, just avoid creating Install/<name> empty directory
BUILD_COMMAND ""
# This command is empty because all necessary targets will
# be built on install stage
CMAKE_ARGS
"-G#CMAKE_GENERATOR#"
"-C#HUNTER_CACHE_FILE#"
"-C#HUNTER_ARGS_FILE#"
"-D${postfix_name}=${${postfix_name}}"
"-DCMAKE_BUILD_TYPE=${configuration}"
"-DCMAKE_CONFIGURATION_TYPES=${configuration}"
"-DCMAKE_INSTALL_PREFIX=#HUNTER_PACKAGE_INSTALL_PREFIX#"
"-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}"
INSTALL_COMMAND
"#CMAKE_COMMAND#"
--build .
--target install
--config ${configuration}
--
${jobs_option}
)
Alternative
or is there a better alternative?
Have you seen Hunter?
You can add zlib just like this:
hunter_add_package(ZLIB)
find_package(ZLIB CONFIG REQUIRED)
target_link_libraries(... ZLIB::zlib)
This code works everywhere. Third party dependencies will be downloaded automatically in the configuration step. Example of building with different generator/toolchains (build.py is just a CMake wrapper that sets CMAKE_TOOLCHAIN_FILE and -G/-B):
build.py --toolchain mingw --config Release # MinGW Makefiles
build.py --toolchain vs-12-2013 --config Debug # Visual Studio 12 2013
build.py --toolchain xcode --config Release # Xcode
build.py --toolchain libcxx --config Release # Makefile with -stdlib=libc++ toolchain
build.py --toolchain ios-8-2 --config Release # Xcode with iOS SDK 8.2 toolchain
You got full control what options, build types or number of jobs you want to have while building third-party packages. For instance, this is how you can build four types, Debug, Release, MinSizeRel, and RelWithDebInfo for zlib and link MinSizeRel to the current project:
> build.py --toolchain xcode --verbose --config MinSizeRel --fwd "HUNTER_CONFIGURATION_TYPES=Release;Debug;MinSizeRel;RelWithDebInfo"
/.../clang /.../lib/libz-MinSizeRel.a ... -o /.../_builds/xcode/MinSizeRel/foo
> ls -la /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz*
99056 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-MinSizeRel.a
307872 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-RelWithDebInfo.a
109536 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz.a
258904 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libzd.a
CMake ExternalProject_Add calls work cross-platform by default and will only fail to do so if one uses particular commands that are only available on a subset of operating systems.
Typically, CMAKE_ARGS is used to pass information to each superbuild unit within an external project build. The CMakeLists.txt files that control each miniature part of the overall build use CMake's declarative syntax (e.g., "add_library(library_name SHARED filename1.hpp filename1.cpp). CMake will convert such syntax to the commands that are specific to the particular build system you wish to use (e.g., make and Ninja).
The sample above re: zlib fails to be cross-platform in part because the ExternalProject_Add_Step contains "COMMAND cd && make install", which necessarily only works in situations where invoking "cd" is actually the correct way to change directories, and where invoking "make" is actually the correct way to build software.
CMake's -E option provides a way to invoke basic operations like changing/copying/making/removing directories without making such assumptions.
(By the way, if you're using IDEs such as Visual Studio or Xcode, you'll likely want to invoke one or more IDE generators when using CMake. For instance, setting
-G "Eclipse CDT4 - Unix Makefiles" -DCMAKE_ECLIPSE_GENERATE_SOURCE_PROJECT=TRUE
will cause Eclipse projects to be generated in each build area, and also in the source code area that is shared for all builds. Of course, if you are using Xcode or Visual Studio, you'll have to substitute the appropriate flag for those IDEs. Alternatively, you could consider using Eclipse with Ninja on all platforms, though at the time of writing, I am not completely certain that Ninja is ready for prime-time on non-Linux, non-Windows operating systems.)