What is INSTALL_DIR useful for in ExternalProject_Add command? - cmake

I don't understand the usage of INSTALL_DIR in ExternalProject_Add command. I try to use it but it does not seem to work. Here is an example of a CMakeLists.txt, using Eigen library which compiles quickly:
cmake_minimum_required (VERSION 2.6)
project (example CXX)
include(ExternalProject)
include(ProcessorCount)
set(CMAKE_VERBOSE_MAKEFILE ON)
ProcessorCount(N)
if(NOT N EQUAL 0)
set(CMAKE_BUILD_FLAGS -j${N})
endif()
ExternalProject_Add
(
mylib
PREFIX myprefix
DOWNLOAD_COMMAND wget http://bitbucket.org/eigen/eigen/get/3.2.4.tar.gz && tar xvzf 3.2.4.tar.gz -C mylib --strip-components=1
)
I chose the following project hierarchy:
project
CMakeLists.txt
build/
From build repository, I type:
cmake ..
make
The installation process fails with the following message:
file cannot create directory: /usr/local/include/eigen3.
Maybe need administrative privileges.
As far as I understand, it means that I need to define a "prefix" during the configuration step:
cmake -D CMAKE_INSTALL_PREFIX=$INSTALL_DIR ..
But, the INSTALL_DIR variable is already defined in the ExternalProject_Add command. However, I get the same error when I modify the value of INSTALL_DIR by adding
INSTALL_DIR myprefix/src/install
in the ExternalProject_Add command.
So, what is INSTALL_DIR useful for?
What am I doing wrong?
Of course, I know how to provide my own configuration command to add a prefix and solve the problem. But it is not my question. My question is: if I have to do that, what is the purpose of INSTALL_DIR?

From what I found in this discussion https://www.mail-archive.com/cmake#cmake.org/msg51663.html (scroll to the end of the page to navigate through the thread messages) it is indeed pretty common thing to use CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}/contrib
Furthermore, lurking through the ExternalProject.cmake module I found out that the only effect setting this directory has is that CMake will create directory specified in INSTALL_DIR before doing anything else.
Also, it will set the property that you can gather through ExternalProject_Get_Property(${project_name} install_dir) command.
And that's pretty much it.
// As of CMake version 3.2.2

Related

ExternalProject_Add autogen project prevent configure on rebuild

I have a CMake project on linux and I'm using ExternalProject to build Google Protobuf. It works great, however any subsequent builds still call the configure step in the external project (which is annoying because protobuf is an autogen project with a rather long step). I used the UPDATE_DISCONNECTED argument so it wouldn't re-clone which helps some, but you'd think if it didn't re-clone, it wouldn't need to re-configure or re-build/install. How can I get CMake to just build it the one time and skip subsequent builds (i.e. my next make from the build directory)?
Here's my CMakeLists.txt
cmake_minimum_required(VERSION 2.8.11)
project(pbuf_test)
include(${PROJECT_SOURCE_DIR}/cmake/protogen.cmake)
set(PBUF_DIR ${PROJECT_BINARY_DIR}/protobuf)
include(ExternalProject)
ExternalProject_Add(protobuf
PREFIX ${PROTOBUF_DIR}
GIT_REPOSITORY https://github.com/google/protobuf.git
GIT_TAG v3.4.1
UPDATE_DISCONNECTED 1
BUILD_IN_SOURCE 1
CONFIGURE_COMMAND ./autogen.sh COMMAND ./configure --prefix=${PBUF_DIR}
)
set(PBUF_INCLUDE_DIR ${PBUF_DIR}/include)
set(PBUF_LIBRARY ${PBUF_DIR}/lib/libprotobuf.so)
set(PBUF_PROTOC ${PBUF_DIR}/bin/protoc)
file(GLOB PBUF_FILES "${PROJECT_SOURCE_DIR}/msg/*.proto")
custom_protobuf_generate_cpp(PBUF_SRCS PBUF_HDRS ${PBUF_FILES})
include_directories(
${PROJECT_BINARY_DIR}
${PBUF_INCLUDE_DIR}
)
add_executable(${PROJECT_NAME} main.cpp ${PBUF_SRCS} ${PBUF_HDRS})
add_dependencies(${PROJECT_NAME} protobuf)
target_link_libraries(${PROJECT_NAME}
${PBUF_LIBRARY}
)
Full example project here
The only way I know is to move the steps to build the external project into a script called by add_command. And then add an dependency between your project and the external project's output/lib/binary.
Pro: Avoids unnecessary rebuilds
Pro: Avoids unexpected updates of the external lib
Con: Does not rebuild in case the external repository was updated (and this is required)
Con: Additional work to write such a script
BTW: Within my project, we do it the hard way. But we do not want any unexpected updates of external libs.
It seems the solution to this for now that still uses ExternalProject is to move the configure step into the download step.
Something like this. Note the name of the target is the directory it expects the download to create. This might only work on certain operating systems because of the use of the && joiner: I'm a little new to cmake.
ExternalProject_Add(protobuf
PREFIX ${PROTOBUF_DIR}
DOWNLOAD_COMMAND git clone https://github.com/google/protobuf.git protobuf &&
cd protobuf &&
git checkout v3.4.1 &&
./autogen.sh &&
./configure --prefix=${PBUF_DIR}
UPDATE_COMMAND git pull
UPDATE_DISCONNECTED 1
BUILD_IN_SOURCE 1
CONFIGURE_COMMAND ""
)

How can I run cmake from within cmake?

My project depends on mariadb-connector-c and I'm trying to automate the download, build and link process with cmake.
I currently download the project into a directory, I then try to execute generate ninja files and run them but I cannot run cmake at all:
execute_process(COMMAND "cmake -GNinja ." WORKING_DIRECTORY ${mariadb-connector-c_SOURCE_DIR})
I know this doesn't work because the next step, running ninja, fails:
execute_process(COMMAND "ninja" WORKING_DIRECTORY ${mariadb-connector-c_SOURCE_DIR})
cmake runs fine in CLI, I've tried using the full path to the cmake executable and replacing the dot with the variable with the full directory (which is also a valid variable, if you're wondering.)
How can I tell cmake to run cmake on that external project?
You can organize your project to a top-level CMakeLists.txt build your subprojects as ExternalProject.
This approach requires more work and maintenance of more CMake modules but it has its own benefits. I download Google Test as follows:
# Create download URL derived from version number.
set(GTEST_HOME https://github.com/google/googletest/archive)
set(GTEST_DOWNLOAD_URL ${GTEST_HOME}/release-${GTEST_VERSION}.tar.gz)
unset(GTEST_HOME)
# Download and build the Google Test library and add its properties to the third party arguments.
set(GTEST_ROOT ${THIRDPARTY_INSTALL_PATH}/gtest CACHE INTERNAL "")
ExternalProject_Add(gtest
URL ${GTEST_DOWNLOAD_URL}
CMAKE_ARGS -DBUILD_GTEST=ON -DBUILD_GMOCK=ON -DCMAKE_INSTALL_PREFIX=${GTEST_ROOT}
INSTALL_COMMAND make install
)
list(APPEND GLOBAL_THIRDPARTY_LIB_ARGS "-DGTEST_ROOT:PATH=${GTEST_ROOT}")
unset(GTEST_DOWNLOAD_URL)
unset(GTEST_ROOT)
The code abowe is inside my ExternalGoogleTest.cmake module which is included by CMakeLists.txt of third-party libraries:
set_directory_properties(PROPERTIES EP_BASE ${CMAKE_BINARY_DIR}/ThirdParty)
get_directory_property(THIRDPARTY_BASE_PATH EP_BASE)
set(THIRDPARTY_INSTALL_PATH ${THIRDPARTY_BASE_PATH}/Install)
set(GTEST_VERSION 1.8.0)
include(ExternalProject)
include(ExternalGoogleTest)
Your own project which depends on an external library will need a CMake module to build it as ExternalProject too. It can looks like:
ExternalProject_Add(my_project
DEPENDS gtest whatever
SOURCE_DIR ${CMAKE_SOURCE_DIR}/lib
CMAKE_ARGS
${GLOBAL_DEFAULT_ARGS}
${GLOBAL_THIRDPARTY_LIB_ARGS}
-DCMAKE_INSTALL_PREFIX=${DESIRED_INSTALL_PATH}/my_project
BUILD_COMMAND make
)
You can found more tips about this pattern here.

Using cmake as the configure tool (CONFIGURE_COMMAND) in CMakes "Add_External_Project" function

I am having an issue with CMakes Add_External_Project functionality (more of an annoyance than anything else). Specifically, I do not understand the keys CONFIGURE_COMMAND, BUILD_COMMAND and INSTALL_COMMAND.
In the following (working) example, which downloads Google's test library, the two files at the end of the question will ensure that the third party libraries are downloaded and built (not installed).
However, when I tried to add configure and build commands as "CONFIGURE_COMMAND" and "BUILD_COMMAND" (cmake . and cmake --build) instead of having to do execute_process CMake craps out with the error message:
[ 55%] Performing configure step for 'googletest'
/bin/sh: 1: cmake .: not found
Am I trying to do something that is obviously not within the scope of the Add_External_Project functionality?
Example Files:
CMakeLists.txt
cmake_minimum_required (VERSION 3.0)
project (Test VERSION 0.1.0.0 LANGUAGES CXX)
# Download and unpack googletest at configure time
configure_file("${CMAKE_CURRENT_SOURCE_DIR}/CMakeLists.txt.in" "${CMAKE_BINARY_DIR}/googletest-download/CMakeLists.txt" #ONLY)
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" . WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/googletest-download" )
execute_process(COMMAND ${CMAKE_COMMAND} --build . WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/googletest-download")
add_subdirectory("${CMAKE_BINARY_DIR}/googletest-src" "${CMAKE_BINARY_DIR}/googletest-build")
CMakeLists.txt.in
cmake_minimum_required(VERSION 3.0)
project(third-party NONE)
include(ExternalProject)
ExternalProject_Add(googletest
GIT_REPOSITORY https://github.com/google/googletest.git
GIT_TAG master
SOURCE_DIR "#CMAKE_BINARY_DIR#/googletest-src"
BINARY_DIR "#CMAKE_BINARY_DIR#/googletest-build"
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
INSTALL_COMMAND ""
TEST_COMMAND ""
)
If you don't specify CONFIGURE_COMMAND at all, it will assume a CMake project and run the appropriate cmake command for you (by appropriate, I mean it will use the same CMake generator as your main build, etc.). Similarly, if you leave out BUILD_COMMAND, it will also assume a CMake project and do cmake --build for you. So in your case, just leave out those two lines and ExternalProject_Add() should do exactly what you want.
The main reason you might specify these two options as empty strings is to prevent those steps from doing anything at all. This can be useful, for example, to use ExternalProject_Add() simply for its download and unpacking functionality. This exact situation is used in a technique described here for downloading the source of GoogleTest so it can be added to your project via add_subdirectory(), making it part of your build (see also this answer and other answers to that question for some related material). I suspect this might be where your code is derived from, as the structure looks similar.
For completeness, if you find yourself in a situation where you do need to specify a CMake command, don't use a bare cmake to refer to the command to run. Instead, always use ${CMAKE_COMMAND}, which is provided by CMake as the location of the CMake executable currently being used to process the file. Using this variable means cmake doesn't have to be on the user's PATH and also ensures that if the developer chooses to run a different version of CMake other than the one on the PATH, that same cmake will still be used for the command you are adding.
You can use PATCH_COMMAND like this:
option(WITH_MBEDTLS "Build with mbedtls" OFF)
if(WITH_MBEDTLS)
ExternalProject_Add(external-mbedtls
URL https://github.com/ARMmbed/mbedtls/archive/mbedtls-2.16.1.tar.gz
UPDATE_COMMAND ""
PATCH_COMMAND ./scripts/config.pl set MBEDTLS_THREADING_C &&
./scripts/config.pl set MBEDTLS_THREADING_PTHREAD
CMAKE_ARGS
-DCMAKE_INSTALL_PREFIX:PATH=${PROJECT_BINARY_DIR}/third_party/mbedtls
-DCMAKE_TOOLCHAIN_FILE:PATH=${TOOLCHAIN_FILE}
-DCMAKE_BUILD_TYPE:STRING=Debug
-DENABLE_TESTING:BOOL=OFF
-DENABLE_PROGRAMS:BOOL=ON
TEST_COMMAND ""
)
set(MBEDTLS_PREFIX ${PROJECT_BINARY_DIR}/third_party/mbedtls PARENT_SCOPE)
endif(WITH_MBEDTLS)

How to download a toolchain for cross compilation in cmake from separate file?

I have a project with a CMakeLists.txt files in the root and the project compiles fine on Linux and OSX. Now I want to cross compile it for MIPS OpenWRT.
I would like to automate it as much as possible, so I would use following code to download the toolchain and set the compiler variables:
ExternalProject_Add(ar71xx-toolchain
PREFIX "${PROJECT_BINARY_DIR}/external/openwrt"
URL "http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/OpenWrt-Toolchain-ar71xx-for-mips_34kc-gcc-4.8-linaro_uClibc-0.9.33.2.tar.bz2"
UPDATE_COMMAND ""
PATCH_COMMAND ""
BUILD_COMMAND ""
CONFIGURE_COMMAND ""
INSTALL_COMMAND ""
)
ExternalProject_Get_Property(ar71xx-toolchain SOURCE_DIR)
SET(CMAKE_C_COMPILER ${SOURCE_DIR}/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/bin/mips-openwrt-linux-gcc)
SET(CMAKE_CXX_COMPILER ${SOURCE_DIR}/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/bin/mips-openwrt-linux-g++)
SET(CMAKE_STRIP ${SOURCE_DIR}/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/bin/mips-openwrt-linux-strip)
I thought that I can put it in a separate toolchain file and pass it with -DCMAKE_TOOLCHAIN_FILE, but it seems that ExternalProject_Add is not executed inside the toolchain file.
I would like to avoid putting the toolchain download step into the main CMakeLists.txt since it's actually not essential for the project itself and would require doing the same for each target platform...
So is there a way to define optional steps for a current cross compile build and pass it somehow as command line parameter to be executed before the main project build?
UPDATE:
Based on Tsyvarev's answer that works for me in the toolchain file:
set(CMAKE_SYSTEM_NAME Linux)
set(TOOLCHAIN_DIR ${PROJECT_BINARY_DIR}/external/openwrt/toolchain)
if(NOT EXISTS ${TOOLCHAIN_DIR})
file(DOWNLOAD http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/OpenWrt-Toolchain-ar71xx-for-mips_34kc-gcc-4.8-linaro_uClibc-0.9.33.2.tar.bz2 ${TOOLCHAIN_DIR}/toolchain.tar.bz2 SHOW_PROGRESS)
execute_process(COMMAND tar --strip-components=2 -xjf ${TOOLCHAIN_DIR}/toolchain.tar.bz2 WORKING_DIRECTORY ${TOOLCHAIN_DIR})
execute_process(COMMAND rm ${TOOLCHAIN_DIR}/toolchain.tar.bz2)
endif()
SET(CMAKE_C_COMPILER ${TOOLCHAIN_DIR}/bin/mips-openwrt-linux-gcc)
SET(CMAKE_CXX_COMPILER ${TOOLCHAIN_DIR}/bin/mips-openwrt-linux-g++)
SET(CMAKE_STRIP ${TOOLCHAIN_DIR}/bin/mips-openwrt-linux-strip)
SET(CMAKE_FIND_ROOT_PATH ${TOOLCHAIN_DIR})
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
There is one issue when passing -DCMAKE_TOOLCHAIN_FILE as CMAKE parameter to other projects added with ExternalProject_Add. Because of it's own ${PROJECT_BINARY_DIR} it will download the toolchain again. But this is another problem...
ExternalProject_add executes all steps at build time, not at configuration time.
For download file you can use file(DOWNLOAD ...) command. For extract files from archive just use execute_process with appropriate command.

How can I distribute/install 3rd-party libraries with CMake?

I'm building and distributing an executable which relies on several 3rd-party libraries, most of which are built outside of CMake's buildsystem (though, if it helps, I can add and build them as custom targets in my CMake file).
Here's how I include the libs:
target_link_libraries(MyApp
Lib1_x64
Lib2_x64
etc.}
I'd like to include a CMake directive which installs my executable along with its dependencies. Is there a better way to do this other than calling the install command for every single dependency?
install(DIRECTORY ${DEV_PATH}/Release/ DESTINATION lib FILES_MATCHING PATTERN "libLib1*" PATTERN "*.svn" EXCLUDE PATTERN "*-obj" EXCLUDE)
Is there maybe a way to do this via the add_library command?
This is possible in nowadays with CMake 3.21, which allows you to get a set of dependent libraries via parameter RUNTIME_DEPENDENCY_SET in the install command. Keep in mind, that it will gather all dependent libraries including system, you will need to filter them.
cmake_minimum_required(VERSION 3.21)
# Install your app or library
install(
TARGETS ${PROJECT_NAME}
RUNTIME_DEPENDENCY_SET runtime_deps
)
# Install third-party libraries (exclude system libraries Windows/Unix)
LIST(APPEND pre_exclude_regexes "api-ms-.*")
LIST(APPEND pre_exclude_regexes "ext-ms-.*")
LIST(APPEND post_exclude_regexes ".*WINDOWS[\\/]system32.*")
LIST(APPEND post_exclude_regexes "^/lib" "^/usr" "^/bin")
install(RUNTIME_DEPENDENCY_SET runtime_deps
PRE_EXCLUDE_REGEXES ${pre_exclude_regexes}
POST_EXCLUDE_REGEXES ${post_exclude_regexes}
)
Sorry, no. The add_library command serves the purpose of creating a target in your makefile to build a library. target_link_dependencies just passes the various -L ... -l ... flags to the compiler for a target.
The normal CMake approach is to test for the presence of other libraries or executables on which yours depends, and fail if they are not available.
Handling dependencies is normally left to a package manage such as apt, rpm, etc. It is unwise to build your dependent components as part of your own makefile for a number of reasons:
Your clients might want to use a slightly different version
Any change in how those dependencies build requires you to modify your own build scripts
Your program may work with multiple versions of a dependency, all of which build in different ways.
As someone who uses CMake to build large parts of an entire embedded Linux distribution (see Open webOS), I advise you to keep your scripts focused very tightly on building just one thing.
I am also finding a way to deal this problem today.
and I found :
include(ExternalProject)
ExternalProject_Add()
does it!
here is my sample CMakeLists.txt:
cmake_minimum_required(VERSION 3.13)
project(third-party-build)
include(ExternalProject)
ExternalProject_Add(libuuid
PREFIX libuuid-prefix
# DOWNLOAD_DIR libuuid-download
# SOURCE_DIR libuuid-source
# DOWNLOAD_COMMAND wget xxx.com/xxx.tar.gz && tar zxf xxx.tar.gz
DOWNLOAD_COMMAND tar zxf ${PROJECT_SOURCE_DIR}/libuuid-1.0.3.tar.gz
CONFIGURE_COMMAND <DOWNLOAD_DIR>/libuuid-1.0.3/configure
BUILD_COMMAND ${MAKE}
BUILD_IN_SOURCE 1
INSTALL_COMMAND mkdir -p ${PROJECT_SOURCE_DIR}/lib/ && cp <SOURCE_DIR>/.libs/libuuid.a <SOURCE_DIR>/.libs/libuuid.so <SOURCE_DIR>/.libs/libuuid.so.1 <SOURCE_DIR>/.libs/libuuid.so.1.0.0 ${PROJECT_SOURCE_DIR}/lib/
LOG_DOWNLOAD 1
LOG_BUILD 1
# LOG_INSTALL 1
)