I have OpenSSL as a part of my CMake project. Since it does not have its own CMake file I build it as an ExternalProject. Unfortunately every time I rebuild my project the ExternalProject is also rebuilt and it is the biggest and slowest part by far.
How can I avoid that ExternalProject is being rebuilt every time?
I tried BUILD_ALWAYS = false but it did not help. OpenSSL has to be built from sources as a static library.
Add empty UPDATE_COMMAND:
ExternalProject_Add(
...
UPDATE_COMMAND ""
...
)
Related
I'm working on some modifications to the openEMS project. This project uses cmake to build all of its components. The top level CMakeLists.txt file contains the following:
# ...
ExternalProject_Add( openEMS
DEPENDS fparser CSXCAD
SOURCE_DIR ${PROJECT_SOURCE_DIR}/openEMS
CMAKE_ARGS -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE} -DFPARSER_ROOT_DIR=${CMAKE_INSTALL_PREFIX} -DCSXCAD_ROOT_DIR=${CMAKE_INSTALL_PREFIX} -DWITH_MPI=${WITH_MPI} -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
)
# ...
Inside the openEMS directory, there's another CMakeLists.txt with the following:
# ...
set(SOURCES
openems.cpp
)
# ...
add_library( openEMS SHARED ${SOURCES})
# ...
After building the project successfully once, make does not rebuild anything when, for example, openems.cpp is modified. Why?
$ mkdir build
$ cd build
$ cmake -DBUILD_APPCSXCAD=NO
$ make
[builds all files]
$ touch ../openEMS/openems.cpp
$ make
[ 33%] Built target fparser
[ 66%] Built target CSXCAD
[100%] Built target openEMS
(noting is built)
I have checked and the modification date of openems.cpp is newer than the target. Even deleting the produced library files and binaries, both in the install directory and in the build directory, does not cause it to rebuild anything. The only way I can get it to rebuild is by deleting everything in the build directory and re-running cmake which, of course, rebuilds everything.
This looks like a case of the following. Quoting from the docs for ExternalProject_Add at the section titled "Build Step Options":
BUILD_ALWAYS <bool>
Enabling this option forces the build step to always be run. This can be the easiest way to robustly ensure that the external project's own build dependencies are evaluated rather than relying on the default success timestamp-based method. This option is not normally needed unless developers are expected to modify something the external project's build depends on in a way that is not detectable via the step target dependencies (e.g. SOURCE_DIR is used without a download method and developers might modify the sources in SOURCE_DIR).
If that's the case, the solution would be to add the BUILD_ALWAYS argument to the ExternalProject_Add call like.
ExternalProject_Add( openEMS
DEPENDS fparser CSXCAD
SOURCE_DIR ${PROJECT_SOURCE_DIR}/openEMS
CMAKE_ARGS -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE} -DFPARSER_ROOT_DIR=${CMAKE_INSTALL_PREFIX} -DCSXCAD_ROOT_DIR=${CMAKE_INSTALL_PREFIX} -DWITH_MPI=${WITH_MPI} -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
BUILD_ALWAYS TRUE
)
If you confirm that this solves the issue, you might want to raise this as an issue to the maintainers of openEMS.
Also note that since the external project there is using CMake as a buildsystem, you could also add the CONFIGURE_HANDLED_BY_BUILD TRUE to the argument list. See the docs for more info.
Edit: The asker opened a GitHub Pull-Request.
I basically ask the same question as has been ask here. The question has however not been answered.
I want to use googletest in my project. For this I'm using ExternalProject_Add() which clones the testsuite with git. After that, I like to use add_subdirectory().
This is also what is described in the official repository. The nice thing about this approach is, that the build scripts in googletest handle the building process themself.
The problem is however, that add_subdirectory() can not find the source folder, since it does not exists from the start. Therefore, add_subdirectory() should depend on the completion of ExternalProject_Add().
Is is possible to make add_subdirectory() dependent of ExternalProject_Add(), like add_dependencies() does for targets?
PS. I can make it all compile if I comment add_subdirectory() out, build it (which ends with an error because the googletest library is missing), uncomment it and build it again (success).
ExternalProject_Add(
googletest
GIT_REPOSITORY https://github.com/google/googletest.git
GIT_TAG master
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
INSTALL_COMMAND ""
TEST_COMMAND ""
)
ExternalProject_Get_Property(googletest source_dir binary_dir)
set(GTEST_INCLUDE_DIR ${source_dir}/googletest/include)
set(GMOCK_INCLUDE_DIR ${source_dir}/googlemock/include)
add_subdirectory(${source_dir}
${binary_dir})
I used this tutorial to accomplish that. Just put your ExternalProject code in a separate file, say "CMakeLists.txt.dependencies" and then launch another cmake with execute_process. I use configure_file first to inject configuration information into the external project and to copy it into the build tree.
configure_file(CMakeLists.txt.dependency.in dependency/CMakeLists.txt)
execute_process(COMMAND "${CMAKE_COMMAND}" -G "${CMAKE_GENERATOR}" .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
execute_process(COMMAND "${CMAKE_COMMAND}" --build .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
Notice that the tutorial has the ExternalProject code in a separate CMakeLists.txt file. So does gtest official documentation. That is because ExternalProject runs at build-time, not configure time. This is a hack to use ExternalProject at configure time by putting it in a third project and executing a separate CMake+build run in the middle of your main CMake configuration step.
With version 3.11 CMake added a FetchContent command. I haven't used it, but the documentation makes it look like a configuration-time replacement for ExternalProject. That would fetch the content at configuration time, making it available for a later add_subdirectory command.
The ExternalProject and FetchContent commands will work fine for a small number of small and/or obscure dependencies. If your dependencies are more than 2, large, and/or popular, I recommend you look at a dependency manager like Conan. It partners with CMake quite well.
You are not following the method correctly. There should only be ExternalProject_Add() as the final command in the CMakeLists.txt.in file.
The CMakeLists.txt.in file is called by the outer CMakeLists.txt file so that the subproject processing happens at configure time via execute_process(). CMakeLists.txt.in is acting as just a glorified downloader.
So, all other commands like add_subdirectory() are added to the outer CMakeLists.txt file.
Goal
I want to define several targets:
make msan: compiles the code with clang with memory sanitizer
make tsan: compiles the code with clang with thread sanitizer
make : compiles the code with gcc
And be able to easily switch between them.
For example I don't want each time I switch rebuild all my objects, (I will have to do it the first time of course, but later if I modify a file and I do make and then make asan it should recompile only this file for each target)
What I have done so far
I have managed to create these targets and from the root directory, but each time I have to do a make clean and recompile.
option(CLANG_MSAN "Enable Clang memory sanitizer" OFF)
if (CLANG_MSAN)
set (CMAKE_CXX_FLAGS "-g -fsanitize=address -fno-omit-frame-pointer")
endif()
add_custom_target(asan
COMMAND ${CMAKE_COMMAND}
-DCLANG_MSAN=ON
-DCMAKE_CXX_COMPILER=clang++
-DCMAKE_C_COMPILER=clang)
Is it possible to do such a thing with CMake?
Yes, but use multiple build directories:
Create a build directory per configuration.
Configure your project in your build directories with the parameters you need. E.g. cmake -DCMAKE_COMPILER=clang -DCMAKE_C_FLAGS="-fsanitize=thread" .. or the stuff from your question.
If you switch the build directory, you changed your setup.
This implies out-of-source builds, which are encouraged by CMake anyway.
I believe you should be able to achieve this with the ExternalProject module. You could add three external projects, one for msan, one for tsan, and one for the basic GCC build. The two sanitiser builds would be marked as EXCLUDE_FROM_ALL 1.
The CMakeLists for all three of them could share the common part via include().
Something like this:
Root CMakeLists.txt
ExternalProject_Add(msan
EXCLUDE_FROM_ALL 1
SOURCE_DIR msan
CMAKE_GENERATOR ...
)
ExternalProject_Add(tsan
EXCLUDE_FROM_ALL 1
SOURCE_DIR tsan
CMAKE_GENERATOR ...
)
ExternalProject_Add(normal
SOURCE_DIR src
CMAKE_GENERATOR ...
)
src/CMakeLists.txt
include(common.cmake)
src/common.cmake
# Normal CMake code for your project
add_library(...)
msan/CMakeLists.txt
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g -fsanitize=address -fno-omit-frame-pointer")
include(../src/common.cmake)
Im build libjpeg as external project. Its build normally.
Here projects folder structure:
${SOURCE_DIR}/
${SOME_BUILD_DIR}/
externals/
jpeg-9a/
jpeg-pre/
externals/
jpeg-9a/
jpeg-overlay/
CMakeLists.txt
tarballs/
jpegsrc.v9a.tar.gz
CMakeLists.txt
But tarball downloaded and unpacked again if I start building project from zero.
In other words I clean ${SOME_BUILD_DIR}/. At next build cmake do the next:
download tarballs/jpegsrc.v9a.tar.gz
unpack libjpeg into externals/jpeg-9a
copy externals/jpeg-overlay/CMakeLists.txt into externals/jpeg-9a
build libjpeg in ${SOME_BUILD_DIR}/externals/jpeg-9a/
Actually first 3 points can be omitted. But my interest only in first action. Is there way to prevent extra download?
Here is my ${SOURCE_DIR}/CMakeLists.txt:
cmake_minimum_required(VERSION 2.8)
include(ExternalProject)
set(EXTERNALS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/externals)
set(JPEG_VERSION "9a")
set(JPEG_URI http://ijg.org/files/jpegsrc.v${JPEG_VERSION}.tar.gz)
set(JPEG_DIR ${EXTERNALS_DIR}/jpeg-${JPEG_VERSION})
ExternalProject_Add(
jpeg
STAMP_DIR ${CMAKE_BINARY_DIR}/externals/jpeg-pre
BINARY_DIR ${CMAKE_BINARY_DIR}/externals/jpeg-${JPEG_VERSION}
URL ${JPEG_URI}
SOURCE_DIR ${JPEG_DIR}
DOWNLOAD_DIR ${CMAKE_CURRENT_SOURCE_DIR}/tarballs
CMAKE_ARGS ""
UPDATE_COMMAND cmake -E copy_directory ${EXTERNALS_DIR}/jpeg-${JPEG_VERSION}-overlay/. ${JPEG_DIR}
INSTALL_COMMAND ""
TEST_COMMAND ""
)
See https://github.com/anton-sergeev/cmake_externalproject for details.
You should not place generated files in the source tree.
In your case, the problematic line is DOWNLOAD_DIR ${CMAKE_CURRENT_SOURCE_DIR}/tarballs. A cleaner approach would be to place the download in a directory in the binary dir and compile from there. Now, of course that means that when wiping the build directory it will also wipe the downloaded sources. Which kind of solves the problem of having to download a file that you already had, although probably not in the way you would have liked.
The thing is, this is by design. Wiping the binary dir is conceptually equivalent to telling CMake to start over from scratch. There is no point in trying to reuse stuff in this case because, well, you want to start from scratch. The correct workflow to enable reusing with CMake is simply: Do not wipe the build directory. Instead rely on make clean to enforce full rebuilds and only wipe the build directory if you want to perform a full reconfigure of CMake.
The only clean way to avoid redownload is to move the download out of the ExternalProject command. For instance, you could place the extracted files into the source tree and check them in as part of the project. Or have them downloaded by a custom execute_process command which implements the desired behavior.
If you run md5 jpegsrc.v9a.tar.gz, you can use the outputted hash with the ExternalProject URL_MD5 option.
ExternalProject_Add(
jpeg
STAMP_DIR ${CMAKE_BINARY_DIR}/externals/jpeg-pre
BINARY_DIR ${CMAKE_BINARY_DIR}/externals/jpeg-${JPEG_VERSION}
URL ${JPEG_URI}
URL_MD5 <md5_hash_of_downloaded_file>
SOURCE_DIR ${JPEG_DIR}
DOWNLOAD_DIR ${CMAKE_CURRENT_SOURCE_DIR}/tarballs
CMAKE_ARGS ""
UPDATE_COMMAND cmake -E copy_directory ${EXTERNALS_DIR}/jpeg-${JPEG_VERSION}-overlay/. ${JPEG_DIR}
INSTALL_COMMAND ""
TEST_COMMAND ""
)
Hello I'm facing a problem regarding CMake and External Projects.
I set a compiler and some flags via CMAKE_CACHE_ARGS and/or CMAKE_ARGS this works the first time I run make but on any subsequent call the CMake Cache of the external project is rebuild (deleted) and the flags are not set accordingly to the flags I specified! So I wonder is there any workaround/way to specify the compiler only once to prevent rebuilding of the cache?
Following is a very basic test project which downloads and compiles GTest, first call to make compiles with Clang++ and the given flags, any following call to make will cause the CMake Cache to be rebuild without the proper flags being set!
cmake_minimum_version_required(VERSION 2.8.6)
project(test)
include(ExternalProject)
ExternalProject_Add(
GTest
SVN_REPOSITORY http://googletest.googlecode.com/svn/tags/release-1.7.0/
CMAKE_ARGS
-DCMAKE_CXX_COMPILER:STRING=clang++
-DCMAKE_CXX_FLAGS:STRING="\"-std=c++1y -stdlib=libc++\""
INSTALL_COMMAND "" # One can not install GTest so dont do anything here
LOG_DOWNLOAD 1
LOG_UPDATE 1
LOG_CONFIGURE 1
LOG_BUILD 1
)
I had the same problem, but with a different setup. Although this answer does not seem to apply to your setup it might be helpful for someone else.
In my case the issue was declaring the project with:
project(test LANGUAGES C)
The external project was a C++ project. Adding CXX to languages (or removing the option altogether, since C CXX is the default) solved the problem for me.