CMAKE : Build only one directory of an downloaded external project - cmake

CMAKE : I only want to build part of a downloadable external project
The external project I want to use has the following structure :
- ExternalProject
- Subproject A <- this i care for
- Subproject B
It's all in the same archive that is available for download. The problem is that Subproject B has more dependencies that I don't want in my project, also it's not relevant for what I'm doing. The subprojects are buildable on their own, so for now I just took Subproject A out of the archive and put it in my main project which is working fine, but I'd like to not deploy the external project with my project but allow the user to download the external project on his own when running cmake. Sadly there is no archive for each of the subprojects on their own so all I can download is the full external project.
What I want is to tell CMAKE to download and unpack the whole archive ExternalProject but then only add Subproject A to my project. I read all the documentation on ExternalProject_add but it mostly allows for detailed configuration of the build parameters of the project. Maybe I'm just looking for the wrong keyword and my question is really simple to answer - or it is just not possible.
If someone could point me towards the right approach here I'd be very thankful.
Actual project :
The mentioned archive has two subfolders "octomap" and "octovis". octomap is the one I want to build while octovis will create a lot of errors if the dependencies are not met.
ExternalProject_Add(octomap-1.6.5
URL https://github.com/OctoMap/octomap/archive/v1.6.5.tar.gz
URL_MD5 de09b1189a03ac8cbe4f813951e86605
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/octomap/
CMAKE_ARGS "-DCMAKE_INSTALL_PREFIX=${EXTERNAL_INSTALL_DIR}"
)

You can define all commands explicitly. For instance you have a directory structure:
FooBar/
- CMakeLists.txt
- Foo/
- - CMakeLists.txt # You wanna build
- Bar/
- - CMakeLists.txt # You wanna ignore
Example with hardcoded paths:
set(ext_dir "${CMAKE_BINARY_DIR}/third-party-activities/ExternalProject/")
set(foobar_dir "${ext_dir}/FooBar")
ExternalProject_Add(
FooBar
URL "your-url-here"
SOURCE_DIR "${foobar_dir}/Source"
CONFIGURE_COMMAND
"${CMAKE_COMMAND}"
"-H${foobar_dir}/Source/Foo"
"-B${foobar_dir}/Builds"
"-DCMAKE_INSTALL_PREFIX=${EXTERNAL_INSTALL_DIR}"
BUILD_COMMAND
"${CMAKE_COMMAND}" --build "${foobar_dir}/Builds"
INSTALL_COMMAND
"${CMAKE_COMMAND}" --build "${foobar_dir}/Builds" --target install
)
Update
Note that it's much easier to patch parent project like this:
# FooBar/CMakeLists.txt
option(BUILD_BAR_SUBPROJECT "Build targets from subproject Bar" ON)
...
add_subdirectory(Foo)
if(BUILD_BAR_SUBPROJECT)
add_subdirectory(Bar)
endif()
... and now you don't need to hack ExternalProject_Add so much:
ExternalProject_Add(
FooBar
URL "your-url-here"
CMAKE_ARGS -DBUILD_BAR_SUBPROJECT=OFF "-DCMAKE_INSTALL_PREFIX=${EXTERNAL_INSTALL_DIR}"
)

Related

cmake not rebuilding a non-download external project after manually editing its sources

I'm working on some modifications to the openEMS project. This project uses cmake to build all of its components. The top level CMakeLists.txt file contains the following:
# ...
ExternalProject_Add( openEMS
DEPENDS fparser CSXCAD
SOURCE_DIR ${PROJECT_SOURCE_DIR}/openEMS
CMAKE_ARGS -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE} -DFPARSER_ROOT_DIR=${CMAKE_INSTALL_PREFIX} -DCSXCAD_ROOT_DIR=${CMAKE_INSTALL_PREFIX} -DWITH_MPI=${WITH_MPI} -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
)
# ...
Inside the openEMS directory, there's another CMakeLists.txt with the following:
# ...
set(SOURCES
openems.cpp
)
# ...
add_library( openEMS SHARED ${SOURCES})
# ...
After building the project successfully once, make does not rebuild anything when, for example, openems.cpp is modified. Why?
$ mkdir build
$ cd build
$ cmake -DBUILD_APPCSXCAD=NO
$ make
[builds all files]
$ touch ../openEMS/openems.cpp
$ make
[ 33%] Built target fparser
[ 66%] Built target CSXCAD
[100%] Built target openEMS
(noting is built)
I have checked and the modification date of openems.cpp is newer than the target. Even deleting the produced library files and binaries, both in the install directory and in the build directory, does not cause it to rebuild anything. The only way I can get it to rebuild is by deleting everything in the build directory and re-running cmake which, of course, rebuilds everything.
This looks like a case of the following. Quoting from the docs for ExternalProject_Add at the section titled "Build Step Options":
BUILD_ALWAYS <bool>
Enabling this option forces the build step to always be run. This can be the easiest way to robustly ensure that the external project's own build dependencies are evaluated rather than relying on the default success timestamp-based method. This option is not normally needed unless developers are expected to modify something the external project's build depends on in a way that is not detectable via the step target dependencies (e.g. SOURCE_DIR is used without a download method and developers might modify the sources in SOURCE_DIR).
If that's the case, the solution would be to add the BUILD_ALWAYS argument to the ExternalProject_Add call like.
ExternalProject_Add( openEMS
DEPENDS fparser CSXCAD
SOURCE_DIR ${PROJECT_SOURCE_DIR}/openEMS
CMAKE_ARGS -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE} -DFPARSER_ROOT_DIR=${CMAKE_INSTALL_PREFIX} -DCSXCAD_ROOT_DIR=${CMAKE_INSTALL_PREFIX} -DWITH_MPI=${WITH_MPI} -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
BUILD_ALWAYS TRUE
)
If you confirm that this solves the issue, you might want to raise this as an issue to the maintainers of openEMS.
Also note that since the external project there is using CMake as a buildsystem, you could also add the CONFIGURE_HANDLED_BY_BUILD TRUE to the argument list. See the docs for more info.
Edit: The asker opened a GitHub Pull-Request.

Using third-party libraries with CMake

I want anyone who cloned the repository can build it immediately, and don't need to install the dependencies.
Therefore, I found several ways:
Use git submodule and add_subdirectory.
Use find_package to find the built libraries and the headers.
The first one takes much time to build, so I think the second might be better. To make people be able to build the project instantly, I put the the files in the project, but it saied it doesn't know the linker language. What's this? And how to solve?
Direstories:
Project Root
lib
SDL2
(generated files when install)
include
(headers)
src
(sources)
CMakeLists.txt
CMakeLists.txt:
# ...
list(APPEND CMAKE_PREFIX_PATH lib)
find_package(SDL2)
# ...

In CMake, how would one run a custom command only if any subdirectory needed to be rebuild

I have a CMake-based project, and I need to run a command after build, if anything changed, including in any subprojects.
Simplified structure
project_root/
CMakeLists.txt (project root)
executable_1/
CMakeLists.txt (project tool1)
library_1/
CMakeLists.txt (project lib1)
executable_2/
CMakeLists.txt (project tool2)
library_2/
CMakeLists.txt (project lib2)
library_3/
CMakeLists.txt (project lib3)
The project root CMakeLists.txt looks like
cmake_minimum_required(VERSION 3.17)
add_subdirectory(tool1)
add_subdirectory(tool2)
add_subdirectory(lib3)
project(root C)
add_custom_target(root ALL
COMMENT
"root target"
DEPENDS
${DEPLOY_PATH}/deploy.tar.xz
)
add_custom_command(
OUTPUT
${DEPLOY_PATH}/deploy.tar.xz
COMMAND
tar cJf ${DEPLOY_PATH}/deploy.tar.xz *
COMMENT
"taring"
DEPENDS
root
)
The tool CMakeLists.txt look about like
cmake_minimum_required(VERSION 3.17)
add_subdirectory(lib1)
project(tool1 C)
add_executable(tool1
src/tool1.c
)
target_include_directories(tool1
PRIVATE
./
)
target_link_libraries(tool1
PRIVATE
lib1
lib3
)
install(TARGETS tool1 DESTINATION ${DEPLOY_PATH})
with the lib's CMakeLists.txt files omitting the install line and providing appropriate public include directories, etc.
I can get the behavior I want if I explicitly list every single subproject in the DEPENDS line of the custom target for "root", but that's tedious and error-prone. In the real project there are dozens of sub-libraries and executables.
I can also easily just re-build the archive every time, but that's slow.
How do I rebuild the archive if (and only if) any of the sub-projects of the root project or their descendants has changed?
List the actual dependencies of the tool.
file(GLOB_RECURSE srcs *) # I hope that's correct
add_custom_command(
OUTPUT
${DEPLOY_PATH}/deploy.tar.xz
COMMAND
# * generally is invalid, * depends on shell filename expansion
tar cJf ${DEPLOY_PATH}/deploy.tar.xz *
COMMENT "taring"
DEPENDS ${srcs}
)
But that's wrong anyway - cmake is meant to be run once and it scans your tree once. If you add a new file, you'll have to reconfigure cmake. Rather consider building the archive only when you actually need it - when deploying.

How do i use add_subdirectory() after ExternalProject_Add() has finished downloading?

I basically ask the same question as has been ask here. The question has however not been answered.
I want to use googletest in my project. For this I'm using ExternalProject_Add() which clones the testsuite with git. After that, I like to use add_subdirectory().
This is also what is described in the official repository. The nice thing about this approach is, that the build scripts in googletest handle the building process themself.
The problem is however, that add_subdirectory() can not find the source folder, since it does not exists from the start. Therefore, add_subdirectory() should depend on the completion of ExternalProject_Add().
Is is possible to make add_subdirectory() dependent of ExternalProject_Add(), like add_dependencies() does for targets?
PS. I can make it all compile if I comment add_subdirectory() out, build it (which ends with an error because the googletest library is missing), uncomment it and build it again (success).
ExternalProject_Add(
googletest
GIT_REPOSITORY https://github.com/google/googletest.git
GIT_TAG master
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
INSTALL_COMMAND ""
TEST_COMMAND ""
)
ExternalProject_Get_Property(googletest source_dir binary_dir)
set(GTEST_INCLUDE_DIR ${source_dir}/googletest/include)
set(GMOCK_INCLUDE_DIR ${source_dir}/googlemock/include)
add_subdirectory(${source_dir}
${binary_dir})
I used this tutorial to accomplish that. Just put your ExternalProject code in a separate file, say "CMakeLists.txt.dependencies" and then launch another cmake with execute_process. I use configure_file first to inject configuration information into the external project and to copy it into the build tree.
configure_file(CMakeLists.txt.dependency.in dependency/CMakeLists.txt)
execute_process(COMMAND "${CMAKE_COMMAND}" -G "${CMAKE_GENERATOR}" .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
execute_process(COMMAND "${CMAKE_COMMAND}" --build .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
Notice that the tutorial has the ExternalProject code in a separate CMakeLists.txt file. So does gtest official documentation. That is because ExternalProject runs at build-time, not configure time. This is a hack to use ExternalProject at configure time by putting it in a third project and executing a separate CMake+build run in the middle of your main CMake configuration step.
With version 3.11 CMake added a FetchContent command. I haven't used it, but the documentation makes it look like a configuration-time replacement for ExternalProject. That would fetch the content at configuration time, making it available for a later add_subdirectory command.
The ExternalProject and FetchContent commands will work fine for a small number of small and/or obscure dependencies. If your dependencies are more than 2, large, and/or popular, I recommend you look at a dependency manager like Conan. It partners with CMake quite well.
You are not following the method correctly. There should only be ExternalProject_Add() as the final command in the CMakeLists.txt.in file.
The CMakeLists.txt.in file is called by the outer CMakeLists.txt file so that the subproject processing happens at configure time via execute_process(). CMakeLists.txt.in is acting as just a glorified downloader.
So, all other commands like add_subdirectory() are added to the outer CMakeLists.txt file.

Is there a way to download the tarball once using cmake

Im build libjpeg as external project. Its build normally.
Here projects folder structure:
${SOURCE_DIR}/
${SOME_BUILD_DIR}/
externals/
jpeg-9a/
jpeg-pre/
externals/
jpeg-9a/
jpeg-overlay/
CMakeLists.txt
tarballs/
jpegsrc.v9a.tar.gz
CMakeLists.txt
But tarball downloaded and unpacked again if I start building project from zero.
In other words I clean ${SOME_BUILD_DIR}/. At next build cmake do the next:
download tarballs/jpegsrc.v9a.tar.gz
unpack libjpeg into externals/jpeg-9a
copy externals/jpeg-overlay/CMakeLists.txt into externals/jpeg-9a
build libjpeg in ${SOME_BUILD_DIR}/externals/jpeg-9a/
Actually first 3 points can be omitted. But my interest only in first action. Is there way to prevent extra download?
Here is my ${SOURCE_DIR}/CMakeLists.txt:
cmake_minimum_required(VERSION 2.8)
include(ExternalProject)
set(EXTERNALS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/externals)
set(JPEG_VERSION "9a")
set(JPEG_URI http://ijg.org/files/jpegsrc.v${JPEG_VERSION}.tar.gz)
set(JPEG_DIR ${EXTERNALS_DIR}/jpeg-${JPEG_VERSION})
ExternalProject_Add(
jpeg
STAMP_DIR ${CMAKE_BINARY_DIR}/externals/jpeg-pre
BINARY_DIR ${CMAKE_BINARY_DIR}/externals/jpeg-${JPEG_VERSION}
URL ${JPEG_URI}
SOURCE_DIR ${JPEG_DIR}
DOWNLOAD_DIR ${CMAKE_CURRENT_SOURCE_DIR}/tarballs
CMAKE_ARGS ""
UPDATE_COMMAND cmake -E copy_directory ${EXTERNALS_DIR}/jpeg-${JPEG_VERSION}-overlay/. ${JPEG_DIR}
INSTALL_COMMAND ""
TEST_COMMAND ""
)
See https://github.com/anton-sergeev/cmake_externalproject for details.
You should not place generated files in the source tree.
In your case, the problematic line is DOWNLOAD_DIR ${CMAKE_CURRENT_SOURCE_DIR}/tarballs. A cleaner approach would be to place the download in a directory in the binary dir and compile from there. Now, of course that means that when wiping the build directory it will also wipe the downloaded sources. Which kind of solves the problem of having to download a file that you already had, although probably not in the way you would have liked.
The thing is, this is by design. Wiping the binary dir is conceptually equivalent to telling CMake to start over from scratch. There is no point in trying to reuse stuff in this case because, well, you want to start from scratch. The correct workflow to enable reusing with CMake is simply: Do not wipe the build directory. Instead rely on make clean to enforce full rebuilds and only wipe the build directory if you want to perform a full reconfigure of CMake.
The only clean way to avoid redownload is to move the download out of the ExternalProject command. For instance, you could place the extracted files into the source tree and check them in as part of the project. Or have them downloaded by a custom execute_process command which implements the desired behavior.
If you run md5 jpegsrc.v9a.tar.gz, you can use the outputted hash with the ExternalProject URL_MD5 option.
ExternalProject_Add(
jpeg
STAMP_DIR ${CMAKE_BINARY_DIR}/externals/jpeg-pre
BINARY_DIR ${CMAKE_BINARY_DIR}/externals/jpeg-${JPEG_VERSION}
URL ${JPEG_URI}
URL_MD5 <md5_hash_of_downloaded_file>
SOURCE_DIR ${JPEG_DIR}
DOWNLOAD_DIR ${CMAKE_CURRENT_SOURCE_DIR}/tarballs
CMAKE_ARGS ""
UPDATE_COMMAND cmake -E copy_directory ${EXTERNALS_DIR}/jpeg-${JPEG_VERSION}-overlay/. ${JPEG_DIR}
INSTALL_COMMAND ""
TEST_COMMAND ""
)