I am trying a very simple ExternalProject usage against ITK. This will allow my automated jenkins slave to retrieve ITK directly instead of using a system installed library (thus I leave it as an option to use ExternalProject or not).
So I wrote the following piece of code:
set(ITK_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/ITK")
set(ITK_INSTALL_PREFIX "${ITK_PREFIX}/install-$<CONFIG>")
ExternalProject_Add(ITK
URL http://sourceforge.net/projects/itk/files/itk/4.6/InsightToolkit-4.6.1.tar.xz
URL_MD5 d8dcab9193b55d4505afa94ab46de699
PREFIX ${ITK_PREFIX}
CMAKE_ARGS -DBUILD_SHARED_LIBS:BOOL=OFF -DBUILD_EXAMPLES:BOOL=OFF -DBUILD_TESTING:BOOL=OFF -DModule_ITKReview:BOOL=ON -DITK_USE_SYSTEM_GDCM:BOOL=ON -DCMAKE_INSTALL_PREFIX=${ITK_INSTALL_PREFIX} -DGDCM_DIR:PATH=${GDCM_INSTALL_PREFIX}
BUILD_COMMAND "${CMAKE_COMMAND}" --build . --target install --config $<CONFIG>
)
# include directory:
include_directories(${ITK_INSTALL_PREFIX}/include/ITK-4.6)
# link directory:
#link_directories(${ITK_INSTALL_PREFIX}/lib/) # $ sign is escaped
link_directories(${ITK_PREFIX}/install-/lib)
But then I fail to understand how I can possibly populate the following variable: ITK_LIBRARIES which I had been using throughout my codebase.
How should I write:
set(ITK_LIBRARIES
itksys-4.6
ITKCommon-4.6
ITKIOImageBase-4.6
ITKIOMeta-4.6
ITKIOGDCM-4.6
pthread
...? possibly others ? possibly different order ? ...
)
This feels like a hack, extremely hard to maintain, esp. considering that I need to link to static libraries (requirements for me).
Obviously the magical solution would be for me to run find_package(ITK) and be done. But since ExternalProject are done at build time and not configure time, I cannot make use of this (ref).
Because people feel it is duplicate, let me insist, on: "Yes I do understand that I cannot use find_package". My question is totally different, and is rather about the complex case of static linking.
So I should not be building the ordered list of static libraries in ITK_LIBRARIES, this is too complex. Instead I should be using the logic from a call to find_package(ITK).
I need to change the way I build my project and switch to a SuperBuild type solution.
Related
I'm trying to use add_custom_command to generate a file during the build. The command never seemed to be run, so I made this test file.
cmake_minimum_required( VERSION 2.6 )
add_custom_command(
OUTPUT hello.txt
COMMAND touch hello.txt
DEPENDS hello.txt
)
I tried running:
cmake .
make
And hello.txt was not generated. What have I done wrong?
The add_custom_target(run ALL ... solution will work for simple cases when you only have one target you're building, but breaks down when you have multiple top level targets, e.g. app and tests.
I ran into this same problem when I was trying to package up some test data files into an object file so my unit tests wouldn't depend on anything external. I solved it using add_custom_command and some additional dependency magic with set_property.
add_custom_command(
OUTPUT testData.cpp
COMMAND reswrap
ARGS testData.src > testData.cpp
DEPENDS testData.src
)
set_property(SOURCE unit-tests.cpp APPEND PROPERTY OBJECT_DEPENDS testData.cpp)
add_executable(app main.cpp)
add_executable(tests unit-tests.cpp)
So now testData.cpp will generated before unit-tests.cpp is compiled, and any time testData.src changes. If the command you're calling is really slow you get the added bonus that when you build just the app target you won't have to wait around for that command (which only the tests executable needs) to finish.
It's not shown above, but careful application of ${PROJECT_BINARY_DIR}, ${PROJECT_SOURCE_DIR} and include_directories() will keep your source tree clean of generated files.
Add the following:
add_custom_target(run ALL
DEPENDS hello.txt)
If you're familiar with makefiles, this means:
all: run
run: hello.txt
The problem with two existing answers is that they either make the dependency global (add_custom_target(name ALL ...)), or they assign it to a specific, single file (set_property(...)) which gets obnoxious if you have many files that need it as a dependency. Instead what we want is a target that we can make a dependency of another target.
The way to do this is to use add_custom_command to define the rule, and then add_custom_target to define a new target based on that rule. Then you can add that target as a dependency of another target via add_dependencies.
# this defines the build rule for some_file
add_custom_command(
OUTPUT some_file
COMMAND ...
)
# create a target that includes some_file, this gives us a name that we can use later
add_custom_target(
some_target
DEPENDS some_file
)
# then let's suppose we're creating a library
add_library(some_library some_other_file.c)
# we can add the target as a dependency, and it will affect only this library
add_dependencies(some_library some_target)
The advantages of this approach:
some_target is not a dependency for ALL, which means you only build it when it's required by a specific target. (Whereas add_custom_target(name ALL ...) would build it unconditionally for all targets.)
Because some_target is a dependency for the library as a whole, it will get built before all of the files in that library. That means that if there are many files in the library, we don't have to do set_property on every single one of them.
If we add DEPENDS to add_custom_command then it will only get rebuilt when its inputs change. (Compare this to the approach that uses add_custom_target(name ALL ...) where the command gets run on every build regardless of whether it needs to or not.)
For more information on why things work this way, see this blog post: https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/
This question is pretty old, but even if I follow the suggested recommendations, it does not work for me (at least not every time).
I am using Android Studio and I need to call cMake to build C++ library. It works fine until I add the code to run my custom script (in fact, at the moment I try to run 'touch', as in the example above).
First of,
add_custom_command
does not work at all.
I tried
execute_process (
COMMAND touch hello.txt
)
it works, but not every time!
I tried to clean the project, remove the created file(s) manually, same thing.
Tried cMake versions:
3.10.2
3.18.1
3.22.1
when they work, they produce different results, depending on cMake version, one file or several. This is not that important as long as they work, but that's the issue.
Can somebody shed light on this mystery?
I'm trying to use add_custom_command to generate a file during the build. The command never seemed to be run, so I made this test file.
cmake_minimum_required( VERSION 2.6 )
add_custom_command(
OUTPUT hello.txt
COMMAND touch hello.txt
DEPENDS hello.txt
)
I tried running:
cmake .
make
And hello.txt was not generated. What have I done wrong?
The add_custom_target(run ALL ... solution will work for simple cases when you only have one target you're building, but breaks down when you have multiple top level targets, e.g. app and tests.
I ran into this same problem when I was trying to package up some test data files into an object file so my unit tests wouldn't depend on anything external. I solved it using add_custom_command and some additional dependency magic with set_property.
add_custom_command(
OUTPUT testData.cpp
COMMAND reswrap
ARGS testData.src > testData.cpp
DEPENDS testData.src
)
set_property(SOURCE unit-tests.cpp APPEND PROPERTY OBJECT_DEPENDS testData.cpp)
add_executable(app main.cpp)
add_executable(tests unit-tests.cpp)
So now testData.cpp will generated before unit-tests.cpp is compiled, and any time testData.src changes. If the command you're calling is really slow you get the added bonus that when you build just the app target you won't have to wait around for that command (which only the tests executable needs) to finish.
It's not shown above, but careful application of ${PROJECT_BINARY_DIR}, ${PROJECT_SOURCE_DIR} and include_directories() will keep your source tree clean of generated files.
Add the following:
add_custom_target(run ALL
DEPENDS hello.txt)
If you're familiar with makefiles, this means:
all: run
run: hello.txt
The problem with two existing answers is that they either make the dependency global (add_custom_target(name ALL ...)), or they assign it to a specific, single file (set_property(...)) which gets obnoxious if you have many files that need it as a dependency. Instead what we want is a target that we can make a dependency of another target.
The way to do this is to use add_custom_command to define the rule, and then add_custom_target to define a new target based on that rule. Then you can add that target as a dependency of another target via add_dependencies.
# this defines the build rule for some_file
add_custom_command(
OUTPUT some_file
COMMAND ...
)
# create a target that includes some_file, this gives us a name that we can use later
add_custom_target(
some_target
DEPENDS some_file
)
# then let's suppose we're creating a library
add_library(some_library some_other_file.c)
# we can add the target as a dependency, and it will affect only this library
add_dependencies(some_library some_target)
The advantages of this approach:
some_target is not a dependency for ALL, which means you only build it when it's required by a specific target. (Whereas add_custom_target(name ALL ...) would build it unconditionally for all targets.)
Because some_target is a dependency for the library as a whole, it will get built before all of the files in that library. That means that if there are many files in the library, we don't have to do set_property on every single one of them.
If we add DEPENDS to add_custom_command then it will only get rebuilt when its inputs change. (Compare this to the approach that uses add_custom_target(name ALL ...) where the command gets run on every build regardless of whether it needs to or not.)
For more information on why things work this way, see this blog post: https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/
This question is pretty old, but even if I follow the suggested recommendations, it does not work for me (at least not every time).
I am using Android Studio and I need to call cMake to build C++ library. It works fine until I add the code to run my custom script (in fact, at the moment I try to run 'touch', as in the example above).
First of,
add_custom_command
does not work at all.
I tried
execute_process (
COMMAND touch hello.txt
)
it works, but not every time!
I tried to clean the project, remove the created file(s) manually, same thing.
Tried cMake versions:
3.10.2
3.18.1
3.22.1
when they work, they produce different results, depending on cMake version, one file or several. This is not that important as long as they work, but that's the issue.
Can somebody shed light on this mystery?
I'm trying to use add_custom_command to generate a file during the build. The command never seemed to be run, so I made this test file.
cmake_minimum_required( VERSION 2.6 )
add_custom_command(
OUTPUT hello.txt
COMMAND touch hello.txt
DEPENDS hello.txt
)
I tried running:
cmake .
make
And hello.txt was not generated. What have I done wrong?
The add_custom_target(run ALL ... solution will work for simple cases when you only have one target you're building, but breaks down when you have multiple top level targets, e.g. app and tests.
I ran into this same problem when I was trying to package up some test data files into an object file so my unit tests wouldn't depend on anything external. I solved it using add_custom_command and some additional dependency magic with set_property.
add_custom_command(
OUTPUT testData.cpp
COMMAND reswrap
ARGS testData.src > testData.cpp
DEPENDS testData.src
)
set_property(SOURCE unit-tests.cpp APPEND PROPERTY OBJECT_DEPENDS testData.cpp)
add_executable(app main.cpp)
add_executable(tests unit-tests.cpp)
So now testData.cpp will generated before unit-tests.cpp is compiled, and any time testData.src changes. If the command you're calling is really slow you get the added bonus that when you build just the app target you won't have to wait around for that command (which only the tests executable needs) to finish.
It's not shown above, but careful application of ${PROJECT_BINARY_DIR}, ${PROJECT_SOURCE_DIR} and include_directories() will keep your source tree clean of generated files.
Add the following:
add_custom_target(run ALL
DEPENDS hello.txt)
If you're familiar with makefiles, this means:
all: run
run: hello.txt
The problem with two existing answers is that they either make the dependency global (add_custom_target(name ALL ...)), or they assign it to a specific, single file (set_property(...)) which gets obnoxious if you have many files that need it as a dependency. Instead what we want is a target that we can make a dependency of another target.
The way to do this is to use add_custom_command to define the rule, and then add_custom_target to define a new target based on that rule. Then you can add that target as a dependency of another target via add_dependencies.
# this defines the build rule for some_file
add_custom_command(
OUTPUT some_file
COMMAND ...
)
# create a target that includes some_file, this gives us a name that we can use later
add_custom_target(
some_target
DEPENDS some_file
)
# then let's suppose we're creating a library
add_library(some_library some_other_file.c)
# we can add the target as a dependency, and it will affect only this library
add_dependencies(some_library some_target)
The advantages of this approach:
some_target is not a dependency for ALL, which means you only build it when it's required by a specific target. (Whereas add_custom_target(name ALL ...) would build it unconditionally for all targets.)
Because some_target is a dependency for the library as a whole, it will get built before all of the files in that library. That means that if there are many files in the library, we don't have to do set_property on every single one of them.
If we add DEPENDS to add_custom_command then it will only get rebuilt when its inputs change. (Compare this to the approach that uses add_custom_target(name ALL ...) where the command gets run on every build regardless of whether it needs to or not.)
For more information on why things work this way, see this blog post: https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/
This question is pretty old, but even if I follow the suggested recommendations, it does not work for me (at least not every time).
I am using Android Studio and I need to call cMake to build C++ library. It works fine until I add the code to run my custom script (in fact, at the moment I try to run 'touch', as in the example above).
First of,
add_custom_command
does not work at all.
I tried
execute_process (
COMMAND touch hello.txt
)
it works, but not every time!
I tried to clean the project, remove the created file(s) manually, same thing.
Tried cMake versions:
3.10.2
3.18.1
3.22.1
when they work, they produce different results, depending on cMake version, one file or several. This is not that important as long as they work, but that's the issue.
Can somebody shed light on this mystery?
I have an interesting chicken-and-egg problem and a potential solution to it (see my posted answer), but that solution uses CMake in an unusual way. Better alternatives or comments would be welcome.
THE PROBLEM:
The simple version of the problem can be described as a single CMake project with the following characteristics:
One of the build targets is a command-line executable which I'll call mycomp, the source of which is in a mycompdir and making any modifications to the contents of that directory is not possible.
The project contains text files (I'll call them foo.my and bar.my) which need mycomp run on them to produce a set of C++ sources and headers and some CMakeLists.txt files defining libraries built from those sources.
Other build targets in the same project need to link against the libraries defined by those generated CMakeLists.txt files. These other targets also have sources which #include some of the generated headers.
You can think of mycomp as being something like a compiler and the text files in step 2 as some sort of source files. This presents a problem, because CMake needs the CMakeLists.txt files at configure time, but mycomp is not available until build time and therefore isn't available on the first run to create the CMakeLists.txt files early enough.
NON-ANSWER:
Normally, an ExternalProject-based superbuild arrangement would be a potential solution to this, but the above is a considerable simplification of the actual project I am dealing with and I don't have the freedom to split the build into different parts or perform other large scale restructuring work.
The crux of the problem is needing mycomp to be available when CMake is run so that the generated CMakeLists.txt files can be created and then pulled in with add_subdirectory(). A possible way to achieve this is to use execute_process() to run a nested cmake-and-build from the main build. That nested cmake-and-build would use the exact same source and binary directories as the top level CMake run (unless cross compiling). The general structure of the main top level CMakeLists.txt would be something like this:
# Usual CMakeLists.txt setup stuff goes here...
if(EARLY_BUILD)
# This is the nested build and we will only be asked to
# build the mycomp target (see (c) below)
add_subdirectory(mycompdir)
# End immediately, we don't want anything else in the nested build
return()
endif()
# This is the main build, setup and execute the nested build
# to ensure the mycomp executable exists before continuing
# (a) When cross compiling, we cannot re-use the same binary dir
# because the host and target are different architectures
if(CMAKE_CROSSCOMPILING)
set(workdir "${CMAKE_BINARY_DIR}/host")
execute_process(COMMAND ${CMAKE_COMMAND} -E make_directory "${workdir}")
else()
set(workdir "${CMAKE_BINARY_DIR}")
endif()
# (b) Nested CMake run. May need more -D... options than shown here.
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}"
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
-DCMAKE_MAKE_PROGRAM=${CMAKE_MAKE_PROGRAM}
-DEARLY_BUILD=ON
${CMAKE_SOURCE_DIR}
WORKING_DIRECTORY "${workdir}")
# (c) Build just mycomp in the nested build. Don't specify a --config
# because we cannot know what config the developer will be using
# at this point. For non-multi-config generators, we've already
# specified CMAKE_BUILD_TYPE above in (b).
execute_process(COMMAND ${CMAKE_COMMAND} --build . --target mycomp
WORKING_DIRECTORY "${workdir}")
# (d) We want everything from mycompdir in our main build,
# not just the mycomp target
add_subdirectory(mycompdir)
# (e) Run mycomp on the sources to generate a CMakeLists.txt in the
# ${CMAKE_BINARY_DIR}/foobar directory. Note that because we want
# to support cross compiling, working out the location of the
# executable is a bit more tricky. We cannot know whether the user
# wants debug or release build types for multi-config generators
# so we have to choose one. We cannot query the target properties
# because they are only known at generate time, which is after here.
# Best we can do is hardcode some basic logic.
if(MSVC)
set(mycompsuffix "Debug/mycomp.exe")
elseif(CMAKE_GENERATOR STREQUAL "Xcode")
set(mycompsuffix "Debug/mycomp")
else()
set(mycompsuffix "mycomp")
endif()
set(mycomp_EXECUTABLE "${workdir}/mycompdir/${mycompsuffix}")
execute_process(COMMAND "${mycomp_EXECUTABLE}" -outdir foobar ${CMAKE_SOURCE_DIR}/foo.my ${CMAKE_SOURCE_DIR}/bar.my)
# (f) Now pull that generated CMakeLists.txt into the main build.
# It will create a CMake library target called foobar.
add_subdirectory(${CMAKE_BINARY_DIR}/foobar ${CMAKE_BINARY_DIR}/foobar-build)
# (g) Another target which links to the foobar library
# and includes headers from there
add_executable(gumby gumby.cpp)
target_link_libraries(gumby PUBLIC foobar)
target_include_directories(gumby PUBLIC foobar)
If we don't re-use the same binary directory at (b) and (c) as we use for the main build, we end up building mycomp twice, which we obviously want to avoid. For cross compiling, we cannot avoid that, so in such cases we build the mycomp tool off to the side in a separate binary directory.
I've experimented with the above approach and indeed it appears to work in the real world project that prompted the original question, at least for the Unix Makefiles, Ninja, Xcode (OS X and iOS) and Visual Studio generators. Part of the attractiveness of this approach is that it only requires a modest amount of code to be added just to the top level CMakeLists.txt file. Nevertheless, there are some observations that should be made:
If the compiler or linker commands for mycomp and its sources are different in any way between the nested build and the main build, the mycomp target ends up getting rebuilt a second time at (d). If there are no differences, mycomp only gets built once when not cross compiling, which is exactly what we want.
I see no easy way to pass exactly the same arguments to the nested invocation of CMake at (b) as was passed to the top level CMake run (basically the problem described here). Reading CMakeCache.txt isn't an option since it won't exist on the first invocation and it would not give you any new or changed arguments from the current run anyway. The best I can do is to set those CMake variables I think are potentially going to be used and which may influence the compiler and linker commands of mycomp. This can be worked around by adding more and more variables as I encounter ones I discover I need, but that's not ideal.
When re-using the same binary directory, we are relying on CMake not starting to write any of its files to the binary directory until the generate stage (well, at least until after the build at (c) completes). For the generators tested, it appears we are okay, but I don't know if all generators on all platforms follow this behaviour too (and I can't test every single combination to find out!). This is the part that gives me the greatest concern. If anyone can confirm with reasoning and/or evidence that this is safe for all generators and platforms, that would be valuable (and worth an upvote if you want to address this as a separate answer).
UPDATE: After using the above strategy on a number of real world projects with staff of varying levels of familiarity with CMake, some observations can be made.
Having the nested build re-use the same build directory as the main build can occasionally lead to problems. Specifically, if a user kills the CMake run after the nested build completes but before the main build does, the CMakeCache.txt file is left with EARLY_BUILD set to ON. This then makes all subsequent CMake runs act like a nested build, so the main build is essentially lost until the CMakeCache.txt file is manually removed. It is possible that an error somewhere in one of the project's CMakeLists.txt file may also lead to a similar situation (unconfirmed). Performing the nested build off to the side in its own separate build directory has worked very well though with no such problems.
The nested build should probably be Release rather than Debug. If not re-using the same build directory as the main build (now what I'd recommend), we no longer care about trying to avoid compiling the same file twice, so may as well make mycomp as fast as possible.
Use ccache so that any costs due to rebuilding some files twice with different settings are minimised. Actually, we found using ccache typically makes the nested build very quick since it rarely changed compared to the main build.
The nested build probably needs to have CMAKE_BUILD_WITH_INSTALL_RPATH set to FALSE on some platforms so that any libraries mycomp needs can be found without having to set environment variables, etc.
I'm trying to use add_custom_command to generate a file during the build. The command never seemed to be run, so I made this test file.
cmake_minimum_required( VERSION 2.6 )
add_custom_command(
OUTPUT hello.txt
COMMAND touch hello.txt
DEPENDS hello.txt
)
I tried running:
cmake .
make
And hello.txt was not generated. What have I done wrong?
The add_custom_target(run ALL ... solution will work for simple cases when you only have one target you're building, but breaks down when you have multiple top level targets, e.g. app and tests.
I ran into this same problem when I was trying to package up some test data files into an object file so my unit tests wouldn't depend on anything external. I solved it using add_custom_command and some additional dependency magic with set_property.
add_custom_command(
OUTPUT testData.cpp
COMMAND reswrap
ARGS testData.src > testData.cpp
DEPENDS testData.src
)
set_property(SOURCE unit-tests.cpp APPEND PROPERTY OBJECT_DEPENDS testData.cpp)
add_executable(app main.cpp)
add_executable(tests unit-tests.cpp)
So now testData.cpp will generated before unit-tests.cpp is compiled, and any time testData.src changes. If the command you're calling is really slow you get the added bonus that when you build just the app target you won't have to wait around for that command (which only the tests executable needs) to finish.
It's not shown above, but careful application of ${PROJECT_BINARY_DIR}, ${PROJECT_SOURCE_DIR} and include_directories() will keep your source tree clean of generated files.
Add the following:
add_custom_target(run ALL
DEPENDS hello.txt)
If you're familiar with makefiles, this means:
all: run
run: hello.txt
The problem with two existing answers is that they either make the dependency global (add_custom_target(name ALL ...)), or they assign it to a specific, single file (set_property(...)) which gets obnoxious if you have many files that need it as a dependency. Instead what we want is a target that we can make a dependency of another target.
The way to do this is to use add_custom_command to define the rule, and then add_custom_target to define a new target based on that rule. Then you can add that target as a dependency of another target via add_dependencies.
# this defines the build rule for some_file
add_custom_command(
OUTPUT some_file
COMMAND ...
)
# create a target that includes some_file, this gives us a name that we can use later
add_custom_target(
some_target
DEPENDS some_file
)
# then let's suppose we're creating a library
add_library(some_library some_other_file.c)
# we can add the target as a dependency, and it will affect only this library
add_dependencies(some_library some_target)
The advantages of this approach:
some_target is not a dependency for ALL, which means you only build it when it's required by a specific target. (Whereas add_custom_target(name ALL ...) would build it unconditionally for all targets.)
Because some_target is a dependency for the library as a whole, it will get built before all of the files in that library. That means that if there are many files in the library, we don't have to do set_property on every single one of them.
If we add DEPENDS to add_custom_command then it will only get rebuilt when its inputs change. (Compare this to the approach that uses add_custom_target(name ALL ...) where the command gets run on every build regardless of whether it needs to or not.)
For more information on why things work this way, see this blog post: https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/
This question is pretty old, but even if I follow the suggested recommendations, it does not work for me (at least not every time).
I am using Android Studio and I need to call cMake to build C++ library. It works fine until I add the code to run my custom script (in fact, at the moment I try to run 'touch', as in the example above).
First of,
add_custom_command
does not work at all.
I tried
execute_process (
COMMAND touch hello.txt
)
it works, but not every time!
I tried to clean the project, remove the created file(s) manually, same thing.
Tried cMake versions:
3.10.2
3.18.1
3.22.1
when they work, they produce different results, depending on cMake version, one file or several. This is not that important as long as they work, but that's the issue.
Can somebody shed light on this mystery?