I have a third party library with autotools project. I want to use ExternalProject_Add to build the library.
This can be done the following way:
ExternalProject_Add(project_lib
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/lib
CONFIGURE_COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/lib/configure --prefix=${LIB_OUTPUT}
BINARY_DIR ${CMAKE_CURRENT_SOURCE_DIR}/build
)
Now the problem is that I need to pass an environment variable to configure but I cannot find a way to do it.
In the console I would do it the following way:
CPPFLAGS="-fPIC" ./configure --prefix=output
Is there a way to pass CPPFLAGS="-fPIC" env to the configure with ExternalProject_Add/CONFIGURE_COMMAND ?
It can be done by executing configure command through cmake (cmake --help) command mode:
-E = CMake command mode.
So instead of calling configure directly we can execute it through cmake command mode with the environment variables:
ExternalProject_Add(project_lib
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/lib
CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env CPPFLAGS=-fPIC ${CMAKE_CURRENT_SOURCE_DIR}/lib/configure --prefix=${LIB_OUTPUT}
BINARY_DIR ${CMAKE_CURRENT_SOURCE_DIR}/build
)
[cmake-developers] Setting up environment using ExternalProject_Add
Related
My project contains a custom target which generates some output via .cmake script. It looks like this:
add_custom_target(TargetName
COMMAND ${CMAKE_COMMAND} -P ${CMAKE_SOURCE_DIR}/cmake/script.cmake
BYPRODUCTS ${CMAKE_CURRENT_BINARY_DIR}/generated/output
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
VERBATIM
)
But now I want to set come cache variables inside the script. I tried doing like that:
message("MY_CACHE_VARIABLE = ${MY_CACHE_VARIABLE}")
set(MY_CACHE_VARIABLE "VALUE" CACHE INTERNAL "")
And I faced with the problem that cache variables are not saved. It always prints me empty output:
MY_CACHE_VARIABLE =
I already tried setting working directory as CMAKE_BINARY_DIR, or passing CMAKE_BINARY_DIR of the last argument of cmake command, or passing -B ${CMAKE_BINARY_DIR} or -C ${CMAKE_BINARY_DIR}/CMakeCache.txt as arguments and etc. None of these worked.
So is there any way to reuse existing cache inside CMake subprocess or I just should write my own cache inside the script?
You have to distinguish between running CMake to generate build files (for Make, Ninja, etc.) and running CMake in script mode:
Script mode simply runs the commands in the given CMake Language source file and does not generate a build system. It does not allow CMake commands that define build targets or actions.
-- cmake-language(7)
No configure or generate step is performed and the cache is not modified.
-- cmake(1)
So in script mode (-P), CMake is not aware of the cache or any variable/target/etc. defined in your regular CMakeLists.txt files. It is more similar to executing a bash/shell script than to processing a "usual" CMakeLists.txt.
But don't worry, there is still a solution to your problem. You can simply pass your arguments as -D options to your script:
add_custom_target(TargetName
COMMAND ${CMAKE_COMMAND}
-DMY_VAR="..."
-DANOTHER_VAR="..."
-P ${CMAKE_SOURCE_DIR}/cmake/script.cmake
...
)
Note however:
If variables are defined using -D, this must be done before the -P argument.
-- cmake(1)
I am calling make in a cmake custom command (I have to bridge between a CMake build and a legacy make build). Everything works fine, except one thing: Instead of seeing the output of the make build live on stdout, it only gets flushed after the build finishes. I have searched online but couldn't find a solution so far. What could the cause for such a behavior be?
I tested this modified simple example which recursively calls itself. This does behave exactly how I expect. I see the echos from the recursive calls as they happen. I can't figure out what the difference to my actual code is:
include(ExternalProject)
ExternalProject_Add(Wrapper
PREFIX "Wrapper"
DOWNLOAD_COMMAND ""
CONFIGURE_COMMAND ""
BUILD_COMMAND ${CMAKE_COMMAND} -E echo hello
COMMAND ${CMAKE_COMMAND} -E sleep 5
COMMAND ${CMAKE_COMMAND} -E env FOO=BAR ${CMAKE_MAKE_PROGRAM} -C "${CMAKE_CURRENT_SOURCE_DIR}"
BUILD_ALWAYS ON
INSTALL_COMMAND ""
)
My actual code, which only shows the output after the external make build is completely finished:
ExternalProject_Add(Wrapper
PREFIX "Wrapper"
DOWNLOAD_COMMAND ""
CONFIGURE_COMMAND ""
BUILD_COMMAND ${CMAKE_COMMAND} -E copy ${target_files} ${CMAKE_INSTALL_PREFIX}
COMMAND ${CMAKE_COMMAND} -E env ASM_OPT="${CMAKE_ASM_FLAGS}" CC_OPT="${CMAKE_C_FLAGS}" CPP_OPT="${CMAKE_CXX_FLAGS}" ${CMAKE_MAKE_PROGRAM} -C "${CMAKE_CURRENT_SOURCE_DIR}/.." -f Makefile.mak -O -j 4
BUILD_ALWAYS ON
INSTALL_COMMAND ""
DEPENDS ${ALL_LIBS}
)
Bad news: In the case of other than Unix Makefiles generator (e.g., Ninja, the CMAKE_MAKE_PROGRAM variable gonna have a path to underlaid build system tool (e.g., /usr/bin/ninja for Ninja generator :)
A better way is to use ExternalProject to build your non-CMake (third party) project.
Or, since CMake 3.18 execute_process got the ECHO_OUTPUT_VARIABLE named keyword (option) which is equal to *nix tee command...
This seems to be a common problem without a clear answer.
The situation is: we have a 3rd party dependency that we want to install at build time when building a target that depends on it. That's roughly:
ExternalProject_Add(target-ep
DOWNLOAD_COMMAND <whatever>
BUILD_COMMAND ""
INSTALL_COMMAND ""
CONFIGURE_COMMAND "")
add_library(target-imp STATIC IMPORTED)
set_target_properties(target-imp PROPERTIES
INTERFACE_INCLUDE_DIRECTORIES /path/to/install/include
IMPORTED_LOCATION /path/to/install/lib/libwhatever.a)
add_library(target INTERFACE)
target_link_libraries(target INTERFACE target-imp)
add_dependencies(target target-ep)
(It takes three to tango here because of cmake issue 15052)
When using Unix Makefiles as the generator, this works great. Only installs dependencies on demand, all the builds work correctly.
However, on Ninja, this fails immediately with something like:
ninja: error: '/path/to/install/lib/libwhatever.a', needed by 'something', missing and no known rule to make it
This is because Ninja scans dependencies differently from Make (see ninja issue 760). So what we have to do is actually tell Ninja that this external dependency exists. We can do that:
ExternalProject_Add(target-ep
DOWNLOAD_COMMAND <whatever>
BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
BUILD_COMMAND ""
INSTALL_COMMAND ""
CONFIGURE_COMMAND "")
Which unfortunately also fails with:
No build step for 'target-ep'ninja: error: mkdir(/path/to/install): Permission denied
This is because my download step has permissions to write to that path, but whatever mkdir command is being run by the underlying add_custom_command() from with ExternalProject_Add() does not.
So:
Is this possible at all with Ninja and CMake? (Version is not an issue, I can use the latest CMake if that solves the problem)
If there is some way to workaround with explicitly listing BUILD_BYPRODUCTS, is there a way to simply communicate that the entire directory that will get installed is a byproduct? That is, /path/to/install/* is a byproduct?
The hidden mkdir step of ExternalProject (which all other steps directly or indirectly depend on) always tries to create the full set of directories, even if they won't be used. You can see this here. For reference, it does this:
ExternalProject_Add_Step(${name} mkdir
COMMENT "Creating directories for '${name}'"
COMMAND ${CMAKE_COMMAND} -E make_directory ${source_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${binary_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${install_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${tmp_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${stamp_dir}${cfgdir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${download_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${log_dir} # This one only since CMake 3.13
)
The default install location on Unix systems is probably going to be /usr/local, so if you don't have write permissions to all of the directories it tries to make, then that may be related to your problem. I suggest you check the permissions of each of these locations and make sure they either already exist or are writable. Alternatively, you could specify an install directory that is local to the build tree so that even though it won't be used, it can at least always be created (see example further below).
If you use Ninja, it will be more rigorous in its dependency checking than make. You have target-ep doing the download that provides libwhatever.a, so you do need BUILD_BYPRODUCTS to tell Ninja that target-ep is what creates that file. As you've found out, if you don't then target-imp will point at a library that won't initially exist and Ninja rightly complains that it is missing and it doesn't know how to create it. If you provide BUILD_BYPRODUCTS, it makes sense that the build step shouldn't be empty, so you probably need to do something as a build step, even if it is just a BUILD_COMMAND that doesn't actually do anything meaningful.
The following modified definition of target-ep should hopefully get things working for you:
ExternalProject_Add(target-ep
INSTALL_DIR ${CMAKE_CURRENT_BUILD_DIR}/dummyInstall
DOWNLOAD_COMMAND <whatever>
BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
BUILD_COMMAND ${CMAKE_COMMAND} -E echo_append
INSTALL_COMMAND ""
CONFIGURE_COMMAND "")
Your original question also creates a dependency on the wrong target. target-imp should depend on target-ep, but you had target depend on target-ep instead. The correct dependency can be expressed by this:
add_dependencies(target-imp target-ep)
With the BUILD_BYPRODUCTS option, Ninja already knows the above dependency, but it is needed for other generators, including make.
You haven't specified what your <whatever> download command does, but I'm assuming it is responsible for ensuring that the library will exist at /path/to/install/lib/libwhatever.a when it has executed. You could also try making the DOWNLOAD_COMMAND empty and putting <whatever> as the BUILD_COMMAND instead.
To address your specific questions:
Is this possible at all with Ninja and CMake? (Version is not an issue, I can use the latest CMake if that solves the problem)
Yes, I verified that the approach mentioned above works with Ninja 1.8.2 for a dummy test project on macOS using CMake 3.11.0. I would expect it to work with CMake 3.2 or later (that's when support for the BUILD_BYPRODUCTS option was added).
If there is some way to workaround with explicitly listing BUILD_BYPRODUCTS, is there a way to simply communicate that the entire directory that will get installed is a byproduct? That is, /path/to/install/* is a byproduct?
Unlikely. How would Ninja know what is expected to be in such a directory? The only way to get reliable dependencies would be to explicitly list each file that was expected to be there, which you do using BUILD_BYPRODUCTS in your case.
If you're willing to download at configuration time, you could follow this post. It uses google-test as the example, but I've used the same technique for other dependencies. Just put your ExternalProject code in a separate file, say "CMakeLists.txt.dependencies" and then launch another cmake with execute_process. I use configure_file first to inject configuration information into the external project and to copy it into the build tree.
configure_file(CMakeLists.txt.dependency.in dependency/CMakeLists.txt)
execute_process(COMMAND "${CMAKE_COMMAND}" -G "${CMAKE_GENERATOR}" .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
execute_process(COMMAND "${CMAKE_COMMAND}" --build .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
I do this at configuration time so find_package and find_library commands can work on the dependencies.
And now it doesn't matter what generator you use.
I have a header only library, that I include in my project using ExternalProject_Add. The install command should just copy a folder. Since this should work on Windows and Linux, I tried to use file(COPY ...).
INSTALL_COMMAND "file(COPY ../src/include DESTINATION ../install/include)"
This gives an error since INSTALL_COMMAND gets executed as shell command. How can I use a CMake macro instead?
CMake has a "command mode", i.e. cmake -E ... which provides some cross-platform filesystem commands. To see all -E options, just run cmake -E.
To invoke CMake itself from within a CMakeLists.txt file, you can use the variable CMAKE_COMMAND:
INSTALL_COMMAND ${CMAKE_COMMAND} -E copy_directory ../src/include ../install/include
I'd like to add a custom target named "package" which depends on install target.
When I run make package it should cause first running make install and after that, running my custom command to create a package.
I have tried the following DEPENDS install but it does not work.
I get error message: No rule to make target CMakeFiles/install.dir/all, needed by CMakeFiles/package.dir/all
install(FILES
"module/module.pexe"
"module/module.nmf"
DESTINATION "./extension")
add_custom_target(package
COMMAND "chromium-browser" "--pack-extension=./extension"
DEPENDS install)
EDIT: I tried DEPENDS install keyword and add_dependencies(package install) but neither of them works.
According to http://public.kitware.com/Bug/view.php?id=8438
it is not possible to add dependencies to built-in targets like install or test
You can create custom target which will run install and some other script after.
CMake script
For instance if you have a CMake script MyScript.cmake:
add_custom_target(
MyInstall
COMMAND
"${CMAKE_COMMAND}" --build . --target install
COMMAND
"${CMAKE_COMMAND}" -P "${CMAKE_CURRENT_LIST_DIR}/MyScript.cmake"
WORKING_DIRECTORY
"${CMAKE_BINARY_DIR}"
)
You can run it by building target MyInstall:
cmake --build /path/to/build/directory --target MyInstall
Python script
Of course you can use any scripting language. Just remember to be polite to other platforms
(so probably it's a bad idea to write bash script, it will not work on windows).
For example python script MyScript.py:
find_package(PythonInterp 3.2 REQUIRED)
add_custom_target(
MyInstall
COMMAND
"${CMAKE_COMMAND}" --build . --target install
COMMAND
"${PYTHON_EXECUTABLE}" "${CMAKE_CURRENT_LIST_DIR}/MyScript.py"
WORKING_DIRECTORY
"${CMAKE_BINARY_DIR}"
)
One of the solutions is to install a script which runs the custom target:
add_custom_target(
custom_target
[...]
)
install(CODE "execute_process(COMMAND make custom_target)")
Refs:
https://cmake.org/cmake/help/v3.13/command/install.html#custom-installation-logic
https://cmake.org/cmake/help/v3.5/command/execute_process.html
EDIT: I tried DEPENDS install keyword and add_dependencies(package install) but neither of them works.
Documentation of add_dependencies mentions that: [...](but not targets generated by CMake like install)[...]