I am calling make in a cmake custom command (I have to bridge between a CMake build and a legacy make build). Everything works fine, except one thing: Instead of seeing the output of the make build live on stdout, it only gets flushed after the build finishes. I have searched online but couldn't find a solution so far. What could the cause for such a behavior be?
I tested this modified simple example which recursively calls itself. This does behave exactly how I expect. I see the echos from the recursive calls as they happen. I can't figure out what the difference to my actual code is:
include(ExternalProject)
ExternalProject_Add(Wrapper
PREFIX "Wrapper"
DOWNLOAD_COMMAND ""
CONFIGURE_COMMAND ""
BUILD_COMMAND ${CMAKE_COMMAND} -E echo hello
COMMAND ${CMAKE_COMMAND} -E sleep 5
COMMAND ${CMAKE_COMMAND} -E env FOO=BAR ${CMAKE_MAKE_PROGRAM} -C "${CMAKE_CURRENT_SOURCE_DIR}"
BUILD_ALWAYS ON
INSTALL_COMMAND ""
)
My actual code, which only shows the output after the external make build is completely finished:
ExternalProject_Add(Wrapper
PREFIX "Wrapper"
DOWNLOAD_COMMAND ""
CONFIGURE_COMMAND ""
BUILD_COMMAND ${CMAKE_COMMAND} -E copy ${target_files} ${CMAKE_INSTALL_PREFIX}
COMMAND ${CMAKE_COMMAND} -E env ASM_OPT="${CMAKE_ASM_FLAGS}" CC_OPT="${CMAKE_C_FLAGS}" CPP_OPT="${CMAKE_CXX_FLAGS}" ${CMAKE_MAKE_PROGRAM} -C "${CMAKE_CURRENT_SOURCE_DIR}/.." -f Makefile.mak -O -j 4
BUILD_ALWAYS ON
INSTALL_COMMAND ""
DEPENDS ${ALL_LIBS}
)
Bad news: In the case of other than Unix Makefiles generator (e.g., Ninja, the CMAKE_MAKE_PROGRAM variable gonna have a path to underlaid build system tool (e.g., /usr/bin/ninja for Ninja generator :)
A better way is to use ExternalProject to build your non-CMake (third party) project.
Or, since CMake 3.18 execute_process got the ECHO_OUTPUT_VARIABLE named keyword (option) which is equal to *nix tee command...
Related
I have a setup where I use a custom command to check the current hash of a git repository so that other commands can clone it if it has updated
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt
COMMAND ${GIT_EXECUTABLE} ls-remote ${MODULE_URL} master > ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt.tmp
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt.tmp ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt
COMMAND ${CMAKE_COMMAND} -E rm ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt.tmp
)
Of course this will only run once as CMake sees no reason to rerun it. I can force it to run by adding a second (dummy) output - CMake then recognises that this output doesn't exist and then reruns the rule. However the Makefile that this generates actually deletes module_VERSION.txt before running the command rendering the whole pursuit pointless (Ninja does not have this problem).
I am able to get this to work but in an extremely hacky way: creating another target that always runs and then generating a dependency on this.
# Use echo_append as a no-op
add_custom_command(
OUTPUT module_FORCERUN
COMMAND ${CMAKE_COMMAND} -E echo_append
)
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt
COMMAND ${GIT_EXECUTABLE} ls-remote ${MODULE_URL} master > ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt.tmp
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt.tmp ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt
COMMAND ${CMAKE_COMMAND} -E rm ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt.tmp
DEPENDS module_FORCERUN
)
This seems just really hacky and like it could be relying on some corner cases in cmake which aren't guaranteed to be stable. Is there a better way to get this working?
I am using cmake 3.21.3
Use add_custom_target for implement "always run" functionality and via BYPRODUCTS keyword specify the file which it could produce/update:
add_custom_target(update_module_version
BYPRODUCTS ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt
COMMAND ${GIT_EXECUTABLE} ls-remote ${MODULE_URL} master > ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt.tmp
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt.tmp ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt
COMMAND ${CMAKE_COMMAND} -E rm ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt.tmp
)
That way, if any other target will depend on update_module_version one, the module_VERSION.txt file will be created/updated before evaluation of the target.
Such target-level dependency will be created automatically by CMake, if given file will be listed as dependency for target/command in the same directory, where target update_module_version is created:
add_custom_command(OUTPUT <...>
DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt
COMMAND <...>
)
From other directories the target-level dependency should be specified explicitly:
# If used in other directories
add_custom_command(OUTPUT <...>
DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/module_VERSION.txt
update_module_version
COMMAND <...>
)
This seems to be a common problem without a clear answer.
The situation is: we have a 3rd party dependency that we want to install at build time when building a target that depends on it. That's roughly:
ExternalProject_Add(target-ep
DOWNLOAD_COMMAND <whatever>
BUILD_COMMAND ""
INSTALL_COMMAND ""
CONFIGURE_COMMAND "")
add_library(target-imp STATIC IMPORTED)
set_target_properties(target-imp PROPERTIES
INTERFACE_INCLUDE_DIRECTORIES /path/to/install/include
IMPORTED_LOCATION /path/to/install/lib/libwhatever.a)
add_library(target INTERFACE)
target_link_libraries(target INTERFACE target-imp)
add_dependencies(target target-ep)
(It takes three to tango here because of cmake issue 15052)
When using Unix Makefiles as the generator, this works great. Only installs dependencies on demand, all the builds work correctly.
However, on Ninja, this fails immediately with something like:
ninja: error: '/path/to/install/lib/libwhatever.a', needed by 'something', missing and no known rule to make it
This is because Ninja scans dependencies differently from Make (see ninja issue 760). So what we have to do is actually tell Ninja that this external dependency exists. We can do that:
ExternalProject_Add(target-ep
DOWNLOAD_COMMAND <whatever>
BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
BUILD_COMMAND ""
INSTALL_COMMAND ""
CONFIGURE_COMMAND "")
Which unfortunately also fails with:
No build step for 'target-ep'ninja: error: mkdir(/path/to/install): Permission denied
This is because my download step has permissions to write to that path, but whatever mkdir command is being run by the underlying add_custom_command() from with ExternalProject_Add() does not.
So:
Is this possible at all with Ninja and CMake? (Version is not an issue, I can use the latest CMake if that solves the problem)
If there is some way to workaround with explicitly listing BUILD_BYPRODUCTS, is there a way to simply communicate that the entire directory that will get installed is a byproduct? That is, /path/to/install/* is a byproduct?
The hidden mkdir step of ExternalProject (which all other steps directly or indirectly depend on) always tries to create the full set of directories, even if they won't be used. You can see this here. For reference, it does this:
ExternalProject_Add_Step(${name} mkdir
COMMENT "Creating directories for '${name}'"
COMMAND ${CMAKE_COMMAND} -E make_directory ${source_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${binary_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${install_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${tmp_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${stamp_dir}${cfgdir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${download_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${log_dir} # This one only since CMake 3.13
)
The default install location on Unix systems is probably going to be /usr/local, so if you don't have write permissions to all of the directories it tries to make, then that may be related to your problem. I suggest you check the permissions of each of these locations and make sure they either already exist or are writable. Alternatively, you could specify an install directory that is local to the build tree so that even though it won't be used, it can at least always be created (see example further below).
If you use Ninja, it will be more rigorous in its dependency checking than make. You have target-ep doing the download that provides libwhatever.a, so you do need BUILD_BYPRODUCTS to tell Ninja that target-ep is what creates that file. As you've found out, if you don't then target-imp will point at a library that won't initially exist and Ninja rightly complains that it is missing and it doesn't know how to create it. If you provide BUILD_BYPRODUCTS, it makes sense that the build step shouldn't be empty, so you probably need to do something as a build step, even if it is just a BUILD_COMMAND that doesn't actually do anything meaningful.
The following modified definition of target-ep should hopefully get things working for you:
ExternalProject_Add(target-ep
INSTALL_DIR ${CMAKE_CURRENT_BUILD_DIR}/dummyInstall
DOWNLOAD_COMMAND <whatever>
BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
BUILD_COMMAND ${CMAKE_COMMAND} -E echo_append
INSTALL_COMMAND ""
CONFIGURE_COMMAND "")
Your original question also creates a dependency on the wrong target. target-imp should depend on target-ep, but you had target depend on target-ep instead. The correct dependency can be expressed by this:
add_dependencies(target-imp target-ep)
With the BUILD_BYPRODUCTS option, Ninja already knows the above dependency, but it is needed for other generators, including make.
You haven't specified what your <whatever> download command does, but I'm assuming it is responsible for ensuring that the library will exist at /path/to/install/lib/libwhatever.a when it has executed. You could also try making the DOWNLOAD_COMMAND empty and putting <whatever> as the BUILD_COMMAND instead.
To address your specific questions:
Is this possible at all with Ninja and CMake? (Version is not an issue, I can use the latest CMake if that solves the problem)
Yes, I verified that the approach mentioned above works with Ninja 1.8.2 for a dummy test project on macOS using CMake 3.11.0. I would expect it to work with CMake 3.2 or later (that's when support for the BUILD_BYPRODUCTS option was added).
If there is some way to workaround with explicitly listing BUILD_BYPRODUCTS, is there a way to simply communicate that the entire directory that will get installed is a byproduct? That is, /path/to/install/* is a byproduct?
Unlikely. How would Ninja know what is expected to be in such a directory? The only way to get reliable dependencies would be to explicitly list each file that was expected to be there, which you do using BUILD_BYPRODUCTS in your case.
If you're willing to download at configuration time, you could follow this post. It uses google-test as the example, but I've used the same technique for other dependencies. Just put your ExternalProject code in a separate file, say "CMakeLists.txt.dependencies" and then launch another cmake with execute_process. I use configure_file first to inject configuration information into the external project and to copy it into the build tree.
configure_file(CMakeLists.txt.dependency.in dependency/CMakeLists.txt)
execute_process(COMMAND "${CMAKE_COMMAND}" -G "${CMAKE_GENERATOR}" .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
execute_process(COMMAND "${CMAKE_COMMAND}" --build .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
I do this at configuration time so find_package and find_library commands can work on the dependencies.
And now it doesn't matter what generator you use.
I have a console application called "foo", which takes a reference text file as input (in.txt) and generates text at standard output (I want to keep this behaviour).
In make (not cmake), I use a test target, which calls foo and redirects the output to a file (out.txt) as follows. Then, I use diff to compare the file out.txt with the expected refernece (ref.txt)
test:
./foo -a test/in.txt > test/out.txt
diff test/out.txt test/ref.txt
This works fine using make. Now my question is; how can I use cmake to create a similar Makefile?
From within a subdrectory called build, I tried
project(foo)
...
add_test(NAME test1 COMMAND ./foo ../test/in.txt > ../test/out.txt)
enable_testing()
Using cmake version 3.5, I get a Makefile without errors, but when I call make test, the test itself fails. It seems the cmake command add_test supports command line arguments, but not the redirection. I tried quotes and escaping witout success. Since I could not pass this part, I didn't try to use diff. I just imagine that I could pack foo and diff in one line using & as you can do with bash. That would be the second step.
Turning my comment into an answer
As #Tsyvarev has stated, CTest commands are not run in a shell's context. But you could just add the shell needed yourself and use e.g. sh as the command to be called with add_test().
I've run some tests with your example code and the following did work successfully:
add_test(NAME test1 COMMAND sh -c "$<TARGET_FILE:foo> ../test/in.txt > ../test/out.txt")
This solution is not platform independent (it depends on sh to be available in the search paths).
So if you want to be more flexible you could do something like:
include(FindUnixCommands)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/in.txt" _in)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/out.txt" _out)
if (BASH)
add_test(
NAME test1
COMMAND ${BASH} -c "$<TARGET_FILE:foo> ${_in} > ${_out}"
)
else()
if (WIN32)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E chdir $<TARGET_FILE_DIR:foo> $ENV{ComSpec} /c "$<TARGET_FILE_NAME:foo> ${_in} > ${_out}"
)
else()
message(FATAL_ERROR "Unknown shell command for ${CMAKE_HOST_SYSTEM_NAME}")
endif()
endif()
Additionally there is the possibility to execute a more platform independent diff with ${CMAKE_COMMAND} -E compare_files <file1> <file2>. So you could simplify your complete makefile based example in CMake with:
add_custom_command(
TARGET foo
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E echo "Running $<TARGET_FILE_NAME:foo> ..."
COMMAND foo in.txt > out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E compare_files in.txt out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
References
Integrate bash test scripts in cmake
CMake: piping commands to executable
cmake: make tests successfully passing part of the build process
They say you cannot:
There is no redirection of output using add_test arguments.
Unlike to commands in add_custom_command, which are executed as a part of makefile receipts (that is, in the context of some shell), tests are executed directly by CTest, without any shell involved. So, shell mechanisms don't work for tests.
You may create wrapper script, which calls program, given as parameter, and performs redirection, futher diff and so on. Then use this script (with appropriate arguments) as a COMMAND for add_test.
I have a console application called "foo", which takes a reference text file as input (in.txt) and generates text at standard output (I want to keep this behaviour).
In make (not cmake), I use a test target, which calls foo and redirects the output to a file (out.txt) as follows. Then, I use diff to compare the file out.txt with the expected refernece (ref.txt)
test:
./foo -a test/in.txt > test/out.txt
diff test/out.txt test/ref.txt
This works fine using make. Now my question is; how can I use cmake to create a similar Makefile?
From within a subdrectory called build, I tried
project(foo)
...
add_test(NAME test1 COMMAND ./foo ../test/in.txt > ../test/out.txt)
enable_testing()
Using cmake version 3.5, I get a Makefile without errors, but when I call make test, the test itself fails. It seems the cmake command add_test supports command line arguments, but not the redirection. I tried quotes and escaping witout success. Since I could not pass this part, I didn't try to use diff. I just imagine that I could pack foo and diff in one line using & as you can do with bash. That would be the second step.
Turning my comment into an answer
As #Tsyvarev has stated, CTest commands are not run in a shell's context. But you could just add the shell needed yourself and use e.g. sh as the command to be called with add_test().
I've run some tests with your example code and the following did work successfully:
add_test(NAME test1 COMMAND sh -c "$<TARGET_FILE:foo> ../test/in.txt > ../test/out.txt")
This solution is not platform independent (it depends on sh to be available in the search paths).
So if you want to be more flexible you could do something like:
include(FindUnixCommands)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/in.txt" _in)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/out.txt" _out)
if (BASH)
add_test(
NAME test1
COMMAND ${BASH} -c "$<TARGET_FILE:foo> ${_in} > ${_out}"
)
else()
if (WIN32)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E chdir $<TARGET_FILE_DIR:foo> $ENV{ComSpec} /c "$<TARGET_FILE_NAME:foo> ${_in} > ${_out}"
)
else()
message(FATAL_ERROR "Unknown shell command for ${CMAKE_HOST_SYSTEM_NAME}")
endif()
endif()
Additionally there is the possibility to execute a more platform independent diff with ${CMAKE_COMMAND} -E compare_files <file1> <file2>. So you could simplify your complete makefile based example in CMake with:
add_custom_command(
TARGET foo
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E echo "Running $<TARGET_FILE_NAME:foo> ..."
COMMAND foo in.txt > out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E compare_files in.txt out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
References
Integrate bash test scripts in cmake
CMake: piping commands to executable
cmake: make tests successfully passing part of the build process
They say you cannot:
There is no redirection of output using add_test arguments.
Unlike to commands in add_custom_command, which are executed as a part of makefile receipts (that is, in the context of some shell), tests are executed directly by CTest, without any shell involved. So, shell mechanisms don't work for tests.
You may create wrapper script, which calls program, given as parameter, and performs redirection, futher diff and so on. Then use this script (with appropriate arguments) as a COMMAND for add_test.
I have a header only library, that I include in my project using ExternalProject_Add. The install command should just copy a folder. Since this should work on Windows and Linux, I tried to use file(COPY ...).
INSTALL_COMMAND "file(COPY ../src/include DESTINATION ../install/include)"
This gives an error since INSTALL_COMMAND gets executed as shell command. How can I use a CMake macro instead?
CMake has a "command mode", i.e. cmake -E ... which provides some cross-platform filesystem commands. To see all -E options, just run cmake -E.
To invoke CMake itself from within a CMakeLists.txt file, you can use the variable CMAKE_COMMAND:
INSTALL_COMMAND ${CMAKE_COMMAND} -E copy_directory ../src/include ../install/include