I am building an external project that generates shared objects with symlinks:
i.e.
libxml.so -> libxml.so.0.0.0
libxml.so.0 -> libxml.so.0.0.0
libxml.so.0.0.0
I want to copy these 3 files to my library output path. However, if I do:
ExternalProject_Add_Step(
xml copy2lib
COMMAND ${CMAKE_COMMAND} -E copy_directory ${EXTERNAL_INSTALL_PATH}/xml/lib/ ${LIBRARY_OUTPUT_PATH}
DEPENDEES install
)
All of the symlinks are destroyed and there are essentially 3 copies of the library. I am aware of the following solution, i.e.:
COMMAND ${CMAKE_COMMAND} -E create_symlink ./libxml.so.0.0.0 ./libxml.so
WORKING_DIRECTORY ${LIBRARY_OUTPUT_PATH}
However, in my case, the external project is generating dozens of libraries that follow this convention. Please tell me I don't have to manually repair every symlink?
CMake has a poor support for file links (symlinks, hardlinks, etc.). Partially, this is because there is no cross-platform view of such thing: Windows and Linux have different interpretation of these terms.
So, when you want to operate with Linux symlinks in a specific way, use Linux-specific commands. Instead of cross-platform cmake -E copy_directory use Linux-specific cp:
COMMAND cp -Pr ${EXTERNAL_INSTALL_PATH}/xml/lib/ ${LIBRARY_OUTPUT_PATH}
Note, that copiing symlink has a sence only for relative symlinks. If symlink contains absolute path, after copy it will refer to original library location, not to the copied one.
Related
I have a set of resource files that have nothing to do with the build steps of GCC or some other compiler's output. I need them copied from a folder in my project to the cmake build output folder. The goal is for the executable, when run from the build output folder, can see the resources.
How do people typically copy and install resources in cmake builds? Additionally, I want them copied regardless of changes in the build and I want it executed every time I run some cmake command, like build. See below for what I tried to solve this issue.
For example:
I have a bunch of shader files that I want copied. All shaders/* files should be copied into a directory in the build output called "shaders", because that's where the executable for the program lives.
file(GLOB out shaders/*)
foreach (o ${out})
message("${o} was copied to shaders")
file(COPY ${o} DESTINATION shaders)
endforeach ()
This only works sometimes, like when I reload the CMake project, e.g.:
/opt/clion-2021.2.3/bin/cmake/linux/bin/cmake \
-DCMAKE_BUILD_TYPE=Debug -DCMAKE_DEPENDS_USE_COMPILER=FALSE \
-G "CodeBlocks - Unix Makefiles" \
/home/hack/glad
Also, it doesn't execute "POST_BUILD", so the lastest version of the shaders/a.vert file doesn't get copied to the shaders/ directory in the output.
I tried using this, too, but it gave me some headaches:
add_custom_command(TARGET my-executable POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy shaders/* shaders)
I think there's something incorrect with that above, because it wasn't run every POST_BUILD if the build's code didn't change. I don't care if the build's code doesn't change because the files in shaders/* could have changed and should be copied regardless of cmake determining if there was a change in my-executable.
This gist on github was very helpful, here but the gist that applies to my question is included below.
add_custom_target(bar
COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/shaders
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/shaders ${CMAKE_BINARY_DIR}/shaders
COMMENT "copying ${CMAKE_SOURCE_DIR}/shaders to ${CMAKE_BINARY_DIR}/shaders"
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
)
The above code creates a bar cmake target that can be run separately with Make bar; then, if you add another target that requires those resources (the shaders above are resources for this other executable) then you can tie the dependency together using add_dependencies like this:
add_executable(a
main.c
opengl.c)
add_dependencies(a bar)
Now, every time the a target is run, the bar target is run as well, which has the effect of creating the directory and copying the files.
This was good enough for me, but to finish up, you can use this to create the files in a post build step after the other dependency is finished running:
add_custom_command(TARGET bar
# Run after all other rules within the target have been executed
POST_BUILD
COMMAND echo "executing a POST_BUILD command"
COMMENT "This command will be executed after building bar"
VERBATIM
)
Note that ${CMAKE_COMMAND} in the above examples of add_custom_command is a variable that points to the cmake executable, and you can run cmake -E to see the very helpful list of commands that come with cmake.
YIKES the post build step is only running after bar's target is built, not a's target. Hopefully, somebody can help me answer this better. I would still like to know how to copy files after a target is built, unless that's completely unnecessary and nobody should ever want to do that.
In my project I'd need to install my libraries in lib/, and to create symlinks to them in plugin/. I'm currently doing it in this way:
install(CODE "execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_INSTALL_PREFIX}/lib/libMyLib.so ${CMAKE_INSTALL_PREFIX}/plugin/lib")
This works, but if I configure the cmake run with -DCMAKE_INSTALL_PREFIX=/tmp/prefix1 and then I want to install just once in another folder by calling cmake --install . --prefix /tmp/prefix2 then I get errors because the symlinks are created referring to the standard installation prefix, i.e. /tmp/prefix2.
So my question is: is there a way to create custom symlinks during installation taking into account the actual installation prefix?
Thanks in advance for any hint.
Instead of expanding CMAKE_INSTALL_PREFIX on configuration side, expand it when it is executed:
install(CODE "execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink \${CMAKE_INSTALL_PREFIX}/lib/libMyLib.so \${CMAKE_INSTALL_PREFIX}/plugin/lib)")
This seems to be a common problem without a clear answer.
The situation is: we have a 3rd party dependency that we want to install at build time when building a target that depends on it. That's roughly:
ExternalProject_Add(target-ep
DOWNLOAD_COMMAND <whatever>
BUILD_COMMAND ""
INSTALL_COMMAND ""
CONFIGURE_COMMAND "")
add_library(target-imp STATIC IMPORTED)
set_target_properties(target-imp PROPERTIES
INTERFACE_INCLUDE_DIRECTORIES /path/to/install/include
IMPORTED_LOCATION /path/to/install/lib/libwhatever.a)
add_library(target INTERFACE)
target_link_libraries(target INTERFACE target-imp)
add_dependencies(target target-ep)
(It takes three to tango here because of cmake issue 15052)
When using Unix Makefiles as the generator, this works great. Only installs dependencies on demand, all the builds work correctly.
However, on Ninja, this fails immediately with something like:
ninja: error: '/path/to/install/lib/libwhatever.a', needed by 'something', missing and no known rule to make it
This is because Ninja scans dependencies differently from Make (see ninja issue 760). So what we have to do is actually tell Ninja that this external dependency exists. We can do that:
ExternalProject_Add(target-ep
DOWNLOAD_COMMAND <whatever>
BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
BUILD_COMMAND ""
INSTALL_COMMAND ""
CONFIGURE_COMMAND "")
Which unfortunately also fails with:
No build step for 'target-ep'ninja: error: mkdir(/path/to/install): Permission denied
This is because my download step has permissions to write to that path, but whatever mkdir command is being run by the underlying add_custom_command() from with ExternalProject_Add() does not.
So:
Is this possible at all with Ninja and CMake? (Version is not an issue, I can use the latest CMake if that solves the problem)
If there is some way to workaround with explicitly listing BUILD_BYPRODUCTS, is there a way to simply communicate that the entire directory that will get installed is a byproduct? That is, /path/to/install/* is a byproduct?
The hidden mkdir step of ExternalProject (which all other steps directly or indirectly depend on) always tries to create the full set of directories, even if they won't be used. You can see this here. For reference, it does this:
ExternalProject_Add_Step(${name} mkdir
COMMENT "Creating directories for '${name}'"
COMMAND ${CMAKE_COMMAND} -E make_directory ${source_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${binary_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${install_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${tmp_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${stamp_dir}${cfgdir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${download_dir}
COMMAND ${CMAKE_COMMAND} -E make_directory ${log_dir} # This one only since CMake 3.13
)
The default install location on Unix systems is probably going to be /usr/local, so if you don't have write permissions to all of the directories it tries to make, then that may be related to your problem. I suggest you check the permissions of each of these locations and make sure they either already exist or are writable. Alternatively, you could specify an install directory that is local to the build tree so that even though it won't be used, it can at least always be created (see example further below).
If you use Ninja, it will be more rigorous in its dependency checking than make. You have target-ep doing the download that provides libwhatever.a, so you do need BUILD_BYPRODUCTS to tell Ninja that target-ep is what creates that file. As you've found out, if you don't then target-imp will point at a library that won't initially exist and Ninja rightly complains that it is missing and it doesn't know how to create it. If you provide BUILD_BYPRODUCTS, it makes sense that the build step shouldn't be empty, so you probably need to do something as a build step, even if it is just a BUILD_COMMAND that doesn't actually do anything meaningful.
The following modified definition of target-ep should hopefully get things working for you:
ExternalProject_Add(target-ep
INSTALL_DIR ${CMAKE_CURRENT_BUILD_DIR}/dummyInstall
DOWNLOAD_COMMAND <whatever>
BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
BUILD_COMMAND ${CMAKE_COMMAND} -E echo_append
INSTALL_COMMAND ""
CONFIGURE_COMMAND "")
Your original question also creates a dependency on the wrong target. target-imp should depend on target-ep, but you had target depend on target-ep instead. The correct dependency can be expressed by this:
add_dependencies(target-imp target-ep)
With the BUILD_BYPRODUCTS option, Ninja already knows the above dependency, but it is needed for other generators, including make.
You haven't specified what your <whatever> download command does, but I'm assuming it is responsible for ensuring that the library will exist at /path/to/install/lib/libwhatever.a when it has executed. You could also try making the DOWNLOAD_COMMAND empty and putting <whatever> as the BUILD_COMMAND instead.
To address your specific questions:
Is this possible at all with Ninja and CMake? (Version is not an issue, I can use the latest CMake if that solves the problem)
Yes, I verified that the approach mentioned above works with Ninja 1.8.2 for a dummy test project on macOS using CMake 3.11.0. I would expect it to work with CMake 3.2 or later (that's when support for the BUILD_BYPRODUCTS option was added).
If there is some way to workaround with explicitly listing BUILD_BYPRODUCTS, is there a way to simply communicate that the entire directory that will get installed is a byproduct? That is, /path/to/install/* is a byproduct?
Unlikely. How would Ninja know what is expected to be in such a directory? The only way to get reliable dependencies would be to explicitly list each file that was expected to be there, which you do using BUILD_BYPRODUCTS in your case.
If you're willing to download at configuration time, you could follow this post. It uses google-test as the example, but I've used the same technique for other dependencies. Just put your ExternalProject code in a separate file, say "CMakeLists.txt.dependencies" and then launch another cmake with execute_process. I use configure_file first to inject configuration information into the external project and to copy it into the build tree.
configure_file(CMakeLists.txt.dependency.in dependency/CMakeLists.txt)
execute_process(COMMAND "${CMAKE_COMMAND}" -G "${CMAKE_GENERATOR}" .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
execute_process(COMMAND "${CMAKE_COMMAND}" --build .
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
I do this at configuration time so find_package and find_library commands can work on the dependencies.
And now it doesn't matter what generator you use.
I am attempting to copy multiple files using the ${CMAKE_COMMAND} -E copy <from> <to> format, but I was wondering if there was a way to provide a number of files to copy to a specific directory. It seems the cmake copy only allows for one file to be copied at a time. I really don't want to use the copy command repeatedly when I would rather provide a list of files to copy as the first argument.
I'm thinking the easiest solution is to use the platform dependent "cp" command. While this definitely is not good for portability, our system is guaranteed to be built on Linux. A simple, platform independent solution would be better.
Copying multiple files is available from CMake 3.5
cmake -E copy <file>... <destination>
"cmake -E copy" support for multiple files
Command-Line Tool Mode
I did it with a loop
# create a list of files to copy
set( THIRD_PARTY_DLLS
C:/DLLFOLDER/my_dll_1.dll
C:/DLLFOLDER/my_dll_2.dll
)
# do the copying
foreach( file_i ${THIRD_PARTY_DLLS})
add_custom_command(
TARGET ${VIEWER_NAME}
POST_BUILD
COMMAND ${CMAKE_COMMAND}
ARGS -E copy ${file_i} "C:/TargetDirectory"
)
endforeach( file_i )
A relatively simple workaround would be to use ${CMAKE_COMMAND} -E tar to bundle the sources, move the tarball and extract it in the destination directory.
This could be more trouble than it's worth if your sources are scattered across many different directories, since extracting would retain the original directory structure (unlike using cp). If all the files are in one directory however, you could achieve the copy in just 2 add_custom_command calls.
Say your sources to be moved are all in ${CMAKE_SOURCE_DIR}/source_dir, the destination is ${CMAKE_SOURCE_DIR}/destination_dir and your list of filenames (not full paths) are in ${FileList}. You could do:
add_custom_command(
TARGET MyExe POST_BUILD
COMMAND ${CMAKE_COMMAND} -E tar cfj ${CMAKE_BINARY_DIR}/temp.tar ${FileList}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/source_dir)
add_custom_command(
TARGET MyExe POST_BUILD
COMMAND ${CMAKE_COMMAND} -E rename ${CMAKE_BINARY_DIR}/temp.tar temp.tar
COMMAND ${CMAKE_COMMAND} -E tar xfj temp.tar ${FileList}
COMMAND ${CMAKE_COMMAND} -E remove temp.tar
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/destination_dir)
Since I had more or less exactly the same issue and didn't like the solutions above I eventually came up with this. It does more than just copy files, but I thought I would post the whole thing as it shows the flexibility of the the technique in conjunction with generator expressions that allow different files and directories depending on the build variant. I believe the COMMAND_EXPAND_LISTS is critical to the functionality here.
This function not only copies some files to a new directory but then runs a command on each of them. In this case it uses the microsoft signtool program to add digital signatures to each file.
cmake_minimum_required (VERSION 3.12)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
SET(ALL_3RD_PARTY_DLLS_DEBUG "${CMAKE_CURRENT_SOURCE_DIR}/file1.dll" "${CMAKE_CURRENT_SOURCE_DIR}/file2.dll")
SET(ALL_3RD_PARTY_DLLS_RELEASE "${CMAKE_CURRENT_SOURCE_DIR}/file3.dll" "${CMAKE_CURRENT_SOURCE_DIR}/file4.dll")
STRING(REPLACE "${CMAKE_CURRENT_SOURCE_DIR}" ";${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/Debug" ALL_OUTPUT_3RD_PARTY_DLLS_DEBUG ${ALL_3RD_PARTY_DLLS_DEBUG})
LIST(REMOVE_AT ALL_OUTPUT_3RD_PARTY_DLLS_DEBUG 0)
STRING(REPLACE "${CMAKE_CURRENT_SOURCE_DIR}" ";${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/Release" ALL_OUTPUT_3RD_PARTY_DLLS_RELEASE ${ALL_3RD_PARTY_DLLS_RELEASE})
LIST(REMOVE_AT ALL_OUTPUT_3RD_PARTY_DLLS_RELEASE 0)
FILE(TO_NATIVE_PATH "C:\\Program\ Files\ (x86)\\Windows\ Kits\\10\\bin\\10.0.17763.0\\x86\\signtool.exe" SIGNTOOL_COMMAND)
add_custom_target(Copy3rdPartyDLLs ALL
COMMENT "Copying and signing 3rd Party DLLs"
VERBATIM
COMMAND_EXPAND_LISTS
COMMAND ${CMAKE_COMMAND} -E
make_directory "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/$<$<CONFIG:Release>:Release>$<$<CONFIG:Debug>:Debug>/"
COMMAND ${CMAKE_COMMAND} -E
copy_if_different
"$<$<CONFIG:Release>:${ALL_3RD_PARTY_DLLS_RELEASE}>"
"$<$<CONFIG:Debug>:${ALL_3RD_PARTY_DLLS_DEBUG}>"
"${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/$<$<CONFIG:Release>:Release>$<$<CONFIG:Debug>:Debug>/"
COMMAND ${SIGNTOOL_COMMAND} sign
"$<$<CONFIG:Release>:${ALL_OUTPUT_3RD_PARTY_DLLS_RELEASE}>"
"$<$<CONFIG:Debug>:${ALL_OUTPUT_3RD_PARTY_DLLS_DEBUG}>"
)
I hope this saves someone the day or so it took me to figure this out.
I'm coding a c++ project in vim.
I'd like to run a ctags command (ctags -R --c++-kinds=+p --fields=+iaS --extra=+q .) to generate references when I run make.
I think the way to do it is to use add_custom_command but I get confused on how to integrate it into CMakeLists.txt .
The most basic way to do this is:
set_source_files_properties( tags PROPERTIES GENERATED true)
add_custom_command ( OUTPUT tags
COMMAND ctags -R --c++-kinds=+p --fields=+iaS --extra=+q .
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR} )
add_executable ( MyProjectOutput tags )
The first line tells CMake that tags will be generated. The add_custom_command is CMake will generate tags when needed, and finally, some target needs to depend on tags. The default working directory is in the build tree, so WORKING_DIRECTORY must be set to your source tree. This is equivalent a Makefile entry:
tags:
ctags -R --c++-kinds=+p --fields=+iaS --extra=+q .
MyProjectOutput: tags
# Whatever here...
New solution:
I think CMake changed since the previous answer was given.
Here is the lines I added in my CMakeLists.txt (tested with version 2.8.12):
# Add "tags" target and make my_project depending on this target.
set_source_files_properties(tags PROPERTIES GENERATED true)
add_custom_target(tags
COMMAND ctags -R --c++-kinds=+p --fields=+iaS --extra=+q .
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR})
add_dependencies(my_project tags)
And it works perfectly now!
Daniel's and Creak's answers got me started, but I ended up with a more complex solution that I thought I'd share:
# Add a top-level "tags" target which includes all files in both
# the build and source versions of src/*.
set_source_files_properties(tags PROPERTIES GENERATED true)
add_custom_target(tags
COMMAND ctags -R --c++-kinds=+p --fields=+iaS --extra=+q
${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_SOURCE_DIR}
COMMAND ln -sf ${CMAKE_CURRENT_BINARY_DIR}/tags ${CMAKE_BINARY_DIR}
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
# ...but only make it a dependency of the project if the ctags program
# is available, else it will fail to build on Windows.
find_program(CTAGS_PATH ctags)
if(CTAGS_PATH)
message(STATUS "Found ctags: ${CTAGS_PATH}")
add_dependencies(MyProjecct tags)
endif(CTAGS_PATH)
It does several things that the simpler solutions do not:
It only adds "tags" as a dependency of the primary build product (MyProject) if there is actually a ctags program on the system. We don't want to break the build just because this is Windows, or because ctags simply hasn't been installed yet on the build system.
It extracts symbols from source files in both the build and source directories. This matters in a couple of cases.
First, you might be using configure_file() and come from an Autotools background, so you've named your true source files *.in, which means ctags -R won't scan them. You need it to scan the generated versions in the build directory. For example, you might have src/mainheader.h.in in your source tree, with the project version number automatically subbed into it as build/src/mainheader.h.
Second, some of your "source" files might be generated by other tools. In my current project, I have a couple of C++ header files that are generated by Perl scripts. I want symbols from both the generated headers and the Perl scripts in the tags file.
It works in a subdirectory.
In the project I'm working on right now, the primary build product is made from src/* relative to the project root, and I only want symbols from that subtree in the tags file. I don't want it to include symbols from the unit tests, the examples, or the utility scripts.
Because it is designed to run in a subdirectory, it creates a symlink to the src/tags file in the top of the build directory, so that vi -t TagName works. (I'm assuming here that if ctags exists, ln does, too.)