The most straight forward way to copy a directory in CMake during build time is:
add_custom_target(my-copy-dir ALL COMMAND ${CMAKE_COMMAND} -E copy_directory
${INPUT_DIR} ${OUTPUT_DIR})
However, this approach is not desired because the copy will be executed every single time even when everything is up-to-date, which slow things down. Therefore, the below approach is better
file(
GLOB_RECURSE FILES
LIST_DIRECTORIES false
RELATIVE ${INPUT_DIR}/
${INPUT_DIR}/*)
foreach(FILE ${FILES})
add_custom_command(
OUTPUT ${OUTPUT_DIR}/${FILE}
DEPENDS ${INPUT_DIR}/${FILE}
COMMAND ${CMAKE_COMMAND} -E copy ${INPUT_DIR}/${FILE} ${OUTPUT_DIR}/${FILE})
list(APPEND ALL_OUTPUT_FILES ${OUTPUT_DIR}/${FILE})
endforeach()
add_custom_target(my-copy-dir ALL DEPENDS ${ALL_OUTPUT_FILES})
The copy will only be executed if there are changes to the files in the input directory. Most of the time this approach works perfectly, however, I recently encountered a situation where there are symlinks in the input folder, and the symlinks are ignored during the file glob, so unfortunately I have to use the first approach to copy this folder in order to preserve the symlink. Is there a way to make the second approach work with a folder that contains symlinks?
For more context, the folder I am copying is a macOS framework. For example, assuming the name of the framework is Foo.framework, Foo.h is inside Foo.framework/Versions/A/Headers, and there's also a symlink to Foo.h in Foo.framework/Headers. If I use the second approach, I can see Foo.h in Foo.framework/Versions/A/Headers but not Foo.framework/Headers, whereas if I use the first approach I can see Foo.h in both, but I would like to make the second approach work in order to avoid copying every single time when everything is up-to-date. How can I achieve this?
Just use the FOLLOW_SYMLINKS argument of file(GLOB_RECURSE) :P
Also, note that globbing also always has a tradeoff. See the CONFIGURE_DEPENDS argument. Either you don't get glob match updates at build time, or you do (in which case there's the cost of the glob matching having to run for every build). Take that into account in your statement "Therefore, the below approach is better".
Related
Edit: my question targets the early configure stage where CMake input files are still being parsed and thus have to be present before include() is being called. So the answer found here: Force CMake to generate configure_file target every build does not solve my problem since it generates files after include() statements have been interpreted.
I have a CMakeLists.txt that includes a file which is generated in the configure stage:
execute_process(COMMAND "my-generator -o generated.cmake")
include(generated.cmake)
Apart from the fact that this approach doesn't feel right (not to say elegant) I now need to re-generate that file before every build (my-generator produces output that incorporates the current time).
My assumption is that I can't use add_custom_command() or add_custom_target() because the file would be generated at compile time but needed in the configure-step.
This very old post suggests to touch the input file so I did this:
execute_process(
COMMAND "my-generator -o generated.cmake"
COMMAND cmake -E touch "${CMAKE_CURRENT_LIST_FILE}")
.. which does not produce errors but calling make multiple times won't run the configure step more than once.
What do I do wrong? Is there a better approach?
(apologies: cross-posted from CMake mailing list)
I'm trying to get my head round CMake's regex implementation; I have a folder containing 4 folders and 2 text files as follows:
build/
projectA/
CMakeLists.txt
extrafiles/
README
temp/
One line of CMakeLists.txt is:
set(CPACK_SOURCE_IGNORE_FILES "[^projectA]$")
In my source package that is then subsequently generated, build/, projectA/ and extrafiles are present, but temp/ and the 2 text files are not. I'm trying to get to a stage where the regex will ignore everything in the folder except for projectA/, README and CMakeLists.txt, but can't work out at the moment how the regex I've
supplied is giving those results.
I guess what this boils down to is how to match a whole string using regex. I realise that the docs say Matches any character(s) not inside the brackets which is where I guess I'm going wrong...
Further exploration
In trying to understand CMake's regex implementation, I thought I'd start from 1st principles and do some easy stuff.
If I do
set(CPACK_SOURCE_IGNORE_FILES projectA)
then the folder projectA doesn't appear in my source package (as expected); however, if I do
set(CPACK_SOURCE_IGNORE_FILES ^projectA$)
or
set(CPACK_SOURCE_IGNORE_FILES ^/projectA/$)
then projectA does appear. What is it about the ^ (beginning of line) and $ (end of line) that I'm not understanding?
Even more
As probably obvious, projectA is not actually the name of my project, but everything above holds true when I physically rename my project folder to projectA. But, when I replace
set(CPACK_SOURCE_IGNORE_FILES projectA)
with
set(CPACK_SOURCE_IGNORE_FILES <name of my project>)
and rename my actual project folder from projectA to its actual name, I end up with an empty tarball! Argh! I have absolutely no idea what strange tricks CMake is playing on me, but I just want to cry.
Any insight will be greatly appreciated!
SELF CONTAINED EXAMPLE
As requested by Fraser, a self contained example showing 2 of the 'features' I've described. However, I do know that I'm running CMake in a slightly non-standard way, in order to keep everything to do with individual builds together, so if there's any proof running CMake in a more standard way eliminates these problems I'd be interested to see them.
Step 1: creating files
Create tree:
cd ~
mkdir
cd projectA
mkdir projectA
Create C file, and save it as ~/projectA/projectA/helloworld.c:
#include <stdio.h>
#include <stdlib.h>
int main(void) {
printf("!!!Hello World!!!\n"); /* prints !!!Hello World!!! */
printf("!!!Hello CMake!!!\n"); /* prints !!!Hello CMake!!! */
return 0;
}
create a file that won't need compiling, and save it as ~/projectA/test.sh:
#A non compiled program
echo "Hello world!"
create ~/projectA/CMakeLists.txt:
cmake_minimum_required (VERSION 2.6)
project (HelloWorld)
set(CMAKE_INSTALL_PREFIX "$ENV{HOME}/projectAinstall")
add_executable(helloworld projectA/helloworld.c)
install(TARGETS helloworld DESTINATION .)
include(InstallRequiredSystemLibraries)
set(CPACK_GENERATOR "TGZ")
set(CPACK_SOURCE_GENERATOR "TGZ")
include(CPack)
Step 2: compiling
In ~/projectA, run:
chris#chris:~/projectA$ cmake -H. -Bbuild
then:
make -C build && make -C build package && make -C build package_source
this results in 2 tarballs in the build folder. Moving these somewhere else and untarring them shows helloworld in the binary tarball (as expected), and everything from the ~/projectA/projectA in the source tarball, including test.sh which won't get compiled (which Fraser seemed surprised about)
Step 3: random tests
Modifying CMakeLists.txt to include
set(CPACK_SOURCE_IGNORE_FILES "projectA")
and rerunning the CMake / Make commands above results in an empty source tarball, but with the same binary tarball as above. I have now realised that changing the directory tree so that the top level directory is testproject (and so different to its child folder) doesn't result in an empty source tarball, and does only remove the files listed in CPACK_SOURCE_IGNORE_FILES
I don't think you can achieve what you're after using CPACK_SOURCE_IGNORE_FILES (although I'm not certain). As you rightly noted, CMake's regex handling allows for excluding groups of characters, but I don't think it allows for negating whole patterns. [See updated answer at the end of the edits.]
That being said, I guess you can list all the folders you wish to exclude in your install command. Not as robust as excluding everything except "projectA", but still here's the syntax:
install(DIRECTORY .
DESTINATION the_install_subdir
REGEX "build|extrafiles|temp+" EXCLUDE)
Regarding the empty tarball, I imagine that you maybe have <name of my project> both as your project's root dir and as a subdir? So in your example, if you called your project "projectA", then you'd have "projectA/build", "projectA/projectA", etc.
If so, the regex will work on the full path, and hence all files within your project will contain projectA/ within their paths.
As for the crying... well, I can only advise you to get a grip and pull yourself together! :-)
Edit: In response to the comments, here's a quick example of using the install command to achieve the goal:
install(DIRECTORY projectA
DESTINATION the_install_subdir)
install(FILES CMakeLists.txt README DESTINATION the_install_subdir)
Further Edit:
OK, your example helps a lot - I had indeed misunderstood what you were doing. I hadn't picked up that you were actually making 2 different targets ("package" and "package_source"). I had thought you were creating the binary package by doing something like
cpack -G DEB
and that you were creating the other package by doing
cpack -G TGZ
These both build the binary package. My mistake - I should have paid more attention. Sorry!
As for your specific questions:
Question 1
It seems to me that installing files / directories that aren't compiled but are at the same level as the folder containing all the compiled files (i.e. bin), and then ignoring the bin folder using CPACK_SOURCE_IGNORE_FILES results in an empty tarball - is this correct?
I take this to mean: "Should doing set(CPACK_SOURCE_IGNORE_FILES "${CMAKE_BINARY_DIR}") result in an empty tarball?" The answer is probably not.
Because CPACK_SOURCE_IGNORE_FILES represents a regex, I'm sure there are cases where the resultant regex could match every file in the project, and this would cause an empty tarball. However I imagine it's fairly unlikely.
If, rather than using the full path to your bin dir via the variable ${CMAKE_BINARY_DIR} you were to just give the folder name, there would be a much greater chance of an empty tarball. Say you call your bin dir "build" and have set(CPACK_SOURCE_IGNORE_FILES "build"). If your project lived in say ~/test_builds/projectA, then the regex "build" would match every file in the project since each contains "test_builds"; resulting in an empty tarball.
I think this is the crux of issue each time you've generated an empty tarball. Whatever the regex is trying to achieve, it actually ends up matching and excluding all files.
Question 2
It also seems that files in the CMAKE_SOURCE_DIR which aren't 'installed' don't end up in the binary tarball but do end up in the source tarball
Yes, the "package_source" is indeed a different target to the binary package. It by default contains all files in the ${CMAKE_SOURCE_DIR}, whereas the "package" target contains only items added via install commands. Here, the term "source files" is probably a slight misnomer since it means all files in the source tree - not just .c, .cc, .cxx, etc.
Original Question
I think there's a reasonably safe way to achieve your original aim after all! If you use file(GLOB ...) to generate a non-recursive list of all files/folders in your root, then remove those you wish to keep in the source package, you should be able to use the remaining list as the regex value of CPACK_SOURCE_IGNORE_FILES:
file(GLOB SourceIgnoreFiles "${CMAKE_SOURCE_DIR}/*")
set(SourceKeepFiles "${CMAKE_SOURCE_DIR}/projectA"
"${CMAKE_SOURCE_DIR}/CMakeLists.txt"
"${CMAKE_SOURCE_DIR}/README")
list(REMOVE_ITEM SourceIgnoreFiles ${SourceKeepFiles})
# Escape any '.' characters
string(REPLACE "." "\\\\." SourceIgnoreFiles "${SourceIgnoreFiles}")
set(CPACK_SOURCE_IGNORE_FILES "${SourceIgnoreFiles}")
Hopefully this should now work for you. Sorry again for the misdirections.
CMake tends to use absolute paths except in contexts where there's a strong argument for using relative paths. So I'm pretty sure it's running each regex in CPACK_SOURCE_IGNORE_FILES against absolute paths of files (which should answer your question "What is it about the ^ (beginning of line) and $ (end of line) that I'm not understanding?"). Anything that isn't matched by any regex in CPACK_SOURCE_IGNORE_FILES is not ignored.
What you want is probably something like:
set(CPACK_SOURCE_IGNORE_FILES
/build/
/extrafiles/
/temp/
)
After my program is linked I need to perform some post-processing on it. I added a add_custom_command(TARGET ... and that worked fine. However, this extra custom command runs a script (that is not generated; it's checked into the codebase), and I want the target to be considered out of date if that script changes so it will be rebuilt properly.
The add_dependencies rule seems to only work between top-level elements, which this is not (it's just a script), and there's no DEPENDS element in this form of the add_custom_command that I can use.
How do I do this?
It's unfortunately a bit convoluted, but you can use add_custom_target to invoke CMake in script-processing mode via -P.
You need to use add_custom_target here since it will always execute, even if everything's up to date.
Having made this decision, we need to have the custom target execute commands which will check for a new version of your post-processing script file (let's call this "my_script" and assume it's in your root dir), and if it's changed cause your dependent target to go out of date.
This would comprise:
Compare a previous copy of "my_script" to the current "my_script" in the source tree. If the current "my_script" is different, or if the copy doesn't exist (i.e. this is the first run of CMake) then...
Copy "my_script" from the source tree to the build tree, and...
Touch a source file of the dependent target so that it goes out of date.
All of the commands required inside the CMake script can be achieved using execute_process to invoke cmake -E.
So the CMake script (called e.g. "copy_script.cmake") would be something like:
execute_process(COMMAND ${CMAKE_COMMAND} -E compare_files
${OriginalScript} ${CopiedScript} RESULT_VARIABLE Result)
if(Result)
execute_process(COMMAND ${CMAKE_COMMAND} -E copy
${OriginalScript} ${CopiedScript})
execute_process(COMMAND ${CMAKE_COMMAND} -E touch_nocreate ${FileToTouch})
endif()
The CMake script needs to have required variables passed in via the -D args before calling -P, so the calling CMakeLists.txt would have something like:
set(FileToTouch ${CMAKE_SOURCE_DIR}/src/main.cpp)
add_custom_target(CopyScript ALL ${CMAKE_COMMAND}
-DOriginalScript=${CMAKE_SOURCE_DIR}/my_script
-DCopiedScript=${CMAKE_BINARY_DIR}/my_script
-DFileToTouch=${FileToTouch}
-P ${CMAKE_SOURCE_DIR}/copy_script.cmake)
add_executable(MyExe ${FileToTouch})
This will cause a full rebuild of the executable, since it thinks a source file has been modified. If you only require to force relinking there may be a better way to achieve this.
I've got a CMake project that contains code and a few data files (images to be precise).
My directory structure is like this:
src
data
src contains the source code, data the data files. CMake suggests out of source builds, so when I invoke make, I have the executable program, but not the data files, thus I cannot execute the program.
Of course, make install would copy my data files to the required location and make it work, therefore I develop like this right now:
cmake -DCMAKE_INSTALL_DIR=dist
<edit source code>
make install
dist/myprogram.exe
That's okay if I'm working with the command line and an editor, but I recently decided to move to Eclipse CDT. Generating an Eclipse project from CMake works great, but manually executing the install target from Eclipse is not so nice.
How do you people tackle this problem? Does your program have some clever algorithms to try and find its data directory even if it's not where the binary is? Or do you not use out of source builds?
configure_file should solve that problem.
I have a CMakeLists.txt file in my data directory which contains the following:
configure_file(data_file ${CMAKE_CURRENT_BINARY_DIR}/data_file COPYONLY)
This copies the specified file into the build directory when cmake is invoked, so it is available in the same location even in out of source builds.
configure_file does not support directories however while the file command does:
file(COPY assets DESTINATION ${CMAKE_CURRENT_BINARY_DIR})
And if copying the files takes too much time (they are images...) you could make it even better by creating a "custom" data_header.h with configure_file which contains the paths to the data still in your source-directory.
This is what I do: I have a file "global_build_config.h.in" in my source, containing the following:
const char* const global_testdatapath = "#Test_Data_Path#";
and then use configure_file in CMake:
# Assume CMake knows a variable Test_Data_Path, it will be filled in automatically
# in the generated config/Global_Build_Config.h
configure_file( Global_Build_Config.h.in ${CMAKE_BINARY_DIR}/config/Global_Build_Config.h )
# The config directory should be added as a include-searchpath
include_directories( ${CMAKE_BINARY_DIR}/config/ )
I can then #include "Global_Build_Config.h" in my cpp files and refer to the fixed path.
Your question is a bit old, but in case you're still interested (or someone else), I have a similar scenario where I copy testdata for a unit-test target:
add_custom_command( TARGET ${UTEST_EXE_NAME}
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E echo "Copying unit test data.."
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_HOME_DIRECTORY}/utest/testdata ${CMAKE_BINARY_DIR}
)
So the main idea is to use a post-build target, and it is executed after each build. For me, it's not much data, and the filesystem caches it, so I don't feel the copy process at all. You could probably enhance this by copying with copy_if_different. In that case, however, you have to create a list of your image files and write a loop, because the command is file based. With the GLOB command, this shouldn't be hard to do if you need to.
I'm trying to use add_custom_command to generate a file during the build. The command never seemed to be run, so I made this test file.
cmake_minimum_required( VERSION 2.6 )
add_custom_command(
OUTPUT hello.txt
COMMAND touch hello.txt
DEPENDS hello.txt
)
I tried running:
cmake .
make
And hello.txt was not generated. What have I done wrong?
The add_custom_target(run ALL ... solution will work for simple cases when you only have one target you're building, but breaks down when you have multiple top level targets, e.g. app and tests.
I ran into this same problem when I was trying to package up some test data files into an object file so my unit tests wouldn't depend on anything external. I solved it using add_custom_command and some additional dependency magic with set_property.
add_custom_command(
OUTPUT testData.cpp
COMMAND reswrap
ARGS testData.src > testData.cpp
DEPENDS testData.src
)
set_property(SOURCE unit-tests.cpp APPEND PROPERTY OBJECT_DEPENDS testData.cpp)
add_executable(app main.cpp)
add_executable(tests unit-tests.cpp)
So now testData.cpp will generated before unit-tests.cpp is compiled, and any time testData.src changes. If the command you're calling is really slow you get the added bonus that when you build just the app target you won't have to wait around for that command (which only the tests executable needs) to finish.
It's not shown above, but careful application of ${PROJECT_BINARY_DIR}, ${PROJECT_SOURCE_DIR} and include_directories() will keep your source tree clean of generated files.
Add the following:
add_custom_target(run ALL
DEPENDS hello.txt)
If you're familiar with makefiles, this means:
all: run
run: hello.txt
The problem with two existing answers is that they either make the dependency global (add_custom_target(name ALL ...)), or they assign it to a specific, single file (set_property(...)) which gets obnoxious if you have many files that need it as a dependency. Instead what we want is a target that we can make a dependency of another target.
The way to do this is to use add_custom_command to define the rule, and then add_custom_target to define a new target based on that rule. Then you can add that target as a dependency of another target via add_dependencies.
# this defines the build rule for some_file
add_custom_command(
OUTPUT some_file
COMMAND ...
)
# create a target that includes some_file, this gives us a name that we can use later
add_custom_target(
some_target
DEPENDS some_file
)
# then let's suppose we're creating a library
add_library(some_library some_other_file.c)
# we can add the target as a dependency, and it will affect only this library
add_dependencies(some_library some_target)
The advantages of this approach:
some_target is not a dependency for ALL, which means you only build it when it's required by a specific target. (Whereas add_custom_target(name ALL ...) would build it unconditionally for all targets.)
Because some_target is a dependency for the library as a whole, it will get built before all of the files in that library. That means that if there are many files in the library, we don't have to do set_property on every single one of them.
If we add DEPENDS to add_custom_command then it will only get rebuilt when its inputs change. (Compare this to the approach that uses add_custom_target(name ALL ...) where the command gets run on every build regardless of whether it needs to or not.)
For more information on why things work this way, see this blog post: https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/
This question is pretty old, but even if I follow the suggested recommendations, it does not work for me (at least not every time).
I am using Android Studio and I need to call cMake to build C++ library. It works fine until I add the code to run my custom script (in fact, at the moment I try to run 'touch', as in the example above).
First of,
add_custom_command
does not work at all.
I tried
execute_process (
COMMAND touch hello.txt
)
it works, but not every time!
I tried to clean the project, remove the created file(s) manually, same thing.
Tried cMake versions:
3.10.2
3.18.1
3.22.1
when they work, they produce different results, depending on cMake version, one file or several. This is not that important as long as they work, but that's the issue.
Can somebody shed light on this mystery?