CMake's add_custom_command doesnt trigger rebuild of library - cmake

Iam using CMake together with ninja to build a library. The library is depending on some code which may be generated before-hand by a custom command. The source for this code is within the source-tree and it must stay there, I have no freedom here.
Here's my CMake code:
add_library(some_lib some_source.c)
#some_source.c may be modified by the following custom command
add_custom_command(
COMMAND codegen.exe -i some_input.xml
COMMAND ${CMAKE_COMMAND} -E touch "${CMAKE_BINARY_DIR}/generated"
OUTPUT ${CMAKE_BINARY_DIR}/generated
DEPENDS some_input.xml
COMMENT "Generating code ..."
)
add_custom_target(generate_something DEPENDS ${CMAKE_BINARY_DIR}/generated)
add_dependencies(some_lib generate_something)
Now if some_input.xml is changed I want to also rebuild some_lib. However in practice this code doesnt seem to work, the command is executed but after it is executed some_lib is not beeing rebuild, though the timestamps of the output files (some_source.c) of the custom command are newer than the library.
Can someone give me a hint on what am I doing wrong or how I can achieve this? Or is there a problem with CMake and ninja?
Thanks in advance, if you need more information please let me know.
Steve
[edit:]
It seems to be caused by the generator tool, when I use the BYPRODUCTS option of add_custom_command and replace the generator tool invocation by a cmake -E touch command on some_source.c everything works as expected. Can someone give me a hint on how to debug this, why ninja cancels the build after the generator tool has finished?
Thanks, Steve
[edit2]
I found the problem, the problem was that the codegen.exe tool was generating not only one source file but several into an output folder. I need to specify all of these potentially modified sources to as "BYPRODUCTS". Than everything works as expected.

I dont want CMake to delete the file if a "clean" is executed
BYPRODUCTS will still be cleaned.
What can be done is to utilize a dependency tracking file like you've shown (the "${CMAKE_BINARY_DIR}/generated", I used *.stamp) and then add the dependency tracking file as an OBJECT_DEPENDS to some_source.c
This only works if some_source.c is pre-existing. Initial testing had some_source.c get cleaned if I tried to add the GENERATED property.
cmake_minimum_required(VERSION 3.18)
project(in_source_generated)
add_library(some_lib some_source.c)
#some_source.c may be modified by the following custom command
add_custom_command(
# This is the command that actually generates the in source code file.
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_LIST_DIR}/some_input.c ${CMAKE_CURRENT_LIST_DIR}/some_source.c
# This command generates a stamp file which is used as the dependency
# tracking
COMMAND ${CMAKE_COMMAND} -E touch "${CMAKE_BINARY_DIR}/some_source.stamp"
OUTPUT ${CMAKE_BINARY_DIR}/some_source.stamp
DEPENDS some_input.c
COMMENT "Generating code ..."
)
add_custom_target(generate_something DEPENDS ${CMAKE_BINARY_DIR}/some_source.stamp)
add_dependencies(some_lib generate_something)
set_source_files_properties(some_source.c PROPERTIES OBJECT_DEPENDS ${CMAKE_BINARY_DIR}/some_source.stamp)
add_executable(main main.c)
target_link_libraries(main PRIVATE some_lib)
A simple main.c
#include <stdio.h>
int get_value();
int main(int argc, char *argv[]) {
printf("The value is %d.", get_value());
}
some_source.c and some_input.c have the same content. Just toggled the return value in some_input.c to test out.
int get_value(){
return 4;
}

Related

Why CMake add_custom_command with DEPFILE reruns every time?

I'm trying to write my cmake script as follows:
set(OUTPUT_FILE "${CMAKE_BINARY_DIR}/static_init/generated/${target}/static_init.cpp")
set(DEP_FILE "${CMAKE_BINARY_DIR}/static_init/input/${target}.d")
write_file(${DEP_FILE} "${OUTPUT_FILE}: ")
add_custom_command(
OUTPUT ${OUTPUT_FILE}
COMMAND ${Python3_EXECUTABLE} myscript.py
DEPFILE ${DEP_FILE}
VERBATIM)
I expect, that I will populate .d file during my script first launch and later custom command will be rerun only when one of files listed after colon will change.
Instead command is running during every compilation, even with empty dependencies list. Printing my own sources during cmake run also leads me to constant command rerunning. What am I doing (or understanding) wrong?
CMake 3.18.5
Ninja
Clang
Ninja removes depfiles by default after reading them. This would cause the behavior you are seeing.
Problem was that DEPFILE option and target name inside depfile must be relative to build dir, not absolute paths.
ninja -d explain helped me to find it.

CMake add_custom_command runs everytime

I am using a custom command for generating C++ Lexers and Parsers from ANTLR4 grammars. Right now I have the following:
set(MY_PARSER_INC
${PROJECT_SOURCE_DIR}/Headers/MyParser/MyLexer.h
${PROJECT_SOURCE_DIR}/Headers/MyParser/MyParser.h
${PROJECT_SOURCE_DIR}/Headers/MyParser/MyParserBaseVisitor.h
${PROJECT_SOURCE_DIR}/Headers/MyParser/MyParserVisitor.h
)
set(MY_PARSER_SRC
${PROJECT_SOURCE_DIR}/Sources/MyParser/MyLexer.cpp
${PROJECT_SOURCE_DIR}/Sources/MyParser/MyParser.cpp
)
add_custom_command(
OUTPUT ${MY_PARSER_INC} ${MY_PARSER_SRC}
DEPENDS ${PROJECT_SOURCE_DIR}/Grammars/MyGrammar.g4
COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_SOURCE_DIR}/Headers/MyParser/
COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_SOURCE_DIR}/Sources/MyParser/
COMMAND java -cp "${ANTLR_CLASSPATH}" "org.antlr.v4.Tool" -Dlanguage=Cpp -visitor -no-listener -package MY::NESTED::NAMESPACE -encoding iso-8859-1 -o ${PROJECT_SOURCE_DIR}/Sources/MyParser/ ${PROJECT_SOURCE_DIR}/Grammars/MyGrammar.g4
COMMAND ${CMAKE_COMMAND} -E copy ${PROJECT_SOURCE_DIR}/Sources/MyParser/*.h ${PROJECT_SOURCE_DIR}/Headers/MyParser/
COMMAND ${CMAKE_COMMAND} -E remove -f ${PROJECT_SOURCE_DIR}/Sources/MyParser/*.h
COMMENT "generating sources for My Parser"
)
and then I use the output files in my add_library command to maintain the dependency.
This does exactly what I expect it to do. It creates the lexers and parsers correctly. It maintains the dependency between those sources and the target library also correctly. Only one problem: It runs every time! Even when the grammar file has not been modified (I checked the file date on the grammar and generated lexers/parsers to be sure)! I have seen some similar questions online but still can't figure out why this is happening.
Any clue?!
EDIT1:
Adding more info, since it may still be unclear.
After the add_custom_command I have the following:
include_directories(${PROJECT_SOURCE_DIR}/Headers/MyParser/)
add_library(MyLibrary SHARED
${MY_PARSER_INC} ${MY_PARSER_SRC}
other_files.hpp other_files.cpp)
which I assume creates a direct dependency between the generated source files and my target library.
Here's my guess.
You need to make a custom target.
https://cmake.org/cmake/help/latest/command/add_custom_target.html?highlight=custom_target
add_custom_target(custom_target_create_parser_code DEPENDS
${MY_PARSER_INC} ${MY_PARSER_SRC}
)
This will make a target that depends on the custom command you wrote.
Now cmake has something to attach your command to.
Now you need to add the dependency to your static library you mentioned.
https://cmake.org/cmake/help/latest/command/add_dependencies.html
add_dependency(your_static_library custom_target_create_parser_code)
Now cmake shouldn't re-run your custom command code everytime.
EDIT BTW:
Craig Scott's book actually has a subchapter devoted to this topic:
https://crascit.com/professional-cmake/
Chapter 18 Custom Targets
Chapter 18.3 Commands That Generate Files
EDIT #2:
If all else fails try the official cmake discourse:
https://discourse.cmake.org/
Your question is well formed and cmake developers look their for questions.
So you should get an answer.

CMake call add_subdirectory within custom command

I'm working with a code generator that produces C++ and a CMakeLists.txt file, unfortunately I cannot use this in my main CMakeLists.txt file for testing purposes.
For example you have the following CMakeLists.txt file:
project(SomeProject CXX C)
add_custom_command(OUTPUT ${SRCS}
COMMAND ${CODEGEN_CLI_PATH} -i "${INPUT}" -o "${OUT}"
COMMENT "Generating sources"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
VERBATIM
)
add_custom_target(CODEGEN
DEPENDS
${SRCS}
)
# Needs to be executed after the custom command
add_subdirectory(${GENERATED_CMAKE_LISTS_LOCATION})
Is it possible to use functions such as add_subdirectory only after you execute custom commands for a particular target, such as CODEGEN?
I've already tried to execute it by adding an extra line to the existing custom command:
COMMAND ${CMAKE_COMMAND} -D DIR=${GENERATED_CMAKE_LISTS_LOCATION} -P add_subdirectories.cmake
Unfortuantly this doesn't work because it isn't allowed to execute functions like add_subdirectory in script mode.
Neither I can manage to call custom made functions (that are executing add_subdirectory) from add_custom_command that are located in the same file.
Nope, it is not possible. The add_subdirectory command is run during configuration step, while CODEGEN is a target that runs during build.
You seem to be doing something wrong, so the only advice I can give you is to use execute_process command to run commands you need. The execute_process command is executed during configuration stage, so it will be able to generate files you need before add_subdirectory.
But again, please describe your problem, why do you want CMake to do that.
I have a huge fixed unsigned char array that I compiled into a static library. The way I work around it is by:
if(NOT EXISTS ${PATH_TO_FOLDER}/smeagol.a)
add_subdirectory(smeagol)
endif()
I'm still looking for a nicer kung-fu way to do it using cmake. I feel that its out there, and I will update this answer once i find it.

Does CMake's add_custom_command delete its first output sometimes?

In my CMake file I have a custom command which calls some external code generation. The generated files are stubs for the user to put his/her own code into. Thus, the code generation makes sure not to overwrite already existing files. My custom command looks like this:
set(generatedOnceSrc foo.h foo.cpp bar.h bar.cpp)
add_custom_command(
OUTPUT ${generatedOnceSrc}
COMMAND ${generateCmd}
VERBATIM
)
add_executable(myProg ${generatedOnceSrc} ${frameworkSrc})
Now I've noticed that at least sometimes the first element of ${generatedOnceSrc} (foo.h in this case) becomes deleted and regenerated. Of course, that will cause trouble once the user started to edit the generated foo.h.
The documentation of add_custom_command does not mention such a behavior but also does not deny it. So my question is:
Am I right with my observation?
Note, that I already found a workaround like this:
set(generatedOnceSrc foo.h foo.cpp bar.h bar.cpp)
set_source_files_properties(generatorOutput PROPERTIES SYMBOLIC true)
add_custom_command(
OUTPUT generatorOutput
BYPRODUCTS ${generatedOnceSrc}
COMMAND ${generateCmd}
VERBATIM
)
add_custom_target(generatorTarget DEPENDS generatorOutput)
add_executable(myProg ${generatedOnceSrc} ${frameworkSrc})
add_dependencies(myProg generatorTarget)
Unfortunately, this calls ${generateCmd} on every build, because the symbolic file generatorOutputis always considered out-of-date and CMake does not check the existence of ${generatedOnceSrc} any more.
Versions
Generator: Unix Makefiles
CMake 3.5.1
Ubuntu 16.04
Edit
Forgot to mention that I observed this behavior with the Unix Makefiles generator, which is used by default on my Ubuntu 16.04 system.
Editing auto-generated files is a bad idea, generally.
If you want to generate your files only once, then consider using execute_process to generate them during configuration step, not the build one.
Otherwise, if you really need them to be generated during build, you'd better arrange thing such way that user code can be plugged in without editing these files.

Trying to trigger a minimal rebuild with CMake and set_source_files_properties

I've got my problem isolated to a very small two-sources project here: https://github.com/ennorehling/versioning/tree/v0.1
What I am trying to do is to not have a hard-coded version number in version.c, but to feed the version number into my build process from an external variable. See s/build for my build process: If $VERSION is defined, I want that to be the version number that the program prints. I achieve this by executing the command VERSION=1.2 s/build form the shell (or, if VERSION is undefined, by using the most recent tag from git).
The problem with this is that I run cmake .. -DVERSION=$VERSION every time I build, and that re-creates the Makefile, which causes a full rebuild. I would like to only rebuild version.o, since it is the only source file that depends on the version number. Building main.o (and in a real project, every other object, too) is unnecessary. I originally used add_definitions, which would add the -D compile switch to every source, and I thought set_source_files_properties was how I would be able to solve this, but since both object files are generated out of the same generated Makefile in build/CMakeFiles/version.dir/build.make, which gets touched by the cmake process. It seems that make errs on the safe side and just rebuilds everything?
Maybe I am barking up the wrong tree entirely, and just haven't found the correct CMake command for this, I don't know. Maybe there are other, proven ways to achieve what I'm trying to do? At this point, I've spent so much time on this, I'm not ashamed to ask for help.
I have found a way to work around this. The root problem here is that CMake create a Makefile for each library or executable target, and since my code only has one target, it was touching that Makefile every time. So the answer is to make a library target for version.c!
add_library(version OBJECT version.c)
add_executable(hello main.c $<TARGET_OBJECTS:version>)
this creates separate files in build/CMakeFiles/version.dir and build/CMakeFiles/hello.dir, and changes to the VERSION number only affect one of them, and the single target in it. The executable's dependencies are unchanged and don't get rebuilt, and only the linker step is executed, cutting down my build times as desired.
It's a bit clunky, but it works for me. New version of the project is here:
https://github.com/ennorehling/versioning/tree/v1.0
Edit: it turns out that Ubuntu Precise doesn't have CMake 2.8.8 yet, and the OBJECT option to add_library needs it. So instead, I have to actually create a library from that single object file and link it, like this:
add_library(version version.c)
add_executable(hello main.c)
target_link_libraries(hello version)
Not a huge deal, but a little annoying.
In your case you don't need to reconfigure every time. Once the make/build environment is written the problem can be reduced to a simple "has the file changed" problem that is checked by make itself.
So I probably would just generate the version.c inside your build script (and add this file to your .gitignore or alternatively directly generate it into the build directory):
s/build
#!/bin/sh
if [ -z "$VERSION" ]; then
VERSION=$(git describe --tags --match "v*.*")
if [ -z "$VERSION" ]; then
VERSION=1.0.0
fi
fi
echo "const char *version(void) { return \"${VERSION}\"; }" > version.c~
if cmake -E compare_files version.c~ version.c
then
cmake -E remove version.c~
else
cmake -E rename version.c~ version.c
fi
if ! [ -d build ]; then
cmake -E make_directory build
cmake -H. -Bbuild
fi
cmake --build build
CMakeLists.txt
cmake_minimum_required(VERSION 2.8)
project(version C)
add_executable(version version.c main.c)
version.c
const char *version(void) { return "1.0.0"; }
main.c
#include "stdio.h"
const char *version(void);
void main(void) {
puts(version());
}
This seems to be a bug in the CMake Unix Makefile generator. I tried your (original) code example using the Ninja generator, and ninja manages to avoid rebuilding main.c if you only change the version number.