I have python script that is run through add_custom_command in my project, this script analyzes target sources and generate additional one for the target.
get_target_property(TARGET_SOURCES ${target} SOURCES)
add_custom_command(
OUTPUT ${CMAKE_BINARY_DIR}/static_init/generated/${target}/static_init.cpp
COMMAND ${Python3_EXECUTABLE} myscript.py
DEPENDS ${TARGET_SOURCES}
VERBATIM)
target_sources( "${target}" PRIVATE "${CMAKE_BINARY_DIR}/static_init/generated/${target}/static_init.cpp" )
I need my script to rerun when one of sources becomes "dirty" (e.g. one of included headers is changed, probably this file is in another target and it is not guaranteed that another target will be rebuild on header change). As we use Ninja, IMPLICIT_DEPENDS isn't available for us, so it looks DEPFILE property is the way to do it, but I cannot understand, how can I get one for all ${TARGET_SOURCES}?
It doesn't seem to me that ninja will generate something like that automatically, I cannot write it by hands, as it will require to recursively analyze all includes during cmake run. Or maybe there is another way to solve this problem?
To get dependencies of a source you have to compile it. Just depend on the target. I would do:
add_custom_target(target_objects OBJECT all_source_files_for_your_target)
add_custom_command(
OUTPUT ${CMAKE_BINARY_DIR}/static_init/generated/${target}/static_init.cpp
COMMAND ${Python3_EXECUTABLE} myscript.py
DEPENDS $<TARGET_OBJECTS:target_objects>
VERBATIM
)
add_custom_library/executable("${target}" PRIVATE
${CMAKE_BINARY_DIR}/static_init/generated/${target}/static_init.cpp
)
link_target_libraries("${target}" PRIVATE target_objects)
That way the dependencies are linear.
Related
This is a follow-up to How can I get a target's output-file's name during CMake's configuration phase (not the generation phase)?
I want to achieve something similar to the below:
add_custom_command(
OUTPUT ${CMAKE_BINARY_DIR}/to_zip/$<TARGET_FILE_NAME:Foo>
DEPENDS Foo
COMMAND ${CMAKE_COMMAND} -E copy $<TARGET_FILE:Foo>
${CMAKE_BINARY_DIR}/to_zip/$<TARGET_FILE_NAME:Foo>)
list(APPEND ALL_FILES_IN_TO_ZIP
${CMAKE_BINARY_DIR}/to_zip/$<TARGET_FILE_NAME:Foo>)
add_custom_command(
OUTPUT ${CMAKE_BINARY_DIR}/myzip.zip
DEPENDS ${ALL_FILES_IN_TO_ZIP}
COMMAND <zip everything up in ${CMAKE_BINARY_DIR}/to_zip>)
add_custom_target(create-zip DEPENDS ${CMAKE_BINARY_DIR}/to_zip/myzip.zip)
Basically, when I invoke the create-zip target, I want to copy a lot of files to ${CMAKE_BINARY_DIR}/to_zip, and then zip up everything in it. The issue with the above code is that I am not allowed to use target based generator expression for OUTPUT in add_custom_command, so the above code doesn't work. From the CMake docs:
New in version 3.20: Arguments to OUTPUT may use a restricted set of generator expressions. Target-dependent expressions are not permitted.
Is there a way to workaround this problem to make it work?
I'm pretty sure you don't need to use the name of the target's output file as the OUTPUT of your custom command which does the copy. All you're really using the OUTPUT field for is CMake's dependency mechanisms (where it tracks what things have changed and does things again if anything in DEPENDS has changed (modification time in the filesystem)). You can just create a dummy file that you touch (update the modification time for in the filesystem) whenever the target has changed and once its copy has been made.
set(targets target_foo target_bar target_baz)
foreach(target "${targets}")
set(indicator "${CMAKE_BINARY_DIR}/to_zip/rezip_indicators/${target}")
add_custom_command(
OUTPUT "${rezip_indicator_file}"
DEPENDS "${target}"
COMMAND "${CMAKE_COMMAND}" -E copy $<TARGET_FILE:${target}>
"${CMAKE_BINARY_DIR}/to_zip/$<TARGET_FILE_NAME:${target}>"
COMMAND "${CMAKE_COMMAND}" -E touch "${rezip_indicator_file}"
)
list(APPEND NEEDS_REZIP_INDICATOR_FILES "${rezip_indicator_file}")
endforeach()
add_custom_command(
OUTPUT "${CMAKE_BINARY_DIR}/myzip.zip"
DEPENDS "${NEEDS_REZIP_INDICATOR_FILES}"
COMMAND <zip everything up in ${CMAKE_BINARY_DIR}/to_zip>
)
add_custom_target(create-zip DEPENDS "${CMAKE_BINARY_DIR}/to_zip/myzip.zip")
Or something to that effect.
The indicator file will get its timestamp updated by the touch command in the custom command when the target file has changed. The target file and the needs-rezip indicator file get "synced" when the custom command runs, which will be whenever you run the buildsystem after the target has gotten rebuilt. At least- I'm pretty sure that's how things work.
I want to collect object files (.o) in a particular directory with CMake
I have tried the below snippet from How to collect object files (.o) in a particular directory with CMake?
add_custom_command(TARGET ${LIBNAME}
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E make_directory ${OBJ_DIR}
COMMAND ${CMAKE_COMMAND} -E copy_if_different
$<TARGET_OBJECTS:${LIBNAME_OBJ}> ${OBJ_DIR})
The command copy_if_different works well when the library contains less number of sources and fails when it's more.
How to collect objects of a project which contains huge number of sources ?
For add_custom_command to execute its COMMAND arguments, something has to create a process environment for the command to execute in. Whether that's done by a command shell or directly using exec(), the size limit for that environment is defined by the limits.h macro ARG_MAX. On POSIX systems, getconf ARG_MAX will show the compile-time value of that configuration. On my Fedora 35 system, the value is 2097152 which seems almost impossible to exceed, but maybe you're dealing with smaller limits.
(Also, the ARG_MAX limit represents the entire environment, not just command arguments. So if there are a lot of environment variables defined, that could eat into your max length.)
If you truly are exceeding that, you have two choices:
Option 1: Install the objects in the install stage, as intended
If I'm understanding the linked q&a correctly, the only reason you need the $<TARGET_OBJECTS> paths is so that you can copy them somewhere else in the build tree, in order to then install them from there?
If you just want to install them, a simpler solution is to write custom install code, e.g.
# necessary because config variables aren't present
# in the install context.
install(CODE "set(OBJECT_DEST \"${CMAKE_INSTALL_LIBDIR}\")"
install(CODE [[
file(INSTALL
DESTINATION "${CMAKE_INSTALL_PREFIX}/${OBJECT_DEST}"
FILES
$<TARGET_OBJECTS:_actual_object_name_>
)]])
Messing around with keeping the object name in a variable is a mistake when using custom install code, IMHO, as the evaluation contexts get too confusing. Instead of ${LIBNAME_OBJ}, just use the actual target name in the generator expression.1
Since the generator-expression expansion is handled by CMake when generating the cmake_install.cmake script, there shouldn't be any limit on the length it can have. (Not beyond whatever limits it has internally, if any.)
Option 2: Break up your targets
If you really want to use add_custom_command with the objects, you'll need smaller targets. Why do all of your sources have to be defined on a single object library? Just break them up:
set(SOURCES1 a.c b.c c.c d.c)
set(SOURCES2 w.c x.c y.c z.c)
add_library(s1_obj OBJECT ${SOURCES1})
add_library(s2_obj OBJECT ${SOURCES2})
add_executable(foo
$<TARGET_OBJECTS:s1_obj>
$<TARGET_OBJECTS:s2_obj>
)
add_custom_target(copy_all_objs
COMMAND ${CMAKE_COMMAND} -E make_directory ${OBJ_ROOT_DIR}/myobjects/
COMMAND ${CMAKE_COMMAND} -E copy_if_different
$<TARGET_OBJECTS:s1_obj>
${OBJ_ROOT_DIR}/myobjects/
COMMAND ${CMAKE_COMMAND} -E copy_if_different
$<TARGET_OBJECTS:s2_obj>
${OBJ_ROOT_DIR}/myobjects/
COMMAND_EXPAND_LISTS)
Notes
Actually, I typically recommend using real target names whenever possible in generator expressions.
With some targets — at least SHARED, STATIC, MODULE and ARCHIVE libraries, plus executables — the target name defines the default output file name. So, I get using a variable for the name. (Though, I did say default — it can be overridden with the OUTPUT_NAME target property, and often that's a better way to make filenames configurable. And it lets you keep static target names.)
But for targets like INTERFACE, OBJECT, and ALIAS libraries, the target name is meaningless — it's only used internally in the CMake code. There's no reason not to just hardcode it.
If you're doing this in a macro/function and the target name has to be in a variable, your best bet is probably to write the install() code as a quoted argument rather than a bracket argument, so that variables get evaluated immediately. Which means lots of fun escaping, for the things you don't want evaluated immediately. (Though you can lose the OBJECT_DEST variable, at least.):
install(CODE "
file(INSTALL
DESTINATION
\"\$\{CMAKE_INSTALL_PREFIX\}/${CMAKE_INSTALL_LIBDIR}\"
FILES
$<TARGET_OBJECTS:${LIBNAME_OBJ}>
)")
I am trying to generate file that depends on a set of files that can
change throughout different make invocations.
To understand it better, let's show you the code:
cmake_minimum_required(VERSION 2.8)
project(demo-one C)
add_custom_command(
OUTPUT
"${CMAKE_BINARY_DIR}/generated.c"
COMMAND
generate -o "${CMAKE_BINARY_DIR}/generated.c"
DEPENDS
"$(shell generate-dependencies-list)"
COMMENT
"Generating generated.c"
)
add_executable(main main.c "${CMAKE_BINARY_DIR}/generated.c")
So, I want to generate the file generated.c with the generate
command and this files needs to be regenerated when the files
specified by generated-dependencies-list command changes. As you may
notice, generated-dependencies-list can generate different set of
files throughout make invocations, so is not feasible to get
the result of generated-dependencies-list at configure time to then
pass the result to add_custom_command.
Actually the above code somewhat works, but it looks like a hack that
will only work for Makefile backend, also the make rule doesn't look
as what I'm expecting, after all, it's a hack:
generated.c: ../$(shell\ generate-dependencies-list)
Basically, I want this rule or something to get the same result:
generated.c: $(shell generate-dependencies-list)
Has CMake any feature to achieve this?
when the files specified by generated-dependencies-list command changes
If the output of the command generated-dependencies-list depends only on this script and script's parameters then you can just add this script to DEPENDS sub-option:
add_custom_command(
OUTPUT
"${CMAKE_BINARY_DIR}/generated.c"
COMMAND
"${CMAKE_CURRENT_LIST_DIR}/generate-dependencies-list"
COMMAND
generate -o "${CMAKE_BINARY_DIR}/generated.c"
DEPENDS
"${CMAKE_CURRENT_LIST_DIR}/generate-dependencies-list"
COMMENT
"Generating generated.c"
)
This question is subsequent to my previous one: How to integrate such kind of source generator into CMake build chain?
Currently, the C source file is generated from XS in this way:
set_source_files_properties(${CMAKE_CURRENT_BINARY_DIR}/${file_src_by_xs} PROPERTIES GENERATED 1)
add_custom_target(${file_src_by_xs}
COMMAND ${XSUBPP_EXECUTABLE} ${XSUBPP_EXTRA_OPTIONS} ${lang_args} ${typemap_args} ${file_xs} >${CMAKE_CURRENT_BINARY_DIR}/${file_src_by_xs}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
DEPENDS ${file_xs} ${files_xsh} ${_XSUBPP_TYPEMAP_FILES}
COMMENT "generating source from XS file ${file_xs}"
)
The GENERATED property let cmake don't check the existence of this source file at configure time, and add_custom_target let the xsubpp always re-run at each compile. The reason for always rerun is because xsubpp will generate an incomplete source file even if it fails, so there are possibility that the whole compiling continues with an incomplete source file.
I found it is time consuming to always re-run source generator and recompile it. So I want to have it re-run only when dependent XS files are modified. However, if I do so, the incomplete generated source file must be deleted.
So my question is: is there any way to remove the generated file, only when the program exit abnormally at compile time?
Or more generic: is there any way to run a command depending on another command's exit status at compile time?
You can always write a wrapper script in your favorite language, e.g. Perl or Ruby, that runs xsubpp and deletes the output file if the command failed. That way you can be sure that if it exists, it is correct.
In addition, I would suggest that you use the OUTPUT keyword of add_custom_command to tell CMake that the file is a result of executing the command. (And, if you do that, you don't have to set the GENERATED property manually.)
Inspired by #Lindydancer's answer, I achieved the purpose by multiple COMMANDs in one target, and it don't need to write an external wrapper script.
set(source_file_ok ${source_file}.ok)
add_custom_command(
OUTPUT ${source_file} ${source_file_ok}
DEPENDS ${xs_file} ${xsh_files}
COMMAND rm -f ${source_file_ok}
COMMAND xsubpp ...... >${source_file}
COMMAND touch ${source_file_ok}
)
add_library(${xs_lib} ${source_file})
add_dependencies(${xs_lib} ${source_file} ${source_file_ok})
The custom target has 3 commands. The OK file only exists when xsubpp is success, and this file is added as a dependency of the library. When xsubpp is not success, the dependency on the OK file will force the custom command to be run again.
The only flaw is cross-platform: not all OS have touch and rm, so the name of these two commands should be decided according to OS type.
I have a CMake project with many subprojects.
Each of them can use a function I provide to generate a small text file with some certain information (by calling add_custom_command).
At the final step I'd like to combine all those files into one big text file.
I've created a custom command which searches for created files (all in one place) and merges them.
The problem is that I'd like to make this final step dependend on all of small steps being made in subprojects while I don't actually know how many files will be provided.
My final command looks like:
add_custom_command(OUTPUT combination.txt
COMMAND create combination.txt from all files from /path/)
and my create-small-text-file-for-each-subproject command looks like:
add_custom_command(OUTPUT /path/${sub_project_name}.txt
COMMAND create /path/${sub_project_name}.txt)
And when I create those small files I'd like to do something like to make "combination.txt" depend on /path/${sub_project_name}.txt
So I wish I could:
add_dependency(combination.txt /path/${sub_project_name}.txt)
However this only works for targets.
I've also tried to use set_source_files_properties with OBJECT_DEPENDS, but it seems to doesn't work (maybe its intend to be used with add_target's cpp files ?)
The last way to get it work I see is to use a cache variable which would accumulate all those small files paths and then use it like this:
add_custom_command(OUTPUT combination.txt
COMMAND create combination.txt from all files from /path/
DEPENDS ${all_small_files_list})
but this is the last thing I want to do.
Instead of using add_custom_command you could use add_custom_target with a correct dependency-definition (so that is it not built every time).
add_custom_target(project
COMMAND touch project.txt)
add_custom_target(project2
COMMAND touch project2.txt)
add_custom_target(combination
COMMAND cat project.txt project2.txt > combination.txt)
add_dependencies(combination project2)
add_dependencies(combination project)
add_executable(t t.c)
add_dependencies(t combination.txt)
Again: make sure you're using the DEPENDS argument of add_custom_target to create a real dependency chain so that a project-target and thus the combination-target gets out of date.
UPDATE: I was too premature. In fact cmake (at least up to 2.8.9) works as follows for dependencies: with a call to add_dependencies you cannot add a dependency which is the OUTPUT of a custom command IOW a (generated) file. With add_dependencies you can only add target as created by add_custom_target. However in a add_custom_target you can depend on an output of add_custom_command by using the DEPENDS-directive. That said this makes it work:
add_custom_command(OUTPUT project.txt
COMMAND uptime >> project.txt MAIN_DEPENDENCY t2.c)
add_custom_target(project DEPENDS project.txt)
add_custom_target(combination
COMMAND cat project.txt project2.txt > combination.txt)
add_dependencies(combination project)
This will make the combination target always be regenerated as it has no MAIN_DEPENDENCY or DEPENDS, but the usage of add_dependencies is allowed.