From the question Defining lex include files for (f)lexers we are already knowing that there is no lex distribution known that has an include file facility for e.g. rules into a lex file.
The suggestion there was to write an own pre-processor.
When defining an own pre-processor one generates, of course, dependencies on the included file(s), so when such a file is changed one wants to do a rebuild of the "lex" file.
With C / C++ the dependencies are handled (to the best of my knowledge) automatically (probably by some CMake script).
How to accomplish the dependency rules for the file(s) included in a lex file?
Edit
Say I have an original .l file configimpl.l that includes the files aa.inc and bb.inc, by converting the original .l file (${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l) by means of a python script into a new .l (${GENERATED_SRC}/${lex_file}.l) file (and we also have an output file ${GENERATED_SRC}/${lex_file}.corr but that is not relevant here).
Of course all "cmake" variables etc. have been set properly.
My code for the preprocessing (i.e. including the include files into the .l file) is:
add_custom_command(
COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_LIST_DIR}/pre_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l ${GENERATED_SRC}/${lex_file}.l ${GENERATED_SRC}/${lex_file}.corr ${CMAKE_CURRENT_LIST_DIR}
DEPENDS ${CMAKE_CURRENT_LIST_DIR}/pre_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l ${LEX_INC_FILES}
OUTPUT ${GENERATED_SRC}/${lex_file}.corr ${GENERATED_SRC}/${lex_file}.l
)
and this works fine, except when the aa.inc or the bb.inc changes the "generated included" ${GENERATED_SRC}/${lex_file}.l is not rebuild.
Based on the question: Dynamic dependency within custom source files I already tried to create something, but failed.
add_custom_command(
COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_LIST_DIR}/dep_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l ${GENERATED_SRC}/${lex_file}.d ${GENERATED_SRC}/${lex_file}.l ${CMAKE_CURRENT_LIST_DIR}
DEPENDS ${CMAKE_CURRENT_LIST_DIR}/dep_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l
OUTPUT ${GENERATED_SRC}/${lex_file}.d
)
set_source_files_properties(${GENERATED_SRC}/${lex_file}.d PROPERTIES GENERATED 1)
add_custom_command(
COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_LIST_DIR}/pre_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l ${GENERATED_SRC}/${lex_file}.l ${GENERATED_SRC}/${lex_file}.corr ${CMAKE_CURRENT_LIST_DIR}
DEPENDS ${CMAKE_CURRENT_LIST_DIR}/pre_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l
DEPFILE ${GENERATED_SRC}/${lex_file}.d
OUTPUT ${GENERATED_SRC}/${lex_file}.l ${GENERATED_SRC}/${lex_file}.corr
)
So I first run a dependency detection step that creates a file ${GENERATED_SRC}/${lex_file}.d with as content:
${GENERATED_SRC}/generated_src/configimpl.l: ${CMAKE_CURRENT_LIST_DIR}/aa.inc
${GENERATED_SRC}/generated_src/configimpl.l: ${CMAKE_CURRENT_LIST_DIR}/bb.inc
in this generated file the paths are, of course, set to the full paths.
After this the original preprocessing step is run but now with the extra DEPFILE ${GENERATED_SRC}/${lex_file}.d that should signal that there are some extra dependencies.
When I change the aa.inc or the bb.inc the ${GENERATED_SRC}/${lex_file}.l is not rebuild, what did I miss?
Used CMake version: cmake version 3.22.1
used generator: NMake Makefiles
Make the DEPFILE output as one long line, then it works the best. From memory, I think the parser for the depfile is not smart enough to handle multiple lines. When in doubt - see the output of -MT -MD of gcc, the format should be the same.
Related
I am using cmake to build a paper, which has an involved build process.
Build an exe
run the exe to generate some data (in ${PROJECT_BINARY_DIR}/data)
run some python to process the data into figures (Postscript images). These are saved in ${CMAKE_SOURCE_DIR}/fig.
convert images to png
build a latex document with those figures
This can all be done in cmake by chaining the outputs of custom_commands to custom targets:
#This is the command that does the work (run python, or some executable etc.). Will only run if ${CUSTOM_DEPENDS} is newer that ${CUSTOM_OUTPUT}
add_custom_command( OUTPUT ${CUSTOM_OUTPUT}
COMMAND ${CUSTOM_COMMAND}
DEPENDS ${CUSTOM_DEPENDS}
WORKING_DIRECTORY ${CUSTOM_WORKING_DIRECTORY}
USES_TERMINAL
VERBATIM
)
#This target allows one to depend on the custom command from another file
add_custom_target(${name}
DEPENDS ${CUSTOM_OUTPUT}
VERBATIM
)
The issue is that when I am actually working on this, I generate the figures on one machine, save the Postscript images files to the git repository, and then work on the actual latex document on a laptop. The laptop can't handle running the executable, and honestly I would like to have some fairly tight controls in general about if and when a figure is regenerated. Ideally the Postscript files would only build if they are missing. It would also be nice if the Postscript files rebuilt if the dependencies (e.g. the executable source files) changed since first config, but that may be asking too much.
This is easy enough to do in config time, but doesn't really the case where a file is deleted post-config.
I've tried something like this (where the output is the Postscript files):
#same custom command as before, but we also generate a copy. This copy should always be newer that the output
add_custom_command( OUTPUT ${CUSTOM_OUTPUT}
COMMAND ${CUSTOM_COMMAND}
COMMAND ${CMAKE_COMMAND} -E copy ${CUSTOM_OUTPUT} ${CMAKE_CURRENT_BINARY_DIR}/copy/
DEPENDS ${CUSTOM_DEPENDS}
WORKING_DIRECTORY ${CUSTOM_WORKING_DIRECTORY} #usually this is ${CMAKE_CURRENT_LIST_DIR}
${USES_TERMINAL}
VERBATIM
)
#for every output make a command that makes the copy depends on the original output. So that they are always up to date if they exist.
foreach(arg IN ITEMS ${CUSTOM_OUTPUT})
if(EXISTS ${COPY_OBJ})
file(COPY ${arg} DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/copy/")
endif()
list(APPEND CUSTOM_TARGET_DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/copy/${arg}")
add_custom_command( OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/copy/${arg}
DEPENDS "${arg}"
)
endforeach()
#now have the target depend on the copies which are always up to date with the original outputs
add_custom_target(${name}
${CUSTOM_ALL}
DEPENDS ${CUSTOM_TARGET_DEPENDS}
VERBATIM
)
This doesn't work, and I'm not sure if this is because these operations are occurring in a source folder, or if I'm misunderstanding of add_custom_command up-to-date logic.
Note: This is different from This Question where they only test for existence. I'm trying to change up-to-dateness with existence.
I have a test code for my c project, which is built as part of my cmake project.
In order to run properly, the SW needs an input binary file.
The binary file is created from a designated .json file (which is in the test's directory) using a separate executable which is also part of the same cmake project.
Before the test is run (i.e. when building it) I need the bin-building executable to be built (easy enough with target dependency), but I also need the resulting executable to run with my .json as input.
What cmake functions enable this?
I tried building a custom command with the binary as output, but the test target doesn't know it needs the binary in order to run, can't accept the binary as a source, and a simple "add_dependancy" resulted in errors.
function(create_binary input_json dst_bin)
if (IS_WINDOWS)
set(exe_path ${OUTPUT_BIN_DIR}/BinaryGenerator.exe)
elseif(IS_LINUX)
set(exe_path ${OUTPUT_BIN_DIR}/BinaryGenerator)
endif()
add_custom_command(OUTPUT ${dst_bin}
COMMAND ${exe_path} ${input_json} ${dst_bin}
DEPENDS ${exe_path} ${dst_bin}
COMMENT STATUS "Creating bin file ${dst_bin} from ${input_json}"
)
endfunction(create_binary)
create_binary(InputParams.json ${OUTPUT_BIN_DIR}/InputParams.bin)
add_executable(MySwTest TestFile.cpp TestFile.h)
add_dependencies(${OUTPUT_BIN_DIR}/InputParams.bin)
I expected cmake to place a target dependency between my test target and the bin creator target, and also on the existence of the test's bin file, thus running the custom command in order to create it.
The actual output from cmake is "The dependency target InputParams.bin doesn't exist". I assume this is because "add_dependency" is meant to work only with targets.
Would adding a custom target including my binary solve this?
This is indeed solvable by defining a custom target that depends on the resulting binary.
When adding a custom command, the actual command is only called if the command's OUTPUT file is needed for another target - so if it's a source file, it will be called.
My command's output is a generated binary, and therefore is not a source file for my target.
However, using add_custom_target() I can create a target dependent on said bin file, and create a dependency between the end target and my custom target.
The working code for the above looks like this:
set(PARAMS_NAME InputParams)
set(INPUT_JSON ${PARAMS_NAME}.json)
set(DST_BIN ${PARAMS_NAME}.bin)
if (IS_WINDOWS)
set(EXE_PATH ${OUTPUT_BIN_DIR}/BinaryGenerator.exe)
elseif(IS_LINUX)
set(EXE_PATH ${OUTPUT_BIN_DIR}/BinaryGenerator)
endif()
add_custom_command(OUTPUT ${DST_BIN}
COMMAND ${EXE_PATH} ${INPUT_JSON} ${DST_BIN}
DEPENDS BinaryGenerator ${INPUT_JSON}
COMMENT STATUS "Creating bin file ${DST_BIN} from ${INPUT_JSON}"
)
add_custom_target(${PARAMS_NAME}BinGenerated DEPENDS ${DST_BIN})
add_executable(MySwTest TestFile.cpp TestFile.h)
add_dependencies(MySwTest ${PARAMS_NAME}BinGenerated)
By adding a dependency between MySwTest and my custom target, I ensure that the command generating my binary is called for each build.
We are porting some rather old code and of course we want to use generator expressions now.
The by configure_file generated .pc files now contain -I$<INSTALL_INTERFACE:include>.
The only hint about how to resolve generator expressions I found was to use
file(GENERATE
Of course this is executed during the configure step so the above expression is resolved to an empty string.
Edit:
here is an example
CMakeLists.txt:
cmake_minimum_required(VERSION 3.11)
project(test CXX)
add_library(foo SHARED main.cpp)
target_include_directories(foo PUBLIC $<INSTALL_INTERFACE:include>)
# now later buried deep in some functions
get_property( _include_dirs TARGET foo PROPERTY INCLUDE_DIRECTORIES )
configure_file(config.in config.out #ONLY)
# content of config.out is "include = -I$<INSTALL_INTERFACE:include>"
file(GENERATE OUTPUT config.out2 INPUT ${CMAKE_CURRENT_BINARY_DIR}/config.out)
# content of config.out2 is "include = -I"
# most likely because the INSTALL_INTERFACE isn't used when the file is generated
config.in:
include = #_include_dirs#
and main.cpp is just empty.
As the CMake documentation states the file (GENERATE ...) command can use generator expressions which are evaluated by the generator.
Generate Files
You can let CMake generate files with custom code directly without the workaround of a configure_file command.
Single-Config Generators
For single-config generators like Makefiles you can use:
file (GENERATE
OUTPUT "config.out"
CONTENT "include = -I$<INSTALL_INTERFACE:include>"
)
As my CMake code has to work with both multi-config and single-config generators I did not test the code specifically.
Single- And Multi-Config Generators
In general for all generators, one can use the following signature:
file (GENERATE
OUTPUT "config_$<CONFIG>.out"
CONTENT "include = -I$<INSTALL_INTERFACE:include>"
)
Due to the nature of the command in relation to multi-config generators like Visual Studio this will generate multiple files in your build folder for each build type specified in this variable CMAKE_CONFIGURATION_TYPES:
config_Debug.out
config_Release.out
config_RelWithDebInfo.out
...
config_< BUILD_TYPE >.out
If you do not specify unique filenames and CMake tries to generate the files it will stop the execution with an error.
Use Generated Files
To use the previously generated files it depends on what you need. First of all the files will exist after the configuration stage.
Single-Config Generators
To use a generated file in a single-config generator scenario with a constant name (e.g. config.out) there should be no additional work necessary.
Single- And Multi-Config Generators
For multi-config generators it is slighty different. As you have to use generator expressions to access the appropriate file at build time. If you have a CMake instruction that supports generator expressions then you can just use the filename config_$<CONFIG>.out.
But if you need the file to be named exactly the same regardless of the build type (like config.out) it gets a little more tricky.
First you have to tell CMake that there should be a file named like config.out by using add_custom_command and specifiying the OUTPUT parameter:
add_custom_command (
COMMAND ${CMAKE_COMMAND} "-E" "copy_if_different" "config_$<CONFIG>.out" "config.out"
VERBATIM
PRE_BUILD
DEPENDS "config_$<CONFIG>.out"
OUTPUT "config.out"
COMMENT "creating config.out file ({event: PRE_BUILD}, {filename: config.out})"
)
CMake will create a file dependency internally and every time one references the filename config.out it will ensure that the add_custom_command gets executed.
But this will not work in every case as it depends on the further instructions which should use the file.
Depending on the commands you are using you can now specify the file config.out as input for some commands (like target_sources, ...) and CMake will detect on a file-dependency level that it has to ensure the existence of config.out.
If you want to generate a file which is not referenced on a file-dependency level (like versioninfo.txt) then you have to ensure that CMake executes the add_custom_command every time your build target gets executed via a target-dependency:
add_custom_target ("generate_config_out" DEPENDS "config.out")
add_dependencies ("MY_LIBRARY_TARGET" "generate_config_out")
Every time CMake builds the MY_LIBRARY_TARGET target it will previously build the generate_config_out target which in turn depends on the config.out that CMake will process on the file-dependency level.
I need to generate a couple of files from txt base file. But if these files already exist, NMake Makefiles generator just omits generation command:
CMakeLists.txt:
add_custom_command(OUTPUT "FILE.FI" "FILE.h"
COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/../../Tools/GEN.exe
DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/FILE.txt"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
There are no generated files
Filesystem:
FILE.txt
build.make
...
D:\Path\To\Project\..\..\Tools\GEN.exe
...
Generated files exist
But if FILE.FI and FILE.h exist:
Filesystem:
FILE.txt
FILE.FI
FILE.h
build.make:
...
So, my question is: How to force CMake update existing files?
P.S. I've tried to add
file(REMOVE "FILE.FI" "FILE.h")
to CMakeLists.txt. It works, but seems like hack.
As written in the comments add_custom_target is missing. Set your DEPENDS right and it will only rerun when the dependencies changed. Its a little bit tricky and hard to remember, so i always consult this example. Also check the comments there
I have a list of files that get generated during the CMake build process. I want to compile these files using "add_library" afterward, but I won't know which files get generated until after they get generated. Is there anyway to build this into a CMake script?
Well, I think it is possible, so I'll share what I've done. My problem was that I had to compile several CORBA idls to use as part of a project's source and I didn't want to manually list every file. I thought it would be better to find the files. So I did it like this:
file(GLOB IDLS "idls/*.idl")
set(ACE_ROOT ${CMAKE_FIND_ROOT_PATH}/ace/ACE-${ACE_VERSION})
foreach(GENERATE_IDL ${IDLS})
get_filename_component(IDLNAME ${GENERATE_IDL} NAME_WE)
set(OUT_NAME ${CMAKE_CURRENT_SOURCE_DIR}/idls_out/${IDLNAME})
list(APPEND IDL_COMPILED_FILES ${OUT_NAME}C.h ${OUT_NAME}C.cpp ${OUT_NAME}S.h ${OUT_NAME}S.cpp)
add_custom_command(OUTPUT ${OUT_NAME}C.h ${OUT_NAME}C.cpp ${OUT_NAME}S.h ${OUT_NAME}S.cpp
COMMAND ${ACE_ROOT}/bin/tao_idl -g ${ACE_ROOT}/bin/ace_gperf -Sci -Ssi -Wb,export_macro=TAO_Export -Wb,export_include=${ACE_ROOT}/include/tao/TAO_Export.h -Wb,pre_include=${ACE_ROOT}/include/ace/pre.h -Wb,post_include=${ACE_ROOT}/include/ace/post.h -I${ACE_ROOT}/include/tao -I${CMAKE_CURRENT_SOURCE_DIR} ${GENERATE_IDL} -o ${CMAKE_CURRENT_SOURCE_DIR}/idls_out/
COMMENT "Compiling ${GENERATE_IDL}")
endforeach(GENERATE_IDL)
set_source_files_properties(${IDL_COMPILED_FILES}
PROPERTIES GENERATED TRUE)
set(TARGET_NAME ${PROJECT_NAME}${DEBUG_SUFFIX})
add_executable(
${TARGET_NAME}
${SOURCE}
${IDL_COMPILED_FILES}
)
The GENERATED properties is useful in case one of my idl compilation outputs (*C.cpp, *C.h, *S.cpp and *S.h) is not created, so that the build command doesn't complain that the file doesn't exist.
Well, it is possible to do so with CMake's CMAKE_CONFIGURE_DEPENDS directory property. This forces CMake to reconfigure if any of the given files changed.
Simple solution
The following code shows the approach for a single model file, that is used as input for the code generation:
set(MODEL_FILE your_model_file)
set_directory_properties(PROPERTIES CMAKE_CONFIGURE_DEPENDS ${MODEL_FILE})
set(GENERATED_SOURCE_DIR ${CMAKE_CURRENT_BINARY_DIR}/${MODEL_FILE})
file(REMOVE_RECURSE ${GENERATED_SOURCE_DIR})
file(MAKE_DIRECTORY ${GENERATED_SOURCE_DIR})
execute_process(COMMAND your_code_generation_tool -o ${GENERATED_SOURCE_DIR} ${MODEL_FILE})
file(GLOB LIBGENERATED_FILES ${GENERATED_SOURCE_DIR}/*)
add_library(libgenerated ${LIBGENERATED_FILES})
target_include_directories(libgenerated ${GENERATED_SOURCE_DIR})
With the above approach, each time the model file has changed CMake will reconfigure which results in the model being regenerated.
Advanced solution
The problem with the simple solution is that even for the smallest possible change in the model the entire dependencies of the generated files have to be rebuilt.
The advanced approach uses CMake's copy_if_different feature to let only generated files that are affected by the model change to appear modified which results in better build times. To achieve that we use a staging directory as destination for the generator and sync the contents subsequently with the generator output of the previous compile run:
set(MODEL_FILE your_model_file)
set(GENERATOR_STAGING_DIR ${CMAKE_CURRENT_BINARY_DIR}/${MODEL_FILE}.staging)
set(GENERATOR_OUTPUT_DIR ${CMAKE_CURRENT_BINARY_DIR}/${MODEL_FILE})
set_directory_properties(PROPERTIES CMAKE_CONFIGURE_DEPENDS ${MODEL_FILE})
# Create fresh staging/final output directory
file(REMOVE_RECURSE ${GENERATOR_STAGING_DIR})
file(MAKE_DIRECTORY ${GENERATOR_STAGING_DIR})
file(MAKE_DIRECTORY ${GENERATOR_OUTPUT_DIR})
# Run code generation
execute_process(COMMAND your_code_generation_tool -o ${GENERATOR_STAGING_DIR} "${CMAKE_CURRENT_SOURCE_DIR}/${MODEL_FILE}")
# Remove stale files from final generator output directory
file(GLOB GENERATED_FILES RELATIVE "${GENERATOR_OUTPUT_DIR}/" "${GENERATOR_OUTPUT_DIR}/*")
foreach(FILE ${GENERATED_FILES})
if(NOT EXISTS "${GENERATOR_STAGING_DIR}/${FILE}")
file(REMOVE "${GENERATOR_OUTPUT_DIR}/${FILE}")
endif()
endforeach()
# Copy modified files from staging to final generator output directory
file(GLOB GENERATED_FILES RELATIVE "${GENERATOR_STAGING_DIR}/" "${GENERATOR_STAGING_DIR}/*")
foreach(FILE ${GENERATED_FILES})
execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different "${GENERATOR_STAGING_DIR}/${FILE}" "${GENERATOR_OUTPUT_DIR}")
endforeach()
file(GLOB LIBGENERATED_FILES "${GENERATOR_OUTPUT_DIR}/*")
add_library(libgenerated ${LIBGENERATED_FILES})
target_include_directories(libgenerated PUBLIC ${GENERATOR_OUTPUT_DIR})
If you don't know the name of the files that will be generated, you can "glob" the folders where they reside.
file( GLOB_RECURSE MY_SRC dest_folder/*.cpp )
add_library( libname SHARED ${MY_SRC} )
Now I'm not sure what triggers the generation of these files. The "globbing" will happen only when you manually run cmake: it will not be able to detect automatically that new files are present.
Treat this as a non-answer, just more info:
I recently had to do something for one case where I had a .cpp file that was auto-generated, but I could not figure out how to get CMake to construct the Visual Studio project file that would then compile it. I had to resort to something quite stinky: I had to #include <the_generated.cpp> file from another file that resided under the ${CMAKE_CURRENT_SOURCE} directory. That won't help you much in your case because I suspect you have several .cpp files, so this approach is not scalable.
Also, I found that the GENERATED source file property, when added to the file, did not help at all.
I consider this condition either a bug in Visual Studio (in my case this was VS2008 SP1), or in how CMake generates the .vcproj files, or both.