CMake : add executable with unknown genererated source files - cmake

I have a tool that generates a set source files whose name I am not able to know beforehand.
How to write a proper CMakeLists.txt script for scenario? This question has been asked before here CMake Compiling Generated Files. But it does not have a proper solution.
For instance, in the first answer (https://stackoverflow.com/a/8748478/2912478), OP can predict which files will generated based on the input .idl files. The second answer (https://stackoverflow.com/a/39258996/2912478) shows three different ways to solve but I really couldn't get his solution working.
Test case
I prepared a simple test case. Suppose I have this static file where the main resides (main.cpp).
# main.cpp
void foo(void);
int main() {
foo();
return 0;
}
Currently, I am using this CMakeLists.txt. The custom command generates the source file under src.
# CMakeLists.txt
add_executable(a.out main.cpp)
add_custom_command(
OUTPUT mylib.cpp
COMMAND ${CMAKE_SOURCE_DIR}/genf.sh ${CMAKE_SOURCE_DIR}/src
DEPENDS ${CMAKE_SOURCE_DIR}/genf.sh
)
add_custom_target(GenFile DEPENDS mylib.cpp)
add_dependencies(a.out GenFile)
Here is the hypothetical code generator genf.sh. I use a random number to mimic the fact that we do NOT know which files the generator will generate.
#!/bin/bash
rm -rf $1/*
fname=$(echo $((1 + RANDOM % 100)))
echo "Generating src$fname.cpp"
echo "void foo(void) {}" > $1/src$fname.cpp
Attempt 1
I tried to use GLOB to find all files generated. So I put the following lines at the end of my CMakeLists.txt. This doesn't work because at the moment of running cmake .. there is no files under src. So this solution never links the generated source files.
file(GLOB GeneratedSourceFiles ${CMAKE_SOURCE_DIR}/src/*.cpp)
target_sources(a.out PUBLIC ${GeneratedSourceFiles})

Related

Dealing with lots and lots of escaped characters in add_custom_command

I have a file that contains a bunch of data. I want to turn it into a C++ string literal, because I need to compile this data into the binary - I cannot read it from disk.
One way of doing this is to just generate a C++ source file that declares a string literal with a known name. The CMake code to do this is straightforward, if somewhat awful:
function(make_literal_from_file dest_file source_file literal_name)
add_custom_command(
OUTPUT ${dest_file}
COMMAND printf \'char const* ${literal_name} = R\"\#\(\' > ${dest_file}
COMMAND cat ${source_file} >> ${dest_file}
COMMAND printf \'\)\#\"\;\' >> ${dest_file}
DEPENDS ${source_file})
endfunction()
This works and does what I want (printf is necessary to avoid a new line after the raw string introducer). However, the amount of escaping going on here makes it very difficult to see what's going on. Is there a way to write this function such that it's actually readable?
Note that I cannot use a file(READ ...)/configure_file(...) combo here because source_file could be something that is generated by CMake at build time and so may not be present at configuration time.
I would recommend writing a script to do this. You could write it in CMake, but I personally prefer a better language such as Python:
# Untested, just to show roughly how to do it
import sys
dest_file, source_file, literal_name = sys.argv[1:]
with open(dest_file) as dest, open(source_file) as source:
literal_contents = source.read()
dest.write(f'char const* {literal_name} = R"({literal_contents})";\n')
Corresponding CMake code:
# String interpolation came in Python 3.6, thus the requirement on 3.6.
# If using CMake < 3.12, use find_package(PythonInterp) instead.
find_package(Python3 3.6 COMPONENTS Interpreter)
# Make sure this resolves correctly. ${CMAKE_CURRENT_LIST_DIR} is helpful;
# it's the directory containing the current file (this cmake file)
set(make_literal_from_file_script "path/to/make_literal_from_file.py")
function(make_literal_from_file dest_file source_file literal_name)
add_custom_command(
OUTPUT "${dest_file}"
COMMAND
"${Python3_EXECUTABLE}" "${make_literal_from_file_script}"
"${dest_file}"
"${source_file}"
"${literal_name}"
DEPENDS "${source_file}")
endfunction()
If you don't want the dependency on Python, you could use C++ (only the CMake code shown):
add_executable(make_literal_from_file_exe
path/to/cpp/file.cpp
)
function(make_literal_from_file dest_file source_file literal_name)
add_custom_command(
OUTPUT "${dest_file}"
COMMAND
make_literal_from_file_exe
"${dest_file}"
"${source_file}"
"${literal_name}"
DEPENDS "${source_file}")
endfunction()

CMake's objects output folder variable

How "${PROJECT_BINARY_DIR}/CMakeFiles/project.dir/", the place object files resulted from compilation will be placed on, can be un-hardcoded?
Going straightly to the problem, we have some tests that check objects resulted from compilation on harfbuzz cmake and we use a hardcoded string there but that doesn't seem right and I hope some ${} or $<> exist for that.
I'm afraid you're out of luck here. CMake keeps this as an internal implementation detail, by design.
I'd say it's unlikely to change, but if you want to be absolutely future-proof, you could use a workaround of creating a static library out of the object files and then manually unpacking it back into object files (using the appropriate archiver/librarian) as part of the test. If the object files are also used by another target, and linking to that static library wouldn't work for that target, you can make an object library out of the files and then use that in both the original target and the for-test static library.
Here's an example of how you might achieve this workaround:
add_library(MyObjectLib OBJECT src/a.cpp src/b.cpp)
add_executable(Main src/main.cpp $<TARGET_OBJECTS:MyObjectLib>)
add_library(LibForTesting STATIC $<TARGET_OBJECTS:MyObjectLib>)
add_test(
NAME check-static-inits.sh
COMMAND ${PROJECT_SOURCE_DIR}/src/prepare-and-check-static-inits.sh $<TARGET_FILE:LibForTesting>
)
And here's what the script prepare-and-check-static-inits.sh would look like (pseudo-code):
ar -x $1 -o some_dir
./check-static-inits.sh some_dir
Turning my comment into an answer
There is at the moment no variable or generator expression to get the list of object files used for linking a archive or library target.
But you could append compiler/archiver/linker calls with any program/script and utilize CMake's expansion rules inside those calls.
Note: That will only work CMake's Command-Line Build Tool Generators. And the list(APPEND ...) calls only have to be there once in your CMake code after your project() call.
Examples
Generate a symbolic link to <OBJECT_DIR> with <TARGET_NAME>
project(MyLib)
list(
APPEND CMAKE_CXX_ARCHIVE_FINISH
"\"${CMAKE_COMMAND}\" -E create_symlink \"<OBJECT_DIR>\" \"<TARGET_NAME>\""
)
[...]
add_library(MyLib STATIC src/a.cpp src/b.cpp)
Call some program to do something with the <OBJECTS> list (e.g. echo or write to a file)
project(MyExe)
list(
APPEND CMAKE_CXX_LINK_EXECUTABLE
"\"${CMAKE_COMMAND}\" -E echo \"<OBJECTS>\""
)
[...]
add_executable(MyExe main.cpp)
Directly do something after each object file is generated. In your case where you want to call objdump for each object file it would e.g. be:
list(
APPEND CMAKE_CXX_COMPILE_OBJECT
"${CMAKE_OBJDUMP} -t \"<OBJECT>\" > $(notdir <OBJECT>.dump)"
)
Unfortunately there is no expansion rule for "output file name" hence the platform/make specific trick with $(notdir ...).
References
CMAKE_<LANG>_ARCHIVE_FINISH
CMAKE_<LANG>_LINK_EXECUTABLE
CMAKE_<LANG>_COMPILE_OBJECT
How to get path to object files with CMake for both multiconfiguration generator and makefile based ones?

How to parameterize cmake add_custom_command and add_custom_target?

I'm creating a library for micro-controller which as part of it's source will have couple of usage examples.
All CMakeList.txt files for the examples look very similar:
set(ESP_TARGET_FW1 "${CMAKE_CURRENT_BINARY_DIR}/${ESP_FW1}.bin")
set(ESP_TARGET_FW2 "${CMAKE_CURRENT_BINARY_DIR}/${ESP_FW2}.bin")
add_executable(esp_main main.c ${ESP_USER_CONFIG})
target_include_directories(esp_main PUBLIC include)
target_link_libraries(esp_main esp_sdo phy pp net80211)
# Create ESP8266 binary files.
add_custom_command(
OUTPUT
${ESP_TARGET_FW1} ${ESP_TARGET_FW2}
COMMAND
${ESPTOOL_PATH} elf2image $<TARGET_FILE:esp_main> -o ${CMAKE_CURRENT_BINARY_DIR}/
DEPENDS
esp_main
)
# Flash binary files to the device.
add_custom_target(esp_main_flash
COMMAND
${ESPTOOL_PATH} -p ${ESP_PORT} -b ${ESP_BAUD} write_flash ${ESP_FW1} ${ESP_TARGET_FW1} ${ESP_FW2} ${ESP_TARGET_FW2}
DEPENDS
${ESP_TARGET_FW1} ${ESP_TARGET_FW2}
)
Only the name of the example (esp_main) changes and where the generated binary files are created.
I'm looking to somehow parameterize add_custom_command and add_custom_target in case like this. Ideally include some file and have it define esp_main_flash target and custom command.
In CMake the most direct way for parameterize some sequence of actions (for later reuse) is creating a macro or a function. Both of them are allowed to perform any operation which can be written in a plain CMakeLists.txt.

In CMake how do I deal with generated source files which number and names are not known before?

Imagine a code generator which reads an input file (say a UML class diagram) and produces an arbitrary number of source files which I want to be handled in my project. (to draw a simple picture let's assume the code generator just produces .cpp files).
The problem is now the number of files generated depends on the input file and thus is not known when writing the CMakeLists.txt file or even in CMakes configure step. E.g.:
>>> code-gen uml.xml
generate class1.cpp..
generate class2.cpp..
generate class3.cpp..
What's the recommended way to handle generated files in such a case? You could use FILE(GLOB.. ) to collect the file names after running code-gen the first time but this is discouraged because CMake would not know any files on the first run and later it would not recognize when the number of files changes.
I could think of some approaches but I don't know if CMake covers them, e.g.:
(somehow) define a dependency from an input file (uml.xml in my example) to a variable (list with generated file names)
in case the code generator can be convinced to tell which files it generates the output of code-gen could be used to create a list of input file names. (would lead to similar problems but at least I would not have to use GLOB which might collect old files)
just define a custom target which runs the code generator and handles the output files without CMake (don't like this option)
Update: This question targets a similar problem but just asks how to glob generated files which does not address how to re-configure when the input file changes.
Together with Tsyvarev's answer and some more googling I came up with the following CMakeList.txt which does what I want:
project(generated)
cmake_minimum_required(VERSION 3.6)
set(IN_FILE "${CMAKE_SOURCE_DIR}/input.txt")
set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS "${IN_FILE}")
execute_process(
COMMAND python3 "${CMAKE_SOURCE_DIR}/code-gen" "${IN_FILE}"
WORKING_DIRECTORY ${PROJECT_BINARY_DIR}
INPUT_FILE "${IN_FILE}"
OUTPUT_VARIABLE GENERATED_FILES
OUTPUT_STRIP_TRAILING_WHITESPACE
)
add_executable(generated main.cpp ${GENERATED_FILES})
It turns an input file (input.txt) into output files using code-gen and compiles them.
execute_process is being executed in the configure step and the set_property() command makes sure CMake is being re-run when the input file changes.
Note: in this example the code-generator must print a CMake-friendly list on stdout which is nice if you can modify the code generator. FILE(GLOB..) would do the trick too but this would for sure lead to problems (e.g. old generated files being compiled, too, colleagues complaining about your code etc.)
PS: I don't like to answer my own questions - If you come up with a nicer or cleaner solution in the next couple of days I'll take yours!

Dynamically generated dependencies

I am trying to generate file that depends on a set of files that can
change throughout different make invocations.
To understand it better, let's show you the code:
cmake_minimum_required(VERSION 2.8)
project(demo-one C)
add_custom_command(
OUTPUT
"${CMAKE_BINARY_DIR}/generated.c"
COMMAND
generate -o "${CMAKE_BINARY_DIR}/generated.c"
DEPENDS
"$(shell generate-dependencies-list)"
COMMENT
"Generating generated.c"
)
add_executable(main main.c "${CMAKE_BINARY_DIR}/generated.c")
So, I want to generate the file generated.c with the generate
command and this files needs to be regenerated when the files
specified by generated-dependencies-list command changes. As you may
notice, generated-dependencies-list can generate different set of
files throughout make invocations, so is not feasible to get
the result of generated-dependencies-list at configure time to then
pass the result to add_custom_command.
Actually the above code somewhat works, but it looks like a hack that
will only work for Makefile backend, also the make rule doesn't look
as what I'm expecting, after all, it's a hack:
generated.c: ../$(shell\ generate-dependencies-list)
Basically, I want this rule or something to get the same result:
generated.c: $(shell generate-dependencies-list)
Has CMake any feature to achieve this?
when the files specified by generated-dependencies-list command changes
If the output of the command generated-dependencies-list depends only on this script and script's parameters then you can just add this script to DEPENDS sub-option:
add_custom_command(
OUTPUT
"${CMAKE_BINARY_DIR}/generated.c"
COMMAND
"${CMAKE_CURRENT_LIST_DIR}/generate-dependencies-list"
COMMAND
generate -o "${CMAKE_BINARY_DIR}/generated.c"
DEPENDS
"${CMAKE_CURRENT_LIST_DIR}/generate-dependencies-list"
COMMENT
"Generating generated.c"
)