How to use cmake file( GET_RUNTIME_DEPENDENCIES in an install statement? - cmake

How do you use file(GET_RUNTIME_DEPENDENCIES...) in a cmake install scripted statement? I can't find an example of this usage online, and the statement in the documentation and errors messages of using [[ ]] embedded custom scripting is not clear to me.
The impression I get is that at install time, this can be used to locate file dependencies of your cmake target and potentially bring them over with your install action, making it usable in standalone form.
For example, my application depends on QT and the expectation is that if this is configured correctly, the QT dlls needed for this application will be copied over to the bin. (I just want to be sure I don't have a misunderstanding of it's function in this context as well). It may not directly copy the files but I assume provides a list of files to copy that install will then process (all done at install time).
My naive attempt to just throw something at it to start is:
set(TARGET_NAME "myapp")
# installation settings
install(TARGETS ${TARGET_NAME}
[[
file(GET_RUNTIME_DEPENDENCIES
RESOLVED_DEPENDENCIES_VAR RES
UNRESOLVED_DEPENDENCIES_VAR UNRES
CONFLICTING_DEPENDENCIES_PREFIX CONFLICTING_DEPENDENCIES
EXECUTABLES ${TARGET_NAME}
)]]
RUNTIME DESTINATION "${INSTALL_X_BIN}" COMPONENT libraries
LIBRARY DESTINATION "${INSTALL_X_LIB}" COMPONENT libraries
)
However this of course gives me:
CMake Error at applications/CMakeLists.txt:117 (install):
install TARGETS given target " file(GET_RUNTIME_DEPENDENCIES
RESOLVED_DEPENDENCIES_VAR RES
UNRESOLVED_DEPENDENCIES_VAR UNRES
CONFLICTING_DEPENDENCIES_PREFIX CONFLICTING_DEPENDENCIES
EXECUTABLES ${TARGET_NAME}
)" which does not exist.
-- Configuring incomplete, errors occurred!
I feel silly about this like I'm missing something pretty basic.

Zeroth, an update
As of the next version of CMake (3.21), you may not want to use file(GET_RUNTIME_DEPENDENCIES) in some cases. (Which would be a good thing, as it works... poorly. It has no ability to differentiate between 32-bit and 64-bit shared libraries, for one thing, so it's irritatingly common to get wrong-arch libs returned on Linux. Then again, this development won't change that fact.)
If you're on Windows, the most common platform to require GET_RUNTIME_DEPENDENCIES logic, the next version of CMake is looking to take another stab at this (hopefully, fourth(?) time's the charm) with a new generator expression: $<TARGET_RUNTIME_DLLS:target>.
It's documented as the "List of DLLs that the target depends on at runtime. This is determined by the locations of all the SHARED and MODULE targets in the target's transitive dependencies. [...] This generator expression can be used to copy all of the DLLs that a target depends on into its output directory in a POST_BUILD custom command."
Considering I currently have custom logic in a CMakeLists.txt to do precisely that, because it's the only way to make the library's unit tests executable from the build directory, I'm hopeful this new expression makes that a bit easier.
Further update...
($<TARGET_RUNTIME_DLLS> won't fix the problems with file(GET_RUNTIME_DEPENDENCIES), but some commits just merged into CMake's upcoming 3.21 branch purport to, by teaching it how to distinguish between libraries for different architectures. Hooray!)
First, a caveat
You mentioned Qt. No matter what you do here, this method is unlikely to work for Qt all by itself, because there's no way using only the runtime dependencies of a program/library that you can discover any Qt plugins or other components that your installation may also require. Qt's dependencies are more complex than just libraries.
(My answer here demonstrates how to obtain Qt plugin information for bundling purposes, using the QCocoaIntegrationPlugin QPA on macOS as an example. All of Qt's plugins are represented by their own IMPORTED CMake targets, in recent releases, so it's typically possible to write install(CODE ...) scripting which picks up those targets using generator expressions in a similar manner to the following code.)
file(GET_RUNTIME_DEPENDENCIES)
As Tsyvarev noted in comments, GET_RUNTIME_DEPENDENCIES is intended to be used in the install stage, not the configure stage. As such, it needs to be placed in an install(CODE ...) or install(SCRIPT ...) statement, which will cause the code evaluation to be delayed until after the build is complete. (In fact, install(CODE ...) inserts the given code right into the current directory's cmake_install.cmake script. You can examine the results just by looking at that file, without even having to run the install.)
The delayed evaluation also comes with a few wrinkles. Primarily: The code doesn't understand targets. The targets no longer exist at the install stage. So, to include any target info, you have to use generator expressions to insert the correct values.
While the CMake documentation indicates that variable references and escapes aren't evaluated inside bracket arguments, generator expressions are. So, you can compose the CODE wrapped in [[ ]] to avoid escaping everything.
You still have to be careful about variable expansion / escaping. Most variables (including any you create) aren't available in the install context — only a few are, like CMAKE_INSTALL_PREFIX. You have to either expand or set any others.
There are, AFAICT, no generator expressions to access arbitrary variables. There are some for specific variables/values, but you can't say something like $<LIST:MY_LIST_VAR> or $<VALUE:MY_STRING_VAR> to combine variables and bracket arguments.
So, if you want to use variables from the configure context in the CODE, where they'll be evaluated at install time, the easiest thing to do is to "transfer" them into the install script by set()-ing a variable in the CODE.
file(INSTALL TYPE SHARED_LIBRARY)
To install shared library dependencies, you can use the same file(INSTALL) command that CMake itself uses in cmake_install.cmake if you build a shared library target. It uses the TYPE SHARED_LIBRARY option to add some extra processing. The FOLLOW_SYMLINK_CHAIN option is also especially handy. Together they'll make file(INSTALL) both resolve symbolic links in the source files, and automatically recreate them in the destination path.
Example code
So all in all, you'd want to do something like this:
set(MY_DEPENDENCY_PATHS /path/one /path/two)
# Transfer the value of ${MY_DEPENDENCY_PATHS} into the install script
install(CODE "set(MY_DEPENDENCY_PATHS \"${MY_DEPENDENCY_PATHS}\")")
install(CODE [[
file(GET_RUNTIME_DEPENDENCIES
LIBRARIES $<TARGET_FILE:mylibtarget>
EXECUTABLES $<TARGET_FILE:myprogtarget>
RESOLVED_DEPENDENCIES_VAR _r_deps
UNRESOLVED_DEPENDENCIES_VAR _u_deps
DIRECTORIES ${MY_DEPENDENCY_PATHS}
)
foreach(_file ${_r_deps})
file(INSTALL
DESTINATION "${CMAKE_INSTALL_PREFIX}/lib"
TYPE SHARED_LIBRARY
FOLLOW_SYMLINK_CHAIN
FILES "${_file}"
)
endforeach()
list(LENGTH _u_deps _u_length)
if("${_u_length}" GREATER 0)
message(WARNING "Unresolved dependencies detected!")
endif()
]])
* – (Note that using the DIRECTORIES argument on a non-Windows system will cause CMake to emit a warning, as files' dependencies are supposed to be resolvable using only the current environment.)
If the code gets too complex, there's always the option to create a separate script file copy_deps.cmake in the ${CMAKE_CURRENT_SOURCE_DIR} and use install(SCRIPT copy_deps.cmake). (A previous version of this answer suggested using file(GENERATE...) to build the script — that won't work, as the file isn't written until after processing the CMakeLists.txt.)

Building onto this answer (thanks!) I created a recursive version for collecting all library dependencies and their dependants (and so on..) for a given executable:
install(CODE [[
function(install_library_with_deps LIBRARY)
file(INSTALL
DESTINATION "${CMAKE_INSTALL_PREFIX}/lib"
TYPE SHARED_LIBRARY
FOLLOW_SYMLINK_CHAIN
FILES "${LIBRARY}"
)
file(GET_RUNTIME_DEPENDENCIES
LIBRARIES ${LIBRARY}
RESOLVED_DEPENDENCIES_VAR RESOLVED_DEPS
UNRESOLVED_DEPENDENCIES_VAR UNRESOLVED_DEPS
)
foreach(FILE ${RESOLVED_DEPS})
if(NOT IS_SYMLINK ${FILE})
install_library_with_deps(${FILE})
endif()
endforeach()
foreach(FILE ${UNRESOLVED_DEPS})
message(STATUS "Unresolved from ${LIBRARY}: ${FILE}")
endforeach()
endfunction()
file(GET_RUNTIME_DEPENDENCIES
EXECUTABLES $<TARGET_FILE:myexecutable>
RESOLVED_DEPENDENCIES_VAR RESOLVED_DEPS
UNRESOLVED_DEPENDENCIES_VAR UNRESOLVED_DEPS
)
foreach(FILE ${RESOLVED_DEPS})
install_library_with_deps(${FILE})
endforeach()
foreach(FILE ${UNRESOLVED_DEPS})
message(STATUS "Unresolved: ${FILE}")
endforeach()
]])
I also think its relevant to note that some variables (like CMAKE_INSTALL_PREFIX) can be used in the inner scope as they are, while others (like CMAKE_PREFIX_PATH) need to be re-set explicitly.
Going from here one might want to exclude specific system directories, this here likely collects too much.

Related

CMake GET_RUNTIME_DEPENDENCIES couldn't find dll library when it is linked through lib (import library)?

Build OS: Windows 10, Cmake 3.16.3.
I use target_link_libraries to link 3rd party .lib file to my .dll library.
But when I use GET_RUNTIME_DEPENDENCIES to install my dll, there is no dependency found.
It happens only on Windows, installing on Linux is ok.
Is there any clues how to solve this problem, or at least how to debug it?
What exact command uses CMake on Windows to determine dependencies?
I call GET_RUNTIME_DEPENDENCIES like this:
file(GET_RUNTIME_DEPENDENCIES
RESOLVED_DEPENDENCIES_VAR RES
UNRESOLVED_DEPENDENCIES_VAR UNRES
CONFLICTING_DEPENDENCIES_PREFIX CONFLICTING_DEPENDENCIES
EXECUTABLES ${EXECS}
LIBRARIES ${LIBS} ${MODULES} ${QTPLUGINS_LIBS}
DIRECTORIES ${RUNTIME_DEPENDENCIES_DIRECTORIES}
POST_EXCLUDE_REGEXES ${PLATFORM_POST_EXCLUDE_REGEXES}
)
Where LIBS contains my dll but no RES no UNRES contains path to 3rd paty dll.
So, there's a serious nastiness to all this runtime-dependency-finding magic in the newer CMakes, and it's not really their fault at all. The problem is that you, I, and roughly 90% of the rest of the CMake user world have been doing find modules wrong #THISWHOLETIME, and now our chickens have come home to roost because, as you've likely discovered, GET_RUNTIME_DEPENDENCIES / RUNTIME_DEPENDENCY_SET, $<TARGET_RUNTIME_DLLS> will all completely sh*t the bed if you try to use them with targets that have (what I now know to be) broken IMPORTED dependencies created by Find modules that don't properly set them up. So, last month I posted this screed (my earlier link) over at the CMake Discourse forum:
Windows libraries, Find modules, and TARGET_RUNTIME_DLLS
The Windows DLL Question™ has come up numerous times before in one form or another, but it’s cast in a new light by $<TARGET_RUNTIME_DLLS>, so here’s a fresh take on it.
If you’re like me (and roughly 90% of all CMake users/developers out there are like me, from what I’ve been able to observe in public projects’ source trees), your approach to writing Find modules on Windows has probably been something like this:
Use the same code on all three desktop platforms
Let CMake discover .lib / .dll.a import libraries instead of actual DLLs, using find_library().
End up creating your targets as UNKNOWN IMPORTED, because if you try to create SHARED IMPORTED library targets with only an import library it won’t work, but UNKNOWN IMPORTED works just fine so meh.
Set the import library as the target’s IMPORTED_LOCATION since that seems to work fine.
Call it a day, because hey — everything compiles.
That’s served us all for years (decades, really) so we’ve mostly just accepted it as the way CMake works on Windows.
But now along comes $<TARGET_RUNTIME_DLLS>. If you’ve tried to actually use it on Windows, you’ve probably discovered is that while all of your CONFIG-mode package dependencies’ DLLs are captured just fine, the generator expression will cheerfully ignore any targets created from a Find module that’s written like I describe above. …Which is probably most of them. (In my own library build, it was all of them, even the ones I didn’t write.)
For $<TARGET_RUNTIME_DLLS> to work, the IMPORTED target has to be correctly defined as a SHARED library target, and it needs to have its IMPORTED_ properties set correctly: import lib path in IMPORTED_IMPLIB, DLL path in IMPORTED_LOCATION.
So, now I have this new module that uses DLLTOOL.EXE and its handy -I flag to get the name of an import library’s DLL, then looks it up using find_program(). (Simply because find_library() won’t match DLLs, and I wanted to look on the PATH. I could’ve used find_file() but I’m pretty sure I’d have to explicitly give it more paths to search.)
The macro takes one argument, the name of your already-configured variable <prefix>_IMPLIB. (Or <prefix>_IMPLIBS, it’s pluralization agnostic and will follow whichever form your input uses when naming its output variable.)
The variable whose name you pass to it should already contain a valid path for an import library. Typically that’s set by find_library(), even though we’ve all been treating them like runtime libraries (DLLs) when they are not.
Armed with find_library(<prefix>_IMPLIB ...) output, implib_to_dll(<prefix>_IMPLIB) will attempt to discover and automatically populate the corresponding variable <prefix>_LIBRARY with the path to the import lib’s associated runtime DLL.
With all of the correct variables set to the correct values, it’s now possible to properly configure SHARED IMPORTED library targets on Windows. $<TARGET_RUNTIME_DLLS> can then be used to discover and operate on the set of DLLs defined by those target(s).
Kind of a pain in the Find, and really does sort of feel like something CMake could be doing at-least-semi-automatically. But, at least for now it works.
Now I just have to rewrite all of my find modules to use it. Sigh.
ImplibUtils.cmake
#[=======================================================================[.rst:
IMPLIB_UTILS
------------
Tools for CMake on WIN32 to associate IMPORTED_IMPLIB paths (as discovered
by the :command:`find_library` command) with their IMPORTED_LOCATION DLLs.
Writing Find modules that create ``SHARED IMPORTED`` targets with the
correct ``IMPORTED_IMPLIB`` and ``IMPORTED_LOCATION`` properties is a
requirement for ``$<TARGET_RUNTIME_DLLS>`` to work correctly. (Probably
``IMPORTED_RUNTIME_DEPENDENCIES`` as well.)
Macros Provided
^^^^^^^^^^^^^^^
Currently the only tool here is ``implib_to_dll``. It takes a single
argument, the __name__ (_not_ value!) of a prefixed ``<prefix>_IMPLIB``
variable (containing the path to a ``.lib`` or ``.dll.a`` import library).
``implib_to_dll`` will attempt to locate the corresponding ``.dll`` file
for that import library, and set the variable ``<prefix>_LIBRARY``
to its location.
``implib_to_dll`` relies on the ``dlltool.exe`` utility. The path can
be set by defining ``DLLTOOL_EXECUTABLE`` in the cache prior to
including this module, if it is not set implib_utils will attempt to locate
``dlltool.exe`` using ``find_program()``.
Revision history
^^^^^^^^^^^^^^^^
2021-11-18 - Updated docs to remove CACHE mentions, fixed formatting
2021-10-14 - Initial version
Author: FeRD (Frank Dana) <ferdnyc#gmail.com>
License: CC0-1.0 (Creative Commons Universal Public Domain Dedication)
#]=======================================================================]
include_guard(DIRECTORY)
if (NOT WIN32)
# Nothing to do here!
return()
endif()
if (NOT DEFINED DLLTOOL_EXECUTABLE)
find_program(DLLTOOL_EXECUTABLE
NAMES dlltool dlltool.exe
DOC "The path to the DLLTOOL utility"
)
if (DLLTOOL_EXECUTABLE STREQUAL "DLLTOOL_EXECUTABLE-NOTFOUND")
message(WARNING "DLLTOOL not available, cannot continue")
return()
endif()
message(DEBUG "Found dlltool at ${DLLTOOL_EXECUTABLE}")
endif()
#
### Macro: implib_to_dll
#
# (Win32 only)
# Uses dlltool.exe to find the name of the dll associated with the
# supplied import library.
macro(implib_to_dll _implib_var)
set(_implib ${${_implib_var}})
set(_library_var "${_implib_var}")
# Automatically update the name, assuming it's in the correct format
string(REGEX REPLACE
[[_IMPLIBS$]] [[_LIBRARIES]]
_library_var "${_library_var}")
string(REGEX REPLACE
[[_IMPLIB$]] [[_LIBRARY]]
_library_var "${_library_var}")
# We can't use the input variable name without blowing away the
# previously-discovered contents, so that's a non-starter
if ("${_implib_var}" STREQUAL "${_library_var}")
message(ERROR "Name collision! You probably didn't pass "
"implib_to_dll() a correctly-formatted variable name. "
"Only <prefix>_IMPLIB or <prefix>_IMPLIBS is supported.")
return()
endif()
if(EXISTS "${_implib}")
message(DEBUG "Looking up dll name for import library ${_implib}")
execute_process(COMMAND
"${DLLTOOL_EXECUTABLE}" -I "${_implib}"
OUTPUT_VARIABLE _dll_name
OUTPUT_STRIP_TRAILING_WHITESPACE
)
message(DEBUG "DLLTOOL returned ${_dll_name}, finding...")
# Check the directory where the import lib is found
get_filename_component(_implib_dir ".." REALPATH
BASE_DIR "${_implib}")
message(DEBUG "Checking import lib directory ${_implib_dir}")
# Add a check in ../../bin/, relative to the import library
get_filename_component(_bindir "../../bin" REALPATH
BASE_DIR "${_implib}")
message(DEBUG "Also checking ${_bindir}")
find_program(${_library_var}
NAMES ${_dll_name}
HINTS
${_bindir}
${_implib_dir}
PATHS
ENV PATH
)
set(${_library_var} "${${_library_var}}" PARENT_SCOPE)
message(DEBUG "Set ${_library_var} to ${${_library_var}}")
endif()
endmacro()
GET_RUNTIME_DEPENDENCIES isn't aware of your configure-time variables, so will you need to specify them manually. This answer states you can pass-on the variables to the install step, but I haven't been able to make it work so far. Fortunately, it does support generator expressions.
Another problem in your snippet is it must be called at install time. For example in an install(CODE ...) block.
So with all this in mind, this should get you started.
install(CODE [[
file(GET_RUNTIME_DEPENDENCIES
RESOLVED_DEPENDENCIES_VAR RES
UNRESOLVED_DEPENDENCIES_VAR UNRES
CONFLICTING_DEPENDENCIES_PREFIX CONFLICTING_DEPENDENCIES
EXECUTABLES $<TARGET_FILE:your_executable_target_name>
LIBRARIES $<TARGET_FILE:a_lib_target_name>
)
message("\n\nFound dependencies :")
foreach(DEP ${RES})
message("${DEP}")
endforeach()
message("\n\nNot found dependencies :")
foreach(DEP ${UNRES})
message("${DEP}")
endforeach()
]])
Build your install target to see the results.
cmake ..
cmake --build . --target install

Creating a library in CMake depending on source files not available when generating build files

I have a CMake configuration file building two libraries:
a third-party library (here called ThirdPartyLib) containing a real-time OS / board support package from a supplier. It is built outside CMake using the autotools toolchain.
an extended version of the former library (here called ExtendedThirdPartyLib)
Unfortunately, some source code that I need (various tools) are not built in the ordinary build script for (1). Since I don't want to mess with the suppliers build script I want to add another library (2), building the missing files and thus extending the library from the supplier.
I want to able to do something like this in CMakeFiles.txt:
cmake_minimum_required(VERSION 3.2)
project(bsp)
include(ExternalProject)
ExternalProject_Add(
ThirdPartyLib
URL <http://some.url/bsp.tar.bz2
BUILD_COMMAND make -C ../external/ThirdPartyLib/src
)
set_target_properties(ThirdPartyLib PROPERTIES EXCLUDE_FROM_ALL TRUE)
add_library(ExtendedThirdPartyLib
${CMAKE_CURRENT_BINARY_DIR}/some/path/missing_file1.c
${CMAKE_CURRENT_BINARY_DIR}/some/path/missing_file2.c
)
add_dependencies(ExtendedThirdPartyLib ThirdPartyLib)
target_include_directories(ExtendedThirdPartyLib PUBLIC
${CMAKE_CURRENT_BINARY_DIR}/some/path/include
)
target_link_libraries(ExtendedThirdPartyLib ThirdPartyLib)
The problem here is that the path to missing_file1.c and missing_file2.c are not valid when CMake is generating the build files (they are extracted from the tarball from the supplier). CMake exits with an error output saying: "Cannot find source file".
Is there a neat way to make this work? I.e. is it possible to convince CMake that certain non-existant input files will exist when building of the library begins? Or is there any other recommended way to solve this issue?
(I have temporary made local copies of the files I need to build from the suppliers tarball, but that is of course not a good solution. If those files are changed in future versions of the suppliers package and I forget to overwrite my local copies it could be a horrible mess...
Another "solution" would be to create a small makefile outside CMake and use another ExternalProject_Add in the CMakeFiles.txt somehow. But that's not a good solution either, e.g. if compile and linker flags are modified I need to remember to change the makefile too.)
Personally, I dislike the ExternalProject_Add command, because it does way too many things for my taste, but I've digressed.
What if you do something like this, where bar is simulating your ExtendedThirdPartyLib target, since it depends on generated files
cmake_minimum_required(VERSION 3.11)
project(lol C)
set(SOURCES lol.c) # only this file exists
add_library(lol ${SOURCES})
set(FOO_FILES "foo1.c" "foo2.c")
add_custom_command(OUTPUT ${FOO_FILES}
COMMAND ${CMAKE_COMMAND} -E touch ${FOO_FILES}
WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
COMMENT "Creating ${FOO_FILES}"
VERBATIM)
add_custom_target(foo DEPENDS ${FOO_FILES})
add_library(bar ${FOO_FILES})
add_dependencies(bar foo)
target_link_libraries(lol bar)
The whole approach hinges on the fact that the method, where produced/generated files are procured, is explicitly defined via the custom command and associated custom target.
You should modify the custom command to extract the required files (e.g. could even call some external script) from the tarball (which might require downloading with curl or something similar).

Directing cmake to link against shared object with debug postfix (_d)

I've got a cmake project that pretty much looks like this:
cmake_minimum_required(VERSION 3.0)
SET(CMAKE_DEBUG_POSTFIX "_d")
include_directories(../TransfunctionerProject)
include_directories(../TransmogrifierProject)
set(Libraries
ContinuumTransfunctioner
Transmogrifier
)
set(SourceFiles
Wrapper.cpp
Logger.cpp
)
add_library(Frobnigator SHARED ${SourceFiles})
add_library(FrobnigatorStatic STATIC ${SourceFiles})
set_target_properties(FrobnigatorStatic PROPERTIES OUTPUT_NAME Frobnigator)
target_link_libraries(Frobnigator ${Libraries})
Where ContinuumTransfunctioner and Transmogrifier projects include the debug postfix directive SET(CMAKE_DEBUG_POSTFIX "_d") so that libContinuumTransfunctioner_d.so and libTransmogrifier_d.so both exist.
The problem is that the current project appears to be linking against the static library without the _d suffix and complains:
/usr/bin/ld: cannot find -lContinuumTransfunctioner
The Libraries that you pass into the target_link_libraries call are interpreted as filenames, not as target names.
This is the unfortunate fallback for that call in CMake: If you pass a random string to it, that cannot be interpreted in a meaningful way, CMake will always assume it to be plain library name. Sometimes this is just what you want, but the name has to be an exact match for an existing library. The whole debug postfix magic will be lost here.
What you might have wanted to do was to pass a library target name instead. This will trigger a much smarter handling of the dependency and would solve your problem. However, that only works if the library is a known target in the context of the target_link_libraries call. You can easily check this as follows:
if(TARGET ContinuumTransfunctioner)
message("Library target name")
else()
message("Plain library name")
endif()
target_link_libraries(Frobnigator ContinuumTransfunctioner)
So how do you get to the target name case? This depends on how your build is structured. If the library is being built as part of your CMake run, simply make sure that the corresponding add_library call is performed from a subdirectory that is pulled in via add_subdirectory from the file that performs the target_link_libraries call.
If the library in question is an external dependency, you need to build an imported target that carries all the relevant information where to find the library files (including any potential debug postfixes). This can be a bit cumbersome to do manually, so if you can, you might want to use CMake's packaging mechanism to generate this automatically as part of the library's build process.
Here's the solution, courtesy of the good people on the cmake mailing list:
# Note:
# $<$<CONFIG:Debug>:_d> is called a generator expression.
# It outputs _d if the build is debug.
#
set(Libraries
ContinuumTransfunctioner$<$<CONFIG:Debug>:_d>
Transmogrifier$<$<CONFIG:Debug>:_d>
)

Building a tool immediately so it can be used later in same CMake run

I have an interesting chicken-and-egg problem and a potential solution to it (see my posted answer), but that solution uses CMake in an unusual way. Better alternatives or comments would be welcome.
THE PROBLEM:
The simple version of the problem can be described as a single CMake project with the following characteristics:
One of the build targets is a command-line executable which I'll call mycomp, the source of which is in a mycompdir and making any modifications to the contents of that directory is not possible.
The project contains text files (I'll call them foo.my and bar.my) which need mycomp run on them to produce a set of C++ sources and headers and some CMakeLists.txt files defining libraries built from those sources.
Other build targets in the same project need to link against the libraries defined by those generated CMakeLists.txt files. These other targets also have sources which #include some of the generated headers.
You can think of mycomp as being something like a compiler and the text files in step 2 as some sort of source files. This presents a problem, because CMake needs the CMakeLists.txt files at configure time, but mycomp is not available until build time and therefore isn't available on the first run to create the CMakeLists.txt files early enough.
NON-ANSWER:
Normally, an ExternalProject-based superbuild arrangement would be a potential solution to this, but the above is a considerable simplification of the actual project I am dealing with and I don't have the freedom to split the build into different parts or perform other large scale restructuring work.
The crux of the problem is needing mycomp to be available when CMake is run so that the generated CMakeLists.txt files can be created and then pulled in with add_subdirectory(). A possible way to achieve this is to use execute_process() to run a nested cmake-and-build from the main build. That nested cmake-and-build would use the exact same source and binary directories as the top level CMake run (unless cross compiling). The general structure of the main top level CMakeLists.txt would be something like this:
# Usual CMakeLists.txt setup stuff goes here...
if(EARLY_BUILD)
# This is the nested build and we will only be asked to
# build the mycomp target (see (c) below)
add_subdirectory(mycompdir)
# End immediately, we don't want anything else in the nested build
return()
endif()
# This is the main build, setup and execute the nested build
# to ensure the mycomp executable exists before continuing
# (a) When cross compiling, we cannot re-use the same binary dir
# because the host and target are different architectures
if(CMAKE_CROSSCOMPILING)
set(workdir "${CMAKE_BINARY_DIR}/host")
execute_process(COMMAND ${CMAKE_COMMAND} -E make_directory "${workdir}")
else()
set(workdir "${CMAKE_BINARY_DIR}")
endif()
# (b) Nested CMake run. May need more -D... options than shown here.
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}"
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
-DCMAKE_MAKE_PROGRAM=${CMAKE_MAKE_PROGRAM}
-DEARLY_BUILD=ON
${CMAKE_SOURCE_DIR}
WORKING_DIRECTORY "${workdir}")
# (c) Build just mycomp in the nested build. Don't specify a --config
# because we cannot know what config the developer will be using
# at this point. For non-multi-config generators, we've already
# specified CMAKE_BUILD_TYPE above in (b).
execute_process(COMMAND ${CMAKE_COMMAND} --build . --target mycomp
WORKING_DIRECTORY "${workdir}")
# (d) We want everything from mycompdir in our main build,
# not just the mycomp target
add_subdirectory(mycompdir)
# (e) Run mycomp on the sources to generate a CMakeLists.txt in the
# ${CMAKE_BINARY_DIR}/foobar directory. Note that because we want
# to support cross compiling, working out the location of the
# executable is a bit more tricky. We cannot know whether the user
# wants debug or release build types for multi-config generators
# so we have to choose one. We cannot query the target properties
# because they are only known at generate time, which is after here.
# Best we can do is hardcode some basic logic.
if(MSVC)
set(mycompsuffix "Debug/mycomp.exe")
elseif(CMAKE_GENERATOR STREQUAL "Xcode")
set(mycompsuffix "Debug/mycomp")
else()
set(mycompsuffix "mycomp")
endif()
set(mycomp_EXECUTABLE "${workdir}/mycompdir/${mycompsuffix}")
execute_process(COMMAND "${mycomp_EXECUTABLE}" -outdir foobar ${CMAKE_SOURCE_DIR}/foo.my ${CMAKE_SOURCE_DIR}/bar.my)
# (f) Now pull that generated CMakeLists.txt into the main build.
# It will create a CMake library target called foobar.
add_subdirectory(${CMAKE_BINARY_DIR}/foobar ${CMAKE_BINARY_DIR}/foobar-build)
# (g) Another target which links to the foobar library
# and includes headers from there
add_executable(gumby gumby.cpp)
target_link_libraries(gumby PUBLIC foobar)
target_include_directories(gumby PUBLIC foobar)
If we don't re-use the same binary directory at (b) and (c) as we use for the main build, we end up building mycomp twice, which we obviously want to avoid. For cross compiling, we cannot avoid that, so in such cases we build the mycomp tool off to the side in a separate binary directory.
I've experimented with the above approach and indeed it appears to work in the real world project that prompted the original question, at least for the Unix Makefiles, Ninja, Xcode (OS X and iOS) and Visual Studio generators. Part of the attractiveness of this approach is that it only requires a modest amount of code to be added just to the top level CMakeLists.txt file. Nevertheless, there are some observations that should be made:
If the compiler or linker commands for mycomp and its sources are different in any way between the nested build and the main build, the mycomp target ends up getting rebuilt a second time at (d). If there are no differences, mycomp only gets built once when not cross compiling, which is exactly what we want.
I see no easy way to pass exactly the same arguments to the nested invocation of CMake at (b) as was passed to the top level CMake run (basically the problem described here). Reading CMakeCache.txt isn't an option since it won't exist on the first invocation and it would not give you any new or changed arguments from the current run anyway. The best I can do is to set those CMake variables I think are potentially going to be used and which may influence the compiler and linker commands of mycomp. This can be worked around by adding more and more variables as I encounter ones I discover I need, but that's not ideal.
When re-using the same binary directory, we are relying on CMake not starting to write any of its files to the binary directory until the generate stage (well, at least until after the build at (c) completes). For the generators tested, it appears we are okay, but I don't know if all generators on all platforms follow this behaviour too (and I can't test every single combination to find out!). This is the part that gives me the greatest concern. If anyone can confirm with reasoning and/or evidence that this is safe for all generators and platforms, that would be valuable (and worth an upvote if you want to address this as a separate answer).
UPDATE: After using the above strategy on a number of real world projects with staff of varying levels of familiarity with CMake, some observations can be made.
Having the nested build re-use the same build directory as the main build can occasionally lead to problems. Specifically, if a user kills the CMake run after the nested build completes but before the main build does, the CMakeCache.txt file is left with EARLY_BUILD set to ON. This then makes all subsequent CMake runs act like a nested build, so the main build is essentially lost until the CMakeCache.txt file is manually removed. It is possible that an error somewhere in one of the project's CMakeLists.txt file may also lead to a similar situation (unconfirmed). Performing the nested build off to the side in its own separate build directory has worked very well though with no such problems.
The nested build should probably be Release rather than Debug. If not re-using the same build directory as the main build (now what I'd recommend), we no longer care about trying to avoid compiling the same file twice, so may as well make mycomp as fast as possible.
Use ccache so that any costs due to rebuilding some files twice with different settings are minimised. Actually, we found using ccache typically makes the nested build very quick since it rarely changed compared to the main build.
The nested build probably needs to have CMAKE_BUILD_WITH_INSTALL_RPATH set to FALSE on some platforms so that any libraries mycomp needs can be found without having to set environment variables, etc.

CMake find_package dependency on subproject

I have the following directory layout:
main_folder
+ static_lib1
+ executable
Both 'static_lib1' and 'executable' have a full CMakeLists so that they can be
built independently.
The 'executable' depends on 'static_lib1'. It uses find_package() to locate 'static_lib1'.
The main_folder contains a CMakeLists that includes both 'static_lib1' and 'executable' via add_subdirectory for conveniently building the whole project in one go.
Everything works fine if I manually build 'static_lib1' and then 'executable'. But when running the CMakeLists from the main folder, I get an error because find_package is unable to find the library files from 'static_lib1' which have not yet been built.
How can I resolve this while keeping the CMakeLists files separate (i.e. without including the static_lib's CMakeLists from the executable's CMakeLists)?
In executable's CMakeLists.txt you can check if you are building stand-alone or as part of project:
if( CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR )
# stand-alone build
find_package(static_lib1)
else()
include_directories(../static_lib1)
link_directories(../static_lib1)
...
target_link_libraries(executable static_lib1)
endif()
Switch from a file-based approach to a target-based approach for handling the dependency from executable to static_lib1.
The original problem occurred because executable called find_package for locating static_lib1, which then attempted to fill a variable like STATIC_LIB1_LIBRARY with the paths to the library files by calling find_library. executable then consumes the content of that variable in a target_link_libraries(executable ${STATIC_LIB1_LIBRARY}) call. The problem here is, since those library files only get generated as part of the build, that call to find_library will not be able to find anything.
Building executable needs to support two scenarios here:
Building standalone, where a pre-compiled version of static_lib1 is located somewhere on the disc.
Building from main_folder, where both executable and static_lib1 are part of the same build.
The approach from the question supports scenario 1, but not scenario 2.
Instead of using using a variable to communicate a dependency between the two builds, use a target. The CMakeLists.txt for static_lib1 likely creates a library target like add_library(static_lib1 [...]). In executable we now simply do target_link_libraries(executable PUBLIC static_lib1). This is sufficient to support scenario 2.
To also allow for scenario 1 at the same time, we look at the call to find_package(static_lib1) in the CMakeLists.txt for executable. Instead of providing a variable like before, this call now needs to provide a target static_lib1 for consumption.
So we adapt the find script for static_lib1 to the following behavior:
If a target static_lib1 already exists, there's nothing to be done and the find script can just return (this is scenario 2).
Otherwise, we call find_library to locate the library file on disc (as before in the original approach) and then create a new imported target: add_library(static_lib1 STATIC IMPORTED). We then configure all relevant properties of the static library to that target. For instance, to add the location of the library file, we could do
set_target_properties(static_lib1 PROPERTIES
IMPORTED_LINK_INTERFACE_LANGUAGES "CXX"
IMPORTED_LOCATION ${STATIC_LIB1_LIBRARY}
)
To support multi-config generators like MSVC, instead of setting IMPORTED_LOCATION and IMPORTED_LINK_INTERFACE_LANGUAGES, you will want to set the configuration specific properties like IMPORTED_LOCATION_DEBUG and IMPORTED_LOCATION_RELEASE instead. Since this can get quite tedious to do manually, you can have CMake generate this information (and a bunch of other convenient stuff) for you in a package script. The find mechanism for package scripts works slightly different under the hood, but the code in the CMakeLists.txt for executable will look just the same, a simple call to find_package(static_lib1). The main difference is that this call will then not dispatch to a hand-written find script, but to a package script that was automatically generated by CMake as part of the build process of static_lib1.
I guess I will leave this answer for posterity since only recently I have searched for a solution to this problem and found out that...
Since CMake 3.24 it is possible!
It is possible to override subsequent calls to find_package() with FetchContent_Declare() flag OVERRIDE_FIND_PACKAGE.
Your
add_subdirectory("path/to/static_lib1")
call has to be replaced in main_folder/CMakeLists.txt with:
include(FetchContent)
FetchContent_Declare(
static_lib1
SOURCE_DIR "path/to/static_lib1"
OVERRIDE_FIND_PACKAGE
)
Any calls to find_package(static_lib1) will call FetchContent_MakeAvailable() for you, virtually making it identical to add_subdirectory() call.
You can read more about OVERRIDE_FIND_PACKAGE in CMake documentation.