I'm trying to correctly manage a dependency of a target on a externally built library, and somehow I'm not succeeding. I have read tutorials, posts and examples aplenty and yet, since I'm new to CMake, I guess I'm missing some obvious thing.
Setup is as follows. An external library built in another (CMake unsupported) language produces a libadatest.a. I've used ExternalProject_Add for this. Then, there is another regular C target that uses this lib. Everything works fine, but if I change the original lib, even if I recompile it, the C target is not recompiled. Here is a complete sample. I'm using CMake 2.8.12:
cmake_minimum_required(VERSION 2.8)
include(ExternalProject)
ExternalProject_Add(
AdaTestExternal # Not important
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}
BUILD_COMMAND gprbuild -P${CMAKE_CURRENT_SOURCE_DIR}/adalibtest -XOBJ_DIR=${CMAKE_CURRENT_BINARY_DIR}${CMAKE_FILES_DIRECTORY} -XLIB_DIR=${CMAKE_CURRENT_BINARY_DIR}
ALWAYS 1 # Force build, gprbuild will take care of dependencies
# BUILD_ALWAYS 1 # For 3.0 higher versions?
INSTALL_COMMAND ""
)
add_custom_target(AdaTest DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/libadatest.a)
link_directories(${CMAKE_CURRENT_BINARY_DIR}) # Needed or won't find it
add_executable(main_ada main.c)
add_dependencies(main_ada AdaTest) # We must depend on the final output lib
target_link_libraries(main_ada adatest)
What I've attempted is to create an intermediate custom target which depends on the actual library, and in turn make the main C target depend on this target.
When I remove the externally built library (libadatest.a), that's properly externally recompiled but the main executable is not re-linked. Plainly seen in that the timestamp of the library is fresher than the executable which uses it.
I've also tried this instead of the custom target, with same negative result:
add_library(AdaTest
UNKNOWN IMPORTED
IMPORTED_LOCATION ${CMAKE_CURRENT_BINARY_DIR}/libadatest.a)
Found the proper solution (which was, as expected, simple) in this old post: http://www.cmake.org/pipermail/cmake/2010-November/041072.html
The gist is to use the actual file in target_link_libraries, so its timestamp is checked. So no need for intermediate or custom dependencies:
set(AdaTestLib ${CMAKE_CURRENT_BINARY_DIR}/libadatest.a)
add_executable(main_ada main.c)
add_dependencies(main_ada AdaTestExternal)
target_link_libraries(main_ada ${AdaTestLib})
Related
How do you use file(GET_RUNTIME_DEPENDENCIES...) in a cmake install scripted statement? I can't find an example of this usage online, and the statement in the documentation and errors messages of using [[ ]] embedded custom scripting is not clear to me.
The impression I get is that at install time, this can be used to locate file dependencies of your cmake target and potentially bring them over with your install action, making it usable in standalone form.
For example, my application depends on QT and the expectation is that if this is configured correctly, the QT dlls needed for this application will be copied over to the bin. (I just want to be sure I don't have a misunderstanding of it's function in this context as well). It may not directly copy the files but I assume provides a list of files to copy that install will then process (all done at install time).
My naive attempt to just throw something at it to start is:
set(TARGET_NAME "myapp")
# installation settings
install(TARGETS ${TARGET_NAME}
[[
file(GET_RUNTIME_DEPENDENCIES
RESOLVED_DEPENDENCIES_VAR RES
UNRESOLVED_DEPENDENCIES_VAR UNRES
CONFLICTING_DEPENDENCIES_PREFIX CONFLICTING_DEPENDENCIES
EXECUTABLES ${TARGET_NAME}
)]]
RUNTIME DESTINATION "${INSTALL_X_BIN}" COMPONENT libraries
LIBRARY DESTINATION "${INSTALL_X_LIB}" COMPONENT libraries
)
However this of course gives me:
CMake Error at applications/CMakeLists.txt:117 (install):
install TARGETS given target " file(GET_RUNTIME_DEPENDENCIES
RESOLVED_DEPENDENCIES_VAR RES
UNRESOLVED_DEPENDENCIES_VAR UNRES
CONFLICTING_DEPENDENCIES_PREFIX CONFLICTING_DEPENDENCIES
EXECUTABLES ${TARGET_NAME}
)" which does not exist.
-- Configuring incomplete, errors occurred!
I feel silly about this like I'm missing something pretty basic.
Zeroth, an update
As of the next version of CMake (3.21), you may not want to use file(GET_RUNTIME_DEPENDENCIES) in some cases. (Which would be a good thing, as it works... poorly. It has no ability to differentiate between 32-bit and 64-bit shared libraries, for one thing, so it's irritatingly common to get wrong-arch libs returned on Linux. Then again, this development won't change that fact.)
If you're on Windows, the most common platform to require GET_RUNTIME_DEPENDENCIES logic, the next version of CMake is looking to take another stab at this (hopefully, fourth(?) time's the charm) with a new generator expression: $<TARGET_RUNTIME_DLLS:target>.
It's documented as the "List of DLLs that the target depends on at runtime. This is determined by the locations of all the SHARED and MODULE targets in the target's transitive dependencies. [...] This generator expression can be used to copy all of the DLLs that a target depends on into its output directory in a POST_BUILD custom command."
Considering I currently have custom logic in a CMakeLists.txt to do precisely that, because it's the only way to make the library's unit tests executable from the build directory, I'm hopeful this new expression makes that a bit easier.
Further update...
($<TARGET_RUNTIME_DLLS> won't fix the problems with file(GET_RUNTIME_DEPENDENCIES), but some commits just merged into CMake's upcoming 3.21 branch purport to, by teaching it how to distinguish between libraries for different architectures. Hooray!)
First, a caveat
You mentioned Qt. No matter what you do here, this method is unlikely to work for Qt all by itself, because there's no way using only the runtime dependencies of a program/library that you can discover any Qt plugins or other components that your installation may also require. Qt's dependencies are more complex than just libraries.
(My answer here demonstrates how to obtain Qt plugin information for bundling purposes, using the QCocoaIntegrationPlugin QPA on macOS as an example. All of Qt's plugins are represented by their own IMPORTED CMake targets, in recent releases, so it's typically possible to write install(CODE ...) scripting which picks up those targets using generator expressions in a similar manner to the following code.)
file(GET_RUNTIME_DEPENDENCIES)
As Tsyvarev noted in comments, GET_RUNTIME_DEPENDENCIES is intended to be used in the install stage, not the configure stage. As such, it needs to be placed in an install(CODE ...) or install(SCRIPT ...) statement, which will cause the code evaluation to be delayed until after the build is complete. (In fact, install(CODE ...) inserts the given code right into the current directory's cmake_install.cmake script. You can examine the results just by looking at that file, without even having to run the install.)
The delayed evaluation also comes with a few wrinkles. Primarily: The code doesn't understand targets. The targets no longer exist at the install stage. So, to include any target info, you have to use generator expressions to insert the correct values.
While the CMake documentation indicates that variable references and escapes aren't evaluated inside bracket arguments, generator expressions are. So, you can compose the CODE wrapped in [[ ]] to avoid escaping everything.
You still have to be careful about variable expansion / escaping. Most variables (including any you create) aren't available in the install context — only a few are, like CMAKE_INSTALL_PREFIX. You have to either expand or set any others.
There are, AFAICT, no generator expressions to access arbitrary variables. There are some for specific variables/values, but you can't say something like $<LIST:MY_LIST_VAR> or $<VALUE:MY_STRING_VAR> to combine variables and bracket arguments.
So, if you want to use variables from the configure context in the CODE, where they'll be evaluated at install time, the easiest thing to do is to "transfer" them into the install script by set()-ing a variable in the CODE.
file(INSTALL TYPE SHARED_LIBRARY)
To install shared library dependencies, you can use the same file(INSTALL) command that CMake itself uses in cmake_install.cmake if you build a shared library target. It uses the TYPE SHARED_LIBRARY option to add some extra processing. The FOLLOW_SYMLINK_CHAIN option is also especially handy. Together they'll make file(INSTALL) both resolve symbolic links in the source files, and automatically recreate them in the destination path.
Example code
So all in all, you'd want to do something like this:
set(MY_DEPENDENCY_PATHS /path/one /path/two)
# Transfer the value of ${MY_DEPENDENCY_PATHS} into the install script
install(CODE "set(MY_DEPENDENCY_PATHS \"${MY_DEPENDENCY_PATHS}\")")
install(CODE [[
file(GET_RUNTIME_DEPENDENCIES
LIBRARIES $<TARGET_FILE:mylibtarget>
EXECUTABLES $<TARGET_FILE:myprogtarget>
RESOLVED_DEPENDENCIES_VAR _r_deps
UNRESOLVED_DEPENDENCIES_VAR _u_deps
DIRECTORIES ${MY_DEPENDENCY_PATHS}
)
foreach(_file ${_r_deps})
file(INSTALL
DESTINATION "${CMAKE_INSTALL_PREFIX}/lib"
TYPE SHARED_LIBRARY
FOLLOW_SYMLINK_CHAIN
FILES "${_file}"
)
endforeach()
list(LENGTH _u_deps _u_length)
if("${_u_length}" GREATER 0)
message(WARNING "Unresolved dependencies detected!")
endif()
]])
* – (Note that using the DIRECTORIES argument on a non-Windows system will cause CMake to emit a warning, as files' dependencies are supposed to be resolvable using only the current environment.)
If the code gets too complex, there's always the option to create a separate script file copy_deps.cmake in the ${CMAKE_CURRENT_SOURCE_DIR} and use install(SCRIPT copy_deps.cmake). (A previous version of this answer suggested using file(GENERATE...) to build the script — that won't work, as the file isn't written until after processing the CMakeLists.txt.)
Building onto this answer (thanks!) I created a recursive version for collecting all library dependencies and their dependants (and so on..) for a given executable:
install(CODE [[
function(install_library_with_deps LIBRARY)
file(INSTALL
DESTINATION "${CMAKE_INSTALL_PREFIX}/lib"
TYPE SHARED_LIBRARY
FOLLOW_SYMLINK_CHAIN
FILES "${LIBRARY}"
)
file(GET_RUNTIME_DEPENDENCIES
LIBRARIES ${LIBRARY}
RESOLVED_DEPENDENCIES_VAR RESOLVED_DEPS
UNRESOLVED_DEPENDENCIES_VAR UNRESOLVED_DEPS
)
foreach(FILE ${RESOLVED_DEPS})
if(NOT IS_SYMLINK ${FILE})
install_library_with_deps(${FILE})
endif()
endforeach()
foreach(FILE ${UNRESOLVED_DEPS})
message(STATUS "Unresolved from ${LIBRARY}: ${FILE}")
endforeach()
endfunction()
file(GET_RUNTIME_DEPENDENCIES
EXECUTABLES $<TARGET_FILE:myexecutable>
RESOLVED_DEPENDENCIES_VAR RESOLVED_DEPS
UNRESOLVED_DEPENDENCIES_VAR UNRESOLVED_DEPS
)
foreach(FILE ${RESOLVED_DEPS})
install_library_with_deps(${FILE})
endforeach()
foreach(FILE ${UNRESOLVED_DEPS})
message(STATUS "Unresolved: ${FILE}")
endforeach()
]])
I also think its relevant to note that some variables (like CMAKE_INSTALL_PREFIX) can be used in the inner scope as they are, while others (like CMAKE_PREFIX_PATH) need to be re-set explicitly.
Going from here one might want to exclude specific system directories, this here likely collects too much.
The library libwebrtc from https://github.com/cloudwebrtc/libwebrtc-build/blob/dev/CMakeLists.txt was built to be used with make; make install and the project which wants to use the library must later use the find_package from CMake.
I, however, want to change libwebrtc so it can be added as a git submodule into my current project as a custom library, as, for instance, https://github.com/itay-grudev/SingleApplication which is compiled when I type: cmake ..; make into a static/dynamic library and then linked in my main application. (The Qt library example I references earlier was confusing since this is build outside of my main project and only linked to afterwards - which is not what I want). Sorry for that confusion.
To be able to do that, I think that the ExternalProject_Add at https://github.com/cloudwebrtc/libwebrtc-build/blob/a24a5e5947658d43339d4bfd85d3f4c52fc71057/CMakeLists.txt#L100 must be changed into a add_library call.
The problem here is that the include_directories is used by the main project before the library has been completely built.
Question
How to rewrite libwebrtc to be used as a simple static library with proper build dependencies so that my main project is only compiled/linked after the libwebrtc build was finished and custom header files were generated in the CMAKE_CURRENT_BINARY_DIR of libwebrtc.
Or in other words, how to rewrite libwebrtc to be used without having to call make install for the library and then use find_package to use that library.
The hack (which is working already)
With this hack I am already able to:
Build the library from my parent project
Depend on the generated header files which exist only after the libwebrtc has been built completely (thus, delay main project building until dependencies are meet)
Depend on the generated webrtc.a static library for the linker step
I imaging that make install will work since libwebrtc is statically linked.
add_dependencies(${PROJECT_NAME} libwebrtcx)
add_subdirectory(third-party/libwebrtcx)
include_directories(
${CMAKE_BINARY_DIR}/sources/third-party/libwebrtcx/include/webrtc
${CMAKE_BINARY_DIR}/sources/third-party/libwebrtcx/include/webrtc/third_party/libyuv/include/
${CMAKE_BINARY_DIR}/sources/third-party/libwebrtcx/webrtc/src/third_party/abseil-cpp
)
add_library(libwebrtc STATIC IMPORTED)
set_property(TARGET libwebrtc PROPERTY IMPORTED_LOCATION "${CMAKE_BINARY_DIR}/sources/third-party/libwebrtcx/webrtc/src/out/Release/obj/libwebrtc.a")
target_link_libraries(${PROJECT_NAME} libwebrtc)
Note: It requires to rename the libwebrtc project to libwebrtcx and also the ExternalProject_Add at https://github.com/cloudwebrtc/libwebrtc-build/blob/a24a5e5947658d43339d4bfd85d3f4c52fc71057/CMakeLists.txt#L100 must be renamed to libwebrtcx.
Note: It also requires to rename all CMAKE_BINARY_DIR into CMAKE_CURRENT_BINARY_DIR and CMAKE_SOURCE_DIR to CMAKE_CURRENT_SOURCE_DIR. Details can be found here: CMake: Using add_subproject with a library using Include ends up in wrong relative path
I have a CMake configuration file building two libraries:
a third-party library (here called ThirdPartyLib) containing a real-time OS / board support package from a supplier. It is built outside CMake using the autotools toolchain.
an extended version of the former library (here called ExtendedThirdPartyLib)
Unfortunately, some source code that I need (various tools) are not built in the ordinary build script for (1). Since I don't want to mess with the suppliers build script I want to add another library (2), building the missing files and thus extending the library from the supplier.
I want to able to do something like this in CMakeFiles.txt:
cmake_minimum_required(VERSION 3.2)
project(bsp)
include(ExternalProject)
ExternalProject_Add(
ThirdPartyLib
URL <http://some.url/bsp.tar.bz2
BUILD_COMMAND make -C ../external/ThirdPartyLib/src
)
set_target_properties(ThirdPartyLib PROPERTIES EXCLUDE_FROM_ALL TRUE)
add_library(ExtendedThirdPartyLib
${CMAKE_CURRENT_BINARY_DIR}/some/path/missing_file1.c
${CMAKE_CURRENT_BINARY_DIR}/some/path/missing_file2.c
)
add_dependencies(ExtendedThirdPartyLib ThirdPartyLib)
target_include_directories(ExtendedThirdPartyLib PUBLIC
${CMAKE_CURRENT_BINARY_DIR}/some/path/include
)
target_link_libraries(ExtendedThirdPartyLib ThirdPartyLib)
The problem here is that the path to missing_file1.c and missing_file2.c are not valid when CMake is generating the build files (they are extracted from the tarball from the supplier). CMake exits with an error output saying: "Cannot find source file".
Is there a neat way to make this work? I.e. is it possible to convince CMake that certain non-existant input files will exist when building of the library begins? Or is there any other recommended way to solve this issue?
(I have temporary made local copies of the files I need to build from the suppliers tarball, but that is of course not a good solution. If those files are changed in future versions of the suppliers package and I forget to overwrite my local copies it could be a horrible mess...
Another "solution" would be to create a small makefile outside CMake and use another ExternalProject_Add in the CMakeFiles.txt somehow. But that's not a good solution either, e.g. if compile and linker flags are modified I need to remember to change the makefile too.)
Personally, I dislike the ExternalProject_Add command, because it does way too many things for my taste, but I've digressed.
What if you do something like this, where bar is simulating your ExtendedThirdPartyLib target, since it depends on generated files
cmake_minimum_required(VERSION 3.11)
project(lol C)
set(SOURCES lol.c) # only this file exists
add_library(lol ${SOURCES})
set(FOO_FILES "foo1.c" "foo2.c")
add_custom_command(OUTPUT ${FOO_FILES}
COMMAND ${CMAKE_COMMAND} -E touch ${FOO_FILES}
WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
COMMENT "Creating ${FOO_FILES}"
VERBATIM)
add_custom_target(foo DEPENDS ${FOO_FILES})
add_library(bar ${FOO_FILES})
add_dependencies(bar foo)
target_link_libraries(lol bar)
The whole approach hinges on the fact that the method, where produced/generated files are procured, is explicitly defined via the custom command and associated custom target.
You should modify the custom command to extract the required files (e.g. could even call some external script) from the tarball (which might require downloading with curl or something similar).
I've got a cmake project that pretty much looks like this:
cmake_minimum_required(VERSION 3.0)
SET(CMAKE_DEBUG_POSTFIX "_d")
include_directories(../TransfunctionerProject)
include_directories(../TransmogrifierProject)
set(Libraries
ContinuumTransfunctioner
Transmogrifier
)
set(SourceFiles
Wrapper.cpp
Logger.cpp
)
add_library(Frobnigator SHARED ${SourceFiles})
add_library(FrobnigatorStatic STATIC ${SourceFiles})
set_target_properties(FrobnigatorStatic PROPERTIES OUTPUT_NAME Frobnigator)
target_link_libraries(Frobnigator ${Libraries})
Where ContinuumTransfunctioner and Transmogrifier projects include the debug postfix directive SET(CMAKE_DEBUG_POSTFIX "_d") so that libContinuumTransfunctioner_d.so and libTransmogrifier_d.so both exist.
The problem is that the current project appears to be linking against the static library without the _d suffix and complains:
/usr/bin/ld: cannot find -lContinuumTransfunctioner
The Libraries that you pass into the target_link_libraries call are interpreted as filenames, not as target names.
This is the unfortunate fallback for that call in CMake: If you pass a random string to it, that cannot be interpreted in a meaningful way, CMake will always assume it to be plain library name. Sometimes this is just what you want, but the name has to be an exact match for an existing library. The whole debug postfix magic will be lost here.
What you might have wanted to do was to pass a library target name instead. This will trigger a much smarter handling of the dependency and would solve your problem. However, that only works if the library is a known target in the context of the target_link_libraries call. You can easily check this as follows:
if(TARGET ContinuumTransfunctioner)
message("Library target name")
else()
message("Plain library name")
endif()
target_link_libraries(Frobnigator ContinuumTransfunctioner)
So how do you get to the target name case? This depends on how your build is structured. If the library is being built as part of your CMake run, simply make sure that the corresponding add_library call is performed from a subdirectory that is pulled in via add_subdirectory from the file that performs the target_link_libraries call.
If the library in question is an external dependency, you need to build an imported target that carries all the relevant information where to find the library files (including any potential debug postfixes). This can be a bit cumbersome to do manually, so if you can, you might want to use CMake's packaging mechanism to generate this automatically as part of the library's build process.
Here's the solution, courtesy of the good people on the cmake mailing list:
# Note:
# $<$<CONFIG:Debug>:_d> is called a generator expression.
# It outputs _d if the build is debug.
#
set(Libraries
ContinuumTransfunctioner$<$<CONFIG:Debug>:_d>
Transmogrifier$<$<CONFIG:Debug>:_d>
)
I have the following directory layout:
main_folder
+ static_lib1
+ executable
Both 'static_lib1' and 'executable' have a full CMakeLists so that they can be
built independently.
The 'executable' depends on 'static_lib1'. It uses find_package() to locate 'static_lib1'.
The main_folder contains a CMakeLists that includes both 'static_lib1' and 'executable' via add_subdirectory for conveniently building the whole project in one go.
Everything works fine if I manually build 'static_lib1' and then 'executable'. But when running the CMakeLists from the main folder, I get an error because find_package is unable to find the library files from 'static_lib1' which have not yet been built.
How can I resolve this while keeping the CMakeLists files separate (i.e. without including the static_lib's CMakeLists from the executable's CMakeLists)?
In executable's CMakeLists.txt you can check if you are building stand-alone or as part of project:
if( CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR )
# stand-alone build
find_package(static_lib1)
else()
include_directories(../static_lib1)
link_directories(../static_lib1)
...
target_link_libraries(executable static_lib1)
endif()
Switch from a file-based approach to a target-based approach for handling the dependency from executable to static_lib1.
The original problem occurred because executable called find_package for locating static_lib1, which then attempted to fill a variable like STATIC_LIB1_LIBRARY with the paths to the library files by calling find_library. executable then consumes the content of that variable in a target_link_libraries(executable ${STATIC_LIB1_LIBRARY}) call. The problem here is, since those library files only get generated as part of the build, that call to find_library will not be able to find anything.
Building executable needs to support two scenarios here:
Building standalone, where a pre-compiled version of static_lib1 is located somewhere on the disc.
Building from main_folder, where both executable and static_lib1 are part of the same build.
The approach from the question supports scenario 1, but not scenario 2.
Instead of using using a variable to communicate a dependency between the two builds, use a target. The CMakeLists.txt for static_lib1 likely creates a library target like add_library(static_lib1 [...]). In executable we now simply do target_link_libraries(executable PUBLIC static_lib1). This is sufficient to support scenario 2.
To also allow for scenario 1 at the same time, we look at the call to find_package(static_lib1) in the CMakeLists.txt for executable. Instead of providing a variable like before, this call now needs to provide a target static_lib1 for consumption.
So we adapt the find script for static_lib1 to the following behavior:
If a target static_lib1 already exists, there's nothing to be done and the find script can just return (this is scenario 2).
Otherwise, we call find_library to locate the library file on disc (as before in the original approach) and then create a new imported target: add_library(static_lib1 STATIC IMPORTED). We then configure all relevant properties of the static library to that target. For instance, to add the location of the library file, we could do
set_target_properties(static_lib1 PROPERTIES
IMPORTED_LINK_INTERFACE_LANGUAGES "CXX"
IMPORTED_LOCATION ${STATIC_LIB1_LIBRARY}
)
To support multi-config generators like MSVC, instead of setting IMPORTED_LOCATION and IMPORTED_LINK_INTERFACE_LANGUAGES, you will want to set the configuration specific properties like IMPORTED_LOCATION_DEBUG and IMPORTED_LOCATION_RELEASE instead. Since this can get quite tedious to do manually, you can have CMake generate this information (and a bunch of other convenient stuff) for you in a package script. The find mechanism for package scripts works slightly different under the hood, but the code in the CMakeLists.txt for executable will look just the same, a simple call to find_package(static_lib1). The main difference is that this call will then not dispatch to a hand-written find script, but to a package script that was automatically generated by CMake as part of the build process of static_lib1.
I guess I will leave this answer for posterity since only recently I have searched for a solution to this problem and found out that...
Since CMake 3.24 it is possible!
It is possible to override subsequent calls to find_package() with FetchContent_Declare() flag OVERRIDE_FIND_PACKAGE.
Your
add_subdirectory("path/to/static_lib1")
call has to be replaced in main_folder/CMakeLists.txt with:
include(FetchContent)
FetchContent_Declare(
static_lib1
SOURCE_DIR "path/to/static_lib1"
OVERRIDE_FIND_PACKAGE
)
Any calls to find_package(static_lib1) will call FetchContent_MakeAvailable() for you, virtually making it identical to add_subdirectory() call.
You can read more about OVERRIDE_FIND_PACKAGE in CMake documentation.