Transitive dependencies with different linkage to third-party libraries - cmake

I have a little bit of a convoluted question. I have a 3rd party dependency, which comes in static (libthird.a) and shared pic form (libthird.so).
I have a library, util, that depends on libthird.
And I have applications that depend on util that want to link libthird statically, and I have some shared libraries I need to produce that depend on util and need to link libthird dynamically.
My current (working) approach is something like the following:
add_library(third INTERFACE)
target_link_libraries(third INTERFACE /path/to/libthird.a)
add_library(third_shared INTERFACE)
target_link_libraries(third_shared INTERFACE /path/to/libthird.so)
add_library(util ${UTIL_SOURCES})
add_library(util_shared ${UTIL_SOURCES}) # same sources again!!
target_link_libraries(util PUBLIC third)
target_link_libraries(util_shared PUBLIC third_shared)
add_executable(some_app ...)
target_link_libraries(some_app PRIVATE util)
add_library(some_shared_object ...)
target_link_libraries(some_shared_object PUBLIC util_shared)
This works. But I'm building util (and, in reality, another half dozen libraries or so) twice... just to get different linker dependencies. Is there a saner way of doing this in cmake?
If I just target_link_libraries() on the top-level some_app and some_shared_object, I get the linker flags emitted in the wrong order, since util does depend on third.

The approach that you're taking is definitely one I've seen and also used myself before. It is fine if it is used for a small archive and/or one-off usage.
For the following examples, I assume a project structure like the following:
foo/
CMakeLists.txt
include/
foo.h
src/
foo.c
So, I've also (naively?) used the following approach based on the knowledge that you can create (on Unix-based systems at least) a shared library from a static archive.
project(foo C)
set(SOURCES
"src/foo.c")
set(LIBNAME "foo")
add_library(${LIBNAME} STATIC ${SOURCES})
target_include_directories(${LIBNAME} PUBLIC "include")
target_compile_options(${LIBNAME} PUBLIC "-fPIC")
#
get_property(CUR_PREFIX TARGET ${LIBNAME} PROPERTY PREFIX)
get_property(CUR_SUFFIX TARGET ${LIBNAME} PROPERTY SUFFIX)
get_property(CUR_NAME TARGET ${LIBNAME} PROPERTY NAME)
get_property(CUR_OUTPUT_NAME TARGET ${LIBNAME} PROPERTY OUTPUT_NAME)
get_property(CUR_ARCHIVE_OUTPUT_NAME TARGET ${LIBNAME} PROPERTY ARCHIVE_OUTPUT_NAME)
message(STATUS "prefix: ${CUR_PREFIX}")
message(STATUS "suffix: ${CUR_SUFFIX}")
message(STATUS "name: ${CUR_NAME}")
message(STATUS "output name: ${CUR_OUTPUT_NAME}")
message(STATUS "archive name: ${CUR_ARCHIVE_OUTPUT_NAME}")
add_custom_command(TARGET ${LIBNAME} POST_BUILD
COMMAND ${CMAKE_C_COMPILER} -shared -o libfoo.so -Wl,--whole-archive libfoo.a -Wl,--no-whole-archive
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
My unresolved problem with this approach is that I haven't found a reliable and portable way to get the name of the static archive (libfoo.a). Hence, all those message commands for the target properties; they might behave differently on your platforms.
The most efficient way, that is also well-supported by cmake, is to use an object library:
# cmake file
cmake_minimum_required(VERSION 3.2)
project(foo C)
set(SOURCES
"src/foo.c")
set(LIBNAME "foo")
set(LIBNAME_OBJ "${LIBNAME}_obj")
add_library(${LIBNAME_OBJ} OBJECT ${SOURCES})
target_include_directories(${LIBNAME_OBJ} PUBLIC "include")
target_compile_options(${LIBNAME_OBJ} PUBLIC "-fPIC")
#
add_library(${LIBNAME} SHARED $<TARGET_OBJECTS:${LIBNAME_OBJ}>)
add_library(${LIBNAME}_static STATIC $<TARGET_OBJECTS:${LIBNAME_OBJ}>)
All examples were tested with cmake version 3.6.1.
Hope this helps. Let us know, if you've found a better way.

Related

Why use add_library({tgt} IMPORTED) versus target_link_libraries( -l {.so | .a})?

What is the purpose of using the statement:
add_library(<tgt> [SHARED|STATIC] IMPORTED)
From what I have found even if you create an imported library target above you still would need to specify the specific location of the actual .so or .a. This would take at least 3 cmake commands to link to an executable and the compiler still would not automatically search through the common include directories on your OS.
Example:
cmake code to link IMPORTED lib
From the CMake documentation I understand there are really 3 ways to link a library that is not built as a target in a subproject of your overall application/library.
CMake target_link_libraries() documentation
Using a CMake package for one of the shipped package scripts.
Using a linker flag:
target_link_libraries(<tgt> [SHARED|STATIC|...] -lncursesw)
Or using the IMPORTED library method (showcased in code at top).
A major difference when using the second method is that it only takes a single line of code and will search through all of your compiler's predefined include directories on you OS. Could anyone help me understand why the add_library() method is used?
Additional Realated SO Posts:
Include directories for IMPORTED libs
CMake imported library behavior
You should use add_library(<tgt> [SHARED|STATIC] IMPORTED) whenever you need to set properties such as dependencies, compile definitions, compile flags etc for <tgt>, and/or by extension, any targets that are linking against <tgt>.
Let's say you have two static libraries; libfoobar.a and libraboof.a, where libfoobar.a requires libraboof.a. Let's also say that these libraries contain some features that are enabled by -DSOME_FEATURE.
add_library(raboof STATIC IMPORTED)
set_target_properties(raboof PROPERTIES
IMPORTED_LOCATION <path-to-libraboof.a>
INTERFACE_COMPILE_DEFINITIONS "SOME_FEATURE"
)
add_library(foobar STATIC IMPORTED)
set_target_properties(foobar PROPERTIES
IMPORTED_LOCATION <path-to-libfoobar.a>
INTERFACE_LINK_LIBRARIES raboof
)
So when you link against libfoobar.a:
add_executable(my_app main.cpp)
target_link_libraries(my_app foobar)
CMake will make sure to link all dependencies in the correct order and will in this case also append -DSOME_FEATURE to the compile flags when you build my_app. Note that since we added libraboof.a as a dependency to libfoobar.a, -DSOME_FEATURE is added to any target that link against libfoobar.a through the transitive property.
If you don't use add_library(<tgt> <SHARED|STATIC> IMPORTED) in a scenario like this, you would have to manage any dependencies and required build options yourself for each target, which is quite error-prone.
This method is also often used in Config-modules for multi-component libraries to manage dependencies between the components.

What to do with the output of ExternalProject_add? IMPORTED vs INTERFACE libraries

This is my topmost CMakeLists.txt:
cmake_minimum_required(VERSION 3.10)
project(test LANGUAGES Fortran)
add_executable(main main.f90)
add_subdirectory(external)
target_link_libraries(main extlib)
Subdirectory "external" defines one external project, which produces some libraries and header files. What is the best practice of collecting the output of ExternalProject_add so that I could link that to the main executable? Currently, I'm using an INTERFACE library, like this:
include(ExternalProject)
ExternalProject_add(my-external
SOURCE_DIR ext_source
CONFIGURE_COMMAND
${CMAKE_CURRENT_LIST_DIR}/configure
--prefix=${CMAKE_CURRENT_BINARY_DIR}
BUILD_COMMAND make)
add_library(extlib INTERFACE)
add_dependencies(extlib my-external)
target_include_directories(extlib INTERFACE ${CMAKE_CURRENT_BINARY_DIR}/include)
foreach(_lib lib1 lib2 lib3)
target_link_libraries(extlib INTERFACE
${CMAKE_CURRENT_BINARY_DIR}/lib/${_lib}.a)
endforeach()
Searching the web, it seems to me that people often use IMPORTED libraries instead for capturing the results of ExternalProject_add. However, you can only connect one IMPORTED library with one file produced by ExternalProject_add and any header directories would need to be separately propagated back to the parent directory so that main could use them. It seems to me than an INTERFACE library is better here because you can glob everything produced by the external project into a single target. In my actual project I have several external projects and ideally I would like to have one target per external project to keep things clean.
In general, should I use IMPORTED or INTERFACE libraries when dealing with external libraries and what would be the main benefits?
EDIT
Based on the dicussion below, I made an attempt using imported libraries only:
foreach(_lib lib1 lib2 lib3)
add_library(${_lib} STATIC IMPORTED GLOBAL)
add_dependencies(${_lib} my-external)
set_target_properties(${_lib} PROPERTIES
IMPORTED_LOCATION ${CMAKE_CURRENT_BINARY_DIR}/lib/${_lib}.a)
endforeach()
set_target_properties(lib1 PROPERTIES
INTERFACE_INCLUDE_DIRECTORIES ${CMAKE_CURRENT_BINARY_DIR}/include)
target_link_libraries(lib1 INTERFACE lib2 lib3)
It takes about the same effort as with an interface library and I guess is ultimately a matter of style.

Library depend on a header file

Let's assume I have a project with a series of libraries. I also need to generate a header file that will be used by all of these projects. So I created a CMake file, like this:
project(main)
add_subdirectory(sub_1)
add_subdirectory(sub_2)
# ...
add_subdirectory(sub_n)
add_custom_command(
OUTPUT CustomHeader.h
COMMENT "Generating custom header for all the libraries"
COMMAND ...)
add_library(${PROJECT_NAME} STATIC ${OBJECT_LIST})
The problem is, that I don't know how to tell CMake to run my custom command (that generates this CustomHeader.h) before it would try to build the libraries in the subfolders.
I tried add_custom_target(TARGET MyPrebuild PRE_BUILD ...) but I'm running on Linux, and this option only works on Windows platform according to the documentation.
add_dependencies only work between targets, and not a target and a single file.
I could, in theory, add the header to be among the source files of the individual libraries (in the sub_1, .., sub_n folders) but it feels wrong, as the header is not required to be part of those libraries.
So I just have no idea how I can make a library depend on an include file, that is not part of it.
Any tips how I can overcome this problem?
For make header file (re)built before a library in subdirectory is compiled, you may create target, which builds the file, and make the library dependent from the target:
# *CMakeLists.txt*
# ...
add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h ...)
add_custom_target(generate_custom_header DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h)
# *sub/CMakeLists.txt*
# ...
add_library(libA ...)
add_dependencies(libA generate_custom_header)
Instead of using add_dependencies, you may create header-only library which "implements" you header and link with it:
# *CMakeLists.txt*
# ...
add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h ...)
add_custom_target(generate_custom_header DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h)
add_library(libCustom INTERFACE) # Header only library
add_dependencies(libCustom generate_custom_header) # which depends on generated header
# You may even assign include directories for the header-only library
target_include_directories(libCustom INTERFACE ${CMAKE_CURRENT_BINARY_DIR})
# *sub/CMakeLists.txt*
# ...
add_library(libA ...)
target_link_libraries(libA libCustom) # Common linking with a header-only library.
Note, that INTERFACE library is a "fake" - it is never created by itself. Instead, all "features" of INTERFACE library are just propagated to its users.
I would suggest to add another library target that will both keep track of the generated headers and will help to properly configure other libraries to know where to find them (i.e. target_include_directories).
cmake_minimum_required(VERSION 3.0)
project(testable)
set(CustomHeaderInPath ${CMAKE_CURRENT_SOURCE_DIR}/CustomHeader.example)
set(CustomHeaderPath ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h)
add_custom_command(
OUTPUT ${CustomHeaderPath}
COMMAND cp ${CustomHeaderInPath} ${CustomHeaderPath}
COMMENT "Generated file"
DEPENDS ${CustomHeaderInPath})
add_library(CustomHeaderLibrary ${CustomHeaderPath})
target_include_directories(CustomHeaderLibrary PUBLIC ${CMAKE_CURRENT_BINARY_DIR})
set_target_properties(CustomHeaderLibrary PROPERTIES LINKER_LANGUAGE C)
add_library(LibA a.c)
target_link_libraries(LibA CustomHeaderLibrary)
add_library(LibB b.c)
target_link_libraries(LibB CustomHeaderLibrary)
add_executable(${PROJECT_NAME} main.c)
target_link_libraries(${PROJECT_NAME} PUBLIC LibA LibB)
Note that I had to explicitly set the LINKER_LANGUAGE of the new target as cmake won't be able to deduce it properly if no c or cpp files are added to the library.

How to make imported target GLOBAL afterwards?

From the FindBoost.cmake module of CMake 3.8:
foreach(COMPONENT ${Boost_FIND_COMPONENTS})
if(_Boost_IMPORTED_TARGETS AND NOT TARGET Boost::${COMPONENT})
string(TOUPPER ${COMPONENT} UPPERCOMPONENT)
if(Boost_${UPPERCOMPONENT}_FOUND)
if(Boost_USE_STATIC_LIBS)
add_library(Boost::${COMPONENT} STATIC IMPORTED)
else()
# Even if Boost_USE_STATIC_LIBS is OFF, we might have static
# libraries as a result.
add_library(Boost::${COMPONENT} UNKNOWN IMPORTED)
endif()
and the corresponding comment from the docu of that module:
It is important to note that the imported targets behave differently than variables created by this module: multiple calls to find_package(Boost) in the same directory or sub-directories with different options (e.g. static or shared) will not override the values of the targets created by the first call.
I see the rational for having the targets not being GLOBAL.
However, what is the preferred way of making them global?
I'm used to defining the dependencies of my project in a sub-directory including any find_package(...) calls. Consequently, the Boost imported targets are not available in another directory, e.g. /tests/CMakeLists.txt:
<project_root>
/3rdparty
/git-submodule-of-a-small-lib
/CMakeLists.txt
/include
/...
/tests
/CMakeLists.txt
/CMakeLists.txt
There is a IMPORTED_GLOBAL target property for this in CMake >= 3.11:
set_target_properties(Boost::unit_test_framework PROPERTIES IMPORTED_GLOBAL TRUE)
For older versions: find_package() uses standard add_library() calls, so you can always change/extend its functionality to have IMPORTED targets always GLOBAL with something like:
3rdparty\CMakeLists.txt
function(add_library)
set(_args ${ARGN})
if ("${_args}" MATCHES ";IMPORTED")
list(APPEND _args GLOBAL)
endif()
_add_library(${_args})
endfunction()
find_package(Boost REQUIRED COMPONENTS unit_test_framework)
Disclaimer
As #CraigScott has commented overwriting CMake's build-in functions is dangerous:
[CMake] infinite loop when using function overriding
CMake Issue #14357: Defining an override macro/function of add_library more than once causes a segmentation fault
References
CMake Issue #1254: Add new target-property IMPORTED_GLOBAL
CMake Issue #1222: [Threads, Boost] Option to create IMPORTED targets with GLOBAL scope
CMake Issue #17256: Possibility to promote IMPORTED target to IMPORTED GLOBAL
I managed to workaround the problem of having the imported Boost targets not available in the global project scope by including 3rdparty/CMakeLists.txt not by add_subdirectory(3rdparty) but via include(3rdparty/CMakeLists.txt) as this evaluates 3rdparty/CMakeLists.txt in the caller's scope.

CMake: how create a single shared library from all static libraries of subprojects?

I have the following layout:
top_project
+ subproject1
+ subproject2
Each of subproject1 and subproject2 creates a static library. I would like to link these static libraries in a single shared library at the top_project level.
The information I gathered so far is:
Either compile using -fPic (necessary on everything but Windows) in order to create position-independent code which will allow linking the static libraries into a single shared library or decompress all static libraries (e.g. using ar) and re-link them into a shared library (which I think is an inelegant & non-portable solution)
All source files must be given explicitly to the add_library command: for some reason which I cannot comprehend, simply writing add_library(${PROJECT_NAME} SHARED subproject1 subproject2) does not work as expected (it essentially creates an empty library & does not register the dependencies properly)
There is an OBJECT library feature in CMake but I don't think it's purpose is really to do what I want.
Any thoughts?
OK, I figured it out: this is much more painful than it should be. Until very recently, people at Kitware didn't understand why anyone would ever want to create a DLL from static libs. Their argument is that there should always be source files in the main (e.g. top_project in my case) directory because it is effectively a project of its own. I see things differently & I need to break top_project into smaller subprojects which should not exist independently (i.e. there is no point in creating a full-blown project for them & add them using ExternalProject_Add). Besides, when I ship my shared library (for use, e.g. with a Java Native Interface), I don't want to ship dozens of shared libraries because that would amount to exposing the internal layout of my project. Anyway, having - I think - made a case for creating a shared library from static libraries, I'll proceed to the technical details.
In the CMakeLists.txt of subproject1 and subproject2, you should create your target using the OBJECT library feature (introduced in CMake 2.8.8):
add_library(${PROJECT_NAME} OBJECT ${SRC})
where SRC designates the list of source files (note that these should be set explicitly in the CMakeLists.txt file as it allows make to re-launch CMake when a modification of CMakeLists.txt is detected, e.g. when adding or removing a file)
In the top_project, add the subprojects using:
add_subdirectory(subproject1)
add_subdirectory(subproject2)
In order to see the symbols from the static library, use:
set(CMAKE_SHARED_LINKER_FLAGS "-Wl,--export-all-symbols")
You can then create the shared library using:
add_library(${PROJECT_NAME} SHARED $<TARGET_OBJECTS:subproject1>
$<TARGET_OBJECTS:subproject2>)
I've found that any "normal" library (i.e. not object) needs to be added in a separate add_library command, otherwise it is simply ignored.
For executables, you can use:
add_executable(name_of_executable $<TARGET_OBJECTS:subproject1>
$<TARGET_OBJECTS:subproject2>)
set(LINK_FLAGS ${LINK_FLAGS} "-Wl,-whole-archive")
target_link_libraries(name_of_executable ${PROJECT_NAME}
I repeat that this only works as of version 2.8.8 of CMake. Just as well CMake manages the dependencies extremely well & is cross-platform because it's not much less painful than plain old Makefiles & certainly less flexible.
My solution is simply to add /WHOLEARCHIVE, -all_load, or --whole-archive to the linker flags, so that when your main library is linked, all of the sub libraries are included, including all their symbols (the default behaviour is to only include symbols of the sub libraries that are used by the main library. For example:
Source Files
$ echo "void Func1() { }" > source1.cpp
$ echo "void Func2() { }" > source2.cpp
$ echo "void Func3() { }" > source3.cpp
$ echo "void Func4() { }" > source4.cpp
Naive CMakeLists.txt
cmake_minimum_required(VERSION 3.7)
# The 'sub' libraries, e.g. from an `add_subdirectory()` call.
add_library(sublib_a STATIC source1.cpp source2.cpp)
add_library(sublib_b STATIC source3.cpp source4.cpp)
# The main library that contains all of the sub libraries.
add_library(mainlib SHARED)
target_link_libraries(mainlib sublib_a sublib_b)
Running it (on OSX):
$ make VERBOSE=1
...
[100%] Linking CXX shared library libmainlib.dylib
/usr/local/Cellar/cmake/3.7.1/bin/cmake -E cmake_link_script CMakeFiles/mainlib.dir/link.txt --verbose=1
/Library/Developer/CommandLineTools/usr/bin/c++ -dynamiclib -Wl,-headerpad_max_install_names -o libmainlib.dylib -install_name #rpath/libmainlib.dylib libsublib_a.a libsublib_b.a
[100%] Built target mainlib
$ nm libmainlib.dylib | grep Func
$
Correct CMakeLists.txt
Append this:
# By default, symbols provided by the sublibs that are not used by mainlib (which is all of them in this case)
# are not used. This changes that.
if (WIN32)
set_target_properties(mainlib PROPERTIES
LINK_FLAGS "/WHOLEARCHIVE"
)
elseif (APPLE)
set_target_properties(mainlib PROPERTIES
LINK_FLAGS "-Wl,-all_load"
)
else ()
set_target_properties(mainlib PROPERTIES
LINK_FLAGS "-Wl,--whole-archive"
)
endif ()
Running it (note the extra -all_load):
$ make VERBOSE=1
[100%] Linking CXX shared library libmainlib.dylib
/usr/local/Cellar/cmake/3.7.1/bin/cmake -E cmake_link_script CMakeFiles/mainlib.dir/link.txt --verbose=1
/Library/Developer/CommandLineTools/usr/bin/c++ -dynamiclib -Wl,-headerpad_max_install_names -Wl,-all_load -o libmainlib.dylib -install_name #rpath/libmainlib.dylib libsublib_a.a libsublib_b.a
[100%] Built target mainlib
$ nm libmainlib.dylib | grep Func
0000000000001da0 T __Z5Func1v
0000000000001db0 T __Z5Func2v
0000000000001dc0 T __Z5Func3v
0000000000001dd0 T __Z5Func4v
Note that I've only actually tested -all_load so far, and /WHOLEARCHIVE is an MSVC 2015 option.
Another way of doing it.
This way seems simpler, but I'm not sure how perfect it is:
https://stackoverflow.com/a/14347487/602340
Another way of doing it is to provide the path of the source files and the header files of all your projects, and build them together to produce the .so . This is usually the recommended way, instead of creating the static libraries and then a shared library out of those.
Basically you should do the following:
FILE(GLOB subproject1_sources
<sub_project1_lib_sources_dir>/file1.c
<sub_project1_lib_sources_dir>/file2.c //... etc
)
FILE(GLOB subproject2_sources
<sub_project2_lib_sources_dir>/file1.c
<sub_project2_lib_sources_dir>/file2.c //... etc
)
FILE(GLOB topProject_sources
<top_project_lib_sources_dir>/file1.c
<top_project_lib_sources_dir>/file2.c //... etc
)
include_directories("<sub_project1_lib_sources_dir>")
include_directories("<sub_project2_lib_sources_dir>")
include_directories("<top_project_lib_sources_dir>") //should be "." if you're building from here
add_library(topProject SHARED ${topProject_sources} ${subproject1_sources} ${subproject2_sources})
I am not sure if this is what suits your need, but cmake also offers INTERFACE libraries, which serve (among others) precisely this need.
add_library(bundle INTERFACE)
target_link_libraries(bundle lib1 lib2)
will bundle lib1 and lib2 into a single library, and inherit the PUBLIC and INTERFACE section of lib1 and lib2.
More info here.
Add following macro to your cmake scripts.
MACRO (TARGET_LINK_LIBRARIES_WHOLE_ARCHIVE target)
IF (WIN32)
FOREACH (arg IN LISTS ARGN)
SET_TARGET_PROPERTIES(
${target} PROPERTIES LINK_FLAGS "/WHOLEARCHIVE:${lib}"
)
ENDFOREACH ()
ELSE ()
IF (APPLE)
SET(LINK_FLAGS "-Wl,-all_load")
SET(UNDO_FLAGS "-Wl,-noall_load")
ELSE ()
SET(LINK_FLAGS "-Wl,--whole-archive")
SET(UNDO_FLAGS "-Wl,--no-whole-archive")
ENDIF ()
TARGET_LINK_LIBRARIES(${target} ${LINK_FLAGS} ${ARGN} ${UNDO_FLAGS})
ENDIF ()
ENDMACRO ()
Then, in CMakeLists.txt/*.cmake, you can use like this TARGET_LINK_LIBRARIES_WHOLE_ARCHIVE(target libs...)