CMake find_package dependency on subproject - cmake

I have the following directory layout:
main_folder
+ static_lib1
+ executable
Both 'static_lib1' and 'executable' have a full CMakeLists so that they can be
built independently.
The 'executable' depends on 'static_lib1'. It uses find_package() to locate 'static_lib1'.
The main_folder contains a CMakeLists that includes both 'static_lib1' and 'executable' via add_subdirectory for conveniently building the whole project in one go.
Everything works fine if I manually build 'static_lib1' and then 'executable'. But when running the CMakeLists from the main folder, I get an error because find_package is unable to find the library files from 'static_lib1' which have not yet been built.
How can I resolve this while keeping the CMakeLists files separate (i.e. without including the static_lib's CMakeLists from the executable's CMakeLists)?

In executable's CMakeLists.txt you can check if you are building stand-alone or as part of project:
if( CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR )
# stand-alone build
find_package(static_lib1)
else()
include_directories(../static_lib1)
link_directories(../static_lib1)
...
target_link_libraries(executable static_lib1)
endif()

Switch from a file-based approach to a target-based approach for handling the dependency from executable to static_lib1.
The original problem occurred because executable called find_package for locating static_lib1, which then attempted to fill a variable like STATIC_LIB1_LIBRARY with the paths to the library files by calling find_library. executable then consumes the content of that variable in a target_link_libraries(executable ${STATIC_LIB1_LIBRARY}) call. The problem here is, since those library files only get generated as part of the build, that call to find_library will not be able to find anything.
Building executable needs to support two scenarios here:
Building standalone, where a pre-compiled version of static_lib1 is located somewhere on the disc.
Building from main_folder, where both executable and static_lib1 are part of the same build.
The approach from the question supports scenario 1, but not scenario 2.
Instead of using using a variable to communicate a dependency between the two builds, use a target. The CMakeLists.txt for static_lib1 likely creates a library target like add_library(static_lib1 [...]). In executable we now simply do target_link_libraries(executable PUBLIC static_lib1). This is sufficient to support scenario 2.
To also allow for scenario 1 at the same time, we look at the call to find_package(static_lib1) in the CMakeLists.txt for executable. Instead of providing a variable like before, this call now needs to provide a target static_lib1 for consumption.
So we adapt the find script for static_lib1 to the following behavior:
If a target static_lib1 already exists, there's nothing to be done and the find script can just return (this is scenario 2).
Otherwise, we call find_library to locate the library file on disc (as before in the original approach) and then create a new imported target: add_library(static_lib1 STATIC IMPORTED). We then configure all relevant properties of the static library to that target. For instance, to add the location of the library file, we could do
set_target_properties(static_lib1 PROPERTIES
IMPORTED_LINK_INTERFACE_LANGUAGES "CXX"
IMPORTED_LOCATION ${STATIC_LIB1_LIBRARY}
)
To support multi-config generators like MSVC, instead of setting IMPORTED_LOCATION and IMPORTED_LINK_INTERFACE_LANGUAGES, you will want to set the configuration specific properties like IMPORTED_LOCATION_DEBUG and IMPORTED_LOCATION_RELEASE instead. Since this can get quite tedious to do manually, you can have CMake generate this information (and a bunch of other convenient stuff) for you in a package script. The find mechanism for package scripts works slightly different under the hood, but the code in the CMakeLists.txt for executable will look just the same, a simple call to find_package(static_lib1). The main difference is that this call will then not dispatch to a hand-written find script, but to a package script that was automatically generated by CMake as part of the build process of static_lib1.

I guess I will leave this answer for posterity since only recently I have searched for a solution to this problem and found out that...
Since CMake 3.24 it is possible!
It is possible to override subsequent calls to find_package() with FetchContent_Declare() flag OVERRIDE_FIND_PACKAGE.
Your
add_subdirectory("path/to/static_lib1")
call has to be replaced in main_folder/CMakeLists.txt with:
include(FetchContent)
FetchContent_Declare(
static_lib1
SOURCE_DIR "path/to/static_lib1"
OVERRIDE_FIND_PACKAGE
)
Any calls to find_package(static_lib1) will call FetchContent_MakeAvailable() for you, virtually making it identical to add_subdirectory() call.
You can read more about OVERRIDE_FIND_PACKAGE in CMake documentation.

Related

Use find_package() on external project

I have an External project called messages. I am using ExternalProject_Add in order to fetch and build the project.
If i use find_package(messages REQUIRED) in top level CMakeLists.txt the cmake .. fails because it couldn't find the package installation files, which is logical as they are only build during make command invocation.
I am not sure, if there is way use find_package() on ExternalProjects. If so, please point me to an example.
Thanks
BhanuKiran
You have misunderstood how ExternalProject is supposed to work. You cannot find_package(messages REQUIRED) because it hasn't been built yet. ExternalProject merely creates the build steps necessary to build the subproject.
You have two options:
Use add_subdirectory or FetchContent in place of ExternalProject. In this case, you don't need a find_package call. This effectively adds the sub-project to the main build and imports the subproject's targets.
Use two ExternalProject calls: one for messages and another for main_project, which depends on messages. If messages uses the export(EXPORT) function, you can point CMAKE_PREFIX_PATH or messages_ROOT to the build directory. Otherwise you'll need to run the install step for messages and set up an install prefix inside your build directory. Then the find_project(messages REQUIRED) call inside main_project will succeed. This will likely require re-structuring your build.
Generally speaking, ExternalProject is only useful for defining super-builds, which are chains of CMake builds that depend on one another. And super builds are only useful when you need completely different configure-time options, like different toolchains (eg. you're cross compiling, but need a code generator to run on the build machine). If that's not the case, prefer FetchContent or add_subdirectory with a git submodule.
It is best to use FetchContent with CMake 3.14+ since it adds the FetchContent_MakeAvailable macro that cuts down on boilerplate.
Docs:
https://cmake.org/cmake/help/latest/module/ExternalProject.html
https://cmake.org/cmake/help/latest/module/FetchContent.html
Since I like keeping my CMake file agnostic on how I get my packages.
I was using FetchContent and added this file (Findalib.cmake):
if(NOT alib_POPULATED)
set(alib_BUILD_TESTS OFF CACHE BOOL INTERNAL)
set(alib_BUILD_EXAMPLES OFF CACHE BOOL INTERNAL)
set(alib_BUILD_DOCS OFF CACHE BOOL INTERNAL)
FetchContent_MakeAvailable(alib)
endif()
set(alib_FOUND TRUE)
Then, in my CMake files:
find_package(alib REQUIRED)
target_link_libraries(my-executable PUBLIC alib::alib)
That way, packages are only declared in my file in which I declare dependencies, and I fetch them only if I try to find them.

CMake have a target depend on a generated file in a subdirectory

I have a scenario that I think is very similar to this one: CMake add_custom_command/_target in different directories for cross-compilation, however the solution for that issue isn't working for me.
In subdir/CMakeLists.txt I have:
add_custom_command(OUTPUT foo.h foo.cpp COMMAND ... DEPENDS foo.xml)
add_custom_target(generate_foo DEPENDS foo.h foo.cpp)
and then CMakeLists.txt:
add_executable(MyTarget
subdir/foo.h
subdir/foo.cpp
${OTHER_SOURCES})
add_dependencies(MyTarget generate_foo)
add_subdirectory(subdir)
This fails with "Cannot find source file: subdir/foo.h". The documentation for add_dependencies suggests that it will ensure that generate_foo builds before MyTarget, but if that's the case it looks like it's at least trying to access all source files before either target builds. Am I doing something wrong here? How can I compile source files that are generated by a custom target/command in a subdirectory?
The problem is that the GENERATED file property (that CMake uses to determine if it needs to check that a file exists at configure time) is not visible outside the directory in which the file is generated. The problem goes away in CMake 3.20. This is explained here.
I usually solve this problem by compiling generated source files into a static or object library in the subdirectory, then linking to that, since targets are globally visible. You can also explicitly set the GENERATED property on the generated files in the scope you wish to use them, but this hack breaks the encapsulation gained by using a subdirectory.
It's also worth noting that you can do away with the custom target and the call to add_dependencies because the generated files are already dependencies of the executable (this has always has been the case AFAIK).

Directing cmake to link against shared object with debug postfix (_d)

I've got a cmake project that pretty much looks like this:
cmake_minimum_required(VERSION 3.0)
SET(CMAKE_DEBUG_POSTFIX "_d")
include_directories(../TransfunctionerProject)
include_directories(../TransmogrifierProject)
set(Libraries
ContinuumTransfunctioner
Transmogrifier
)
set(SourceFiles
Wrapper.cpp
Logger.cpp
)
add_library(Frobnigator SHARED ${SourceFiles})
add_library(FrobnigatorStatic STATIC ${SourceFiles})
set_target_properties(FrobnigatorStatic PROPERTIES OUTPUT_NAME Frobnigator)
target_link_libraries(Frobnigator ${Libraries})
Where ContinuumTransfunctioner and Transmogrifier projects include the debug postfix directive SET(CMAKE_DEBUG_POSTFIX "_d") so that libContinuumTransfunctioner_d.so and libTransmogrifier_d.so both exist.
The problem is that the current project appears to be linking against the static library without the _d suffix and complains:
/usr/bin/ld: cannot find -lContinuumTransfunctioner
The Libraries that you pass into the target_link_libraries call are interpreted as filenames, not as target names.
This is the unfortunate fallback for that call in CMake: If you pass a random string to it, that cannot be interpreted in a meaningful way, CMake will always assume it to be plain library name. Sometimes this is just what you want, but the name has to be an exact match for an existing library. The whole debug postfix magic will be lost here.
What you might have wanted to do was to pass a library target name instead. This will trigger a much smarter handling of the dependency and would solve your problem. However, that only works if the library is a known target in the context of the target_link_libraries call. You can easily check this as follows:
if(TARGET ContinuumTransfunctioner)
message("Library target name")
else()
message("Plain library name")
endif()
target_link_libraries(Frobnigator ContinuumTransfunctioner)
So how do you get to the target name case? This depends on how your build is structured. If the library is being built as part of your CMake run, simply make sure that the corresponding add_library call is performed from a subdirectory that is pulled in via add_subdirectory from the file that performs the target_link_libraries call.
If the library in question is an external dependency, you need to build an imported target that carries all the relevant information where to find the library files (including any potential debug postfixes). This can be a bit cumbersome to do manually, so if you can, you might want to use CMake's packaging mechanism to generate this automatically as part of the library's build process.
Here's the solution, courtesy of the good people on the cmake mailing list:
# Note:
# $<$<CONFIG:Debug>:_d> is called a generator expression.
# It outputs _d if the build is debug.
#
set(Libraries
ContinuumTransfunctioner$<$<CONFIG:Debug>:_d>
Transmogrifier$<$<CONFIG:Debug>:_d>
)

Compile gtest from source with catkin

I am trying to compile gtest from source (instead of using the existing installed version). I am working on a catkin based cmake project.
I have added the sourcecode from https://github.com/google/googletest to my workspace and included the folder with add_subdirectory.
However, I get a nameclash with the existing gtest:
CMake Error at src/test_env/GTest/googletest/cmake/internal_utils.cmake:151 (add_library):
add_library cannot create target "gtest" because another target with the
same name already exists. The existing target is a shared library created
in source directory "/usr/src/gtest". See documentation for policy CMP0002
for more details.
From other posts, and the googletest instructions itself (https://github.com/google/googletest/tree/master/googletest#incorporating-into-an-existing-cmake-project) I understand that this should be no problem.
I think the problem might lie in how catkin handles gtest. And, admittedly, normally I could just use the installed version. But I want to make sure, that everyone uses the same (bundled) version of gtest.
Any suggestions and hints are welcomed.
Okay, so the error message is actually quite clear. A cmake "target" is "something that will be produced by the build", be it a library, or an executable, or something else. So, the problem is that you are trying to add a target named "gtest", and catkin already does the same thing. Both would produce the library "libgtest.so", and of course there can only be one of those in the same folder. You could rename "your" gtest by changing the target name in googletest/CMakelists.txt, but I would strongly advise you to not do that.
In my opinion, gtest shouldn't even be a shared library at all, especially if you are using different build flags for different projects in your repository. There is an alternative, and that is basically only including the gtest source code in a folder, and then including the header files and source files in your unittests main.cpp. googletest already comes with helpers for that, that is src/gtest-main.cc.
This is how I would structure it:
Add the gtest version you want as submodule to git (in case you use git). This way, you have a specified version for all projects in your repo, and can update it in a different branch. I will call that folder "GTEST_DIR".
Write your unittests in .cpp files, that #include <gtest/gtest.h>, one per hpp you want to test, and #include both the hpp and the cpp in your test.cpp. This enforces the separation of your tests from other classes and makes it very easy to switch out dependent classes with mocks or fake objects. You will not need a main() function, as that one is already in gtest-main.cc.
Write a cmake macro like this:
macro(add_gtest NAME FILES)
add_executable(my_gtest_$NAME
$FILES
GTEST_DIR/src/gtest.cc
GTEST_DIR/src/gtest-death-test.cc
GTEST_DIR/src/gtest-filepath.cc
GTEST_DIR/src/gtest-port.cc
GTEST_DIR/src/gtest-printers.cc
GTEST_DIR/src/gtest-test-part.cc
GTEST_DIR/src/gtest-typed-test.cc
GTEST_DIR/src/gtest-main.cc
)
target_include_directories(my_gtest_$NAME GTEST_DIR/include)
endmacro()
Of course, you can make this more complicated or less complicated, but that is the gist. Of course, compile times will be longer this way over using gtest as a shared library, but it actually makes sure your units get tested in isolation, which is very valueable in my opinion. Also, you can use ccache to greatly improve compile times in this scenario, because the gtest object files never change. Also, this will make sure gtest is compiled with exactly the flags you want it to. You could for example create 2 separate unit tests for the same class, one with exceptions enabled and one without.

Specifying libraries for cmake to link to from command line

I have a huge project managed with CMake and this project has hundreds of components each of them having own source files and each of them linking to a list of libraries, specified with target_link_libraries(${project} some_libraries, some_other_libraries)
Now, what I am aiming for is that:
Without actually modifying any of the CMakeLists.txt I want ALL the projects's target executable to link to some specific libraries.
Is there a way of achieving this? Since this is a one time trial, I don't want to manually hunt down all the CMakeLists.txt files and modify them (yes, this is the other alternative). Just a note, I compile the entire project from command line, using cmake (no cmake gui).
This is kind of a hack, but for a C++ project, you can use CMAKE_CXX_STANDARD_LIBRARIES. For a C project, I think you would use CMAKE_C_STANDARD_LIRBARIES.
Example for C++ that links to libbar and libfoo:
cmake ... -DCMAKE_CXX_STANDARD_LIBRARIES="-lbar -lfoo"
See the documentation here:
https://cmake.org/cmake/help/v3.6/variable/CMAKE_LANG_STANDARD_LIBRARIES.html
This won't be available for older versions of CMake; it was added some time after version 3.0.
This is a dirty, dirty hack, so please only use it for testing.
You can actually overload the add_executable command by defining a function of the same name. Do this close to the top of the top-level CMakeLists.txt:
function (add_executable name)
message("Added executable: " ${name})
_add_executable(${name} ${ARGN})
target_link_libraries(${name$} your_additional_lib)
endfunction()
Note that _add_executable is an internal CMake name that may break in future CMake versions. As of now (version 3.0) it seems to work with all versions though.
You can overload add_library the same way if required.
For more fine-grained control over what is linked, instead of calling target_link_libraries you can also mess with the LINK_LIBRARIES and INTERFACE_LINK_LIBRARIES target properties directly.