I have several projects consisting of a few libraries, each living in its own subdirectory, knitted together by the topmost CMakeLists.txt file. I am in the habit of using project(<DIRNAME>) at the top of each CMakeLists.txt file and I try to structure the subprojects in such a way that they could be compiled separately from the top project. However, while this might make sense for standalone, core libraries, it cannot work for the libraries that depend on them because I need to do stuff like
target_link_libraries(gui core)
And core will nor be defined if I am trying to compile gui as a standalone project.
Is it wrong to use project() in this context, or am I missing something?
A Matter of Taste
This is in my opinion mainly a matter of taste. I don't think multiple project() commands itself are a problem, its more that projects I have seen using this approach tend to repeat itself in other parts and sometimes are running into problems with global cached variables.
Depending Libraries
The more relevant fact is, that the depending libraries will also add an include dependencies.
For standalone static library targets - not executable or shared library targets who really link the library - the simple target_link_libraries() command could be ignored with something like:
if (TARGET core)
target_link_libraries(gui core)
endif()
But the header files include dependency remains.
Standalone Projects in CMake
For me a (sub-)project to be really standalone needs not only the project() command, but it should also have a export(TARGETS ...) command. Then you could e.g. use find_package() commands to resolve any open dependencies with something like:
if (NOT TARGET core)
find_package(core REQUIRED)
endif()
target_link_libraries(gui core)
References
Making cmake library accessible by other cmake packages automatically
CMake share library with multiple executables
Related
I'm trying to understand what some of the best practices are when using modern CMake (3.13+) with respect to building and including vendored or submoduled code.
Say I'm building a library MyLib. My file structure is something like this
MyLib
|-CMakeLists.txt
|-src
|-include
|-submodules
|-libgeos
In this example, I've included libgeos as a git submodule, because it's really convenient to be able to clone the project and immediately build and run tests because that dependency is present. This could also be solved by using FetchContent or something, and my question still stands; the important thing is that I do not want to rely on libgeos being installed in build environment.
Note I picked libgeos arbitrarily; I have no idea if libgeos is set up as a cmake project appropriately for this example, but this is all theoretical and I just needed some concrete library name. Please do not use the specific details of how libgeos is configured to answer this, unless libgeos is a good example of conventional cmake.
But now, there's some other project that wants to use my project, and it needs libgeos and doesn't want to depend on my project providing it.
OtherProject
|-CMakeLists.txt
|-src
|-include
|-submodules
|-libgeos
|-MyLib
|submodules
|-libgeos
When you clone OtherProject, you get two versions of libgeos, and maybe that's not great; but it's not a huge issue either. And maybe they're not the same version; say MyLib requires libgeos >= 2.0, so 2.0 is what MyLib includes, and OtherProject requires libgeos>=2.1 so OtherProject includes libgeos >= 2.1.
Now we potentially end up with some build issues. If we have the following line in OtherProject/CMakeLists.txt
add_subdirectory(submodules/libgeos)
and then again, that same line within MyLib/CMakeLists.txt, we end up with cmake errors because libgeos as a target is defined twice in the build. This can be solved a couple of ways.
Check if geos exists before adding it
if(NOT TARGET geos)
add_subdirectory(submodules/libgeos)
endif()
But this case has some issues; if that blob is in OtherProject at the top, it's fine and both projects use libgeos 2.1. But if it's in OtherProject after add_subdirectory(submodules/MyLib), then the geos 2.0 version gets added to the build, which may or may not fail loudly (Hopefully it would).
This could also be solved with find_package. Both projects include cmake/FindGeos.cmake which use that blurb above (if(NOT TARGET...)) to add geos the build and then the top project cmake files can do this
list(APPEND CMAKE_MODULE_PATH cmake)
find_package(geos 2) # (or 2.1)
then it doesn't matter what order they try to include geos, because they will both defer to FindGeos.cmake in OtherProject because it's first in the module path.
But now there's a new issue, some ThirdProject wants to use MyLib also, but ThirdProject wants to depend on libgeos which is in the system environment. It uses find_package(geos 2.1 CONFIG) to use the installed GeosConfig.cmake file, which adds geos::geos to the build and sets geos_FOUND. Suddenly, MyLib fails to build, because geos_FOUND was set, but I'm doing target_link_library(mylib PUBLIC geos).
So this could be solved by adding add_library(geos::geos ALIAS geos) in both custom FindGeos.cmake files, then it doesn't matter if geos was built from source or using the installed version, the target names are the same either way.
Now we get to my actual questions:
Lets start with
Am I crazy, no one does this, and my team is trying to use cmake all wrong?
Is there some feature of cmake that I've just completely missed that solves all these problems?
I suspect there's a good few books or presentations that cover this topic, but I just don't know where to look because there's so many; what should I be looking at? I've seen the CMake Packages page, which looks like it solves the problem when you're using all projects which are configured according to that page; but it doesn't really answer how to bridge the gap between older and newer projects.
If I'm not crazy and there's no straightforward answer or presentation that I can look at, then
What should the cmake configuration for both MyLib and libgeos look like so that these cases work?
MyLib is built alone
MyLib is built as part of a larger project which provides a different version of geos
MyLib is built as part of a larger project which depends on a different version of geos in the environment
I understand that cmake provides helpers that could be used to produce MyLibConfig.cmake if I wanted to install it in the environment. I also see that the export() function exists, which could be used to save those files in the build tree somewhere and then find them with find_package in config mode. But this feels a bit odd to me to do because it's not a multi-stage build, it's just one invocation of cmake then make.
But lets say that's the right answer and the CMake for libgeos doesn't follow it. Would it be appropriate to have FindGeos.cmake do something like this?
if(NOT geos_FOUND)
add_subdirectory(submodules/libgeos)
export(geos NAMESPACE geos)
find_package(geos CONFIG)
endif()
Existing solutions suggest using add_custom_command, and that works, but I consider it ugly and error prone, so I wonder if there is a way when I declare my 3rdparty library to say that any executable using my library should have dll copied to where executable is built. Obviously support for different dlls in Release/Debug is also necessary.
You are clearly on Windows since you are building dll's. But on Linux I would just comment that one can set the RPATH for a binary. In Cmake this looks something like
set_target_properties(target PROPERTIES INSTALL_RPATH ${path_to_libs})
# Maybe also this line...
target_link_libraries(target PUBLIC -Wl,--disable-new-dtags)
I have no idea what effect this has in Windows. However, in Linux this basically tells the binary where to look for libraries that it will need to link to, thus avoiding the need to either copy the libraries to the same directory or to copy the libraries to the standard library path.
If you have a target A in Cmake, and you either export that target, or include it in another Cmake project as a subdirectory (via add_subdirectory), then you could easily just bundle all of the shared libraries together into the target like this:
CMakeLists.txt for target A:
add_library(A ... )
target_link_libraries(A PUBLIC other_libs_needed_by_consumers_of_A)
CMakeLists.txt for target B:
target_link_libraries(B PUBLIC A)
My project contains several executables that share some common code. I would like to put the common code in a static library that the executables can link to. (The common code is pretty small and I prefer not to deal with shared libraries).
The source tree looks something like this:
project
CMakeLists.txt
common
CMakeLists.txt
src
include
app1
src
CMakeLists.txt
app2
src
CMakeLists.txt
app1 and app2 both depend on the code in common.
This common code is very application specific and will never need to be used by another project outside this directory tree. For that reason I would prefer not to install the library in any sort of global location.
The top-level CMakeLists.txt file just adds the subdirectories:
project(toplevel)
cmake_minimum_required(VERSION 3.1)
add_subdirectory(common)
add_subdirectory(app1)
add_subdirectory(app2)
The common library's CMakeLists.txt file creates the static library and sets include directories:
add_library(common STATIC common.cpp)
target_include_directories(common PUBLIC "${CMAKE_CURRENT_LIST_DIR}/include")
And the file for the executables looks like this:
project(app1)
cmake_minimum_required(VERSION 3.1)
add_executable(${PROJECT_NAME} main.cpp)
target_link_libraries(${PROJECT_NAME} common)
Now for my question. If I run CMake from the top level project directory, I can build app1 and app2 and they build successfully. However, if I want to build a single one of these projects (by running CMake from app1, for example) instead of building from the top level directory, I get an error because common/include is not added to the header search path.
I can see why this happens. There is nothing in the CMakeLists.txt file for app1 or app2 that "pulls in" common. This is only done at the top level.
Is there a way around this, or is this behavior generally considered acceptable? Is something about my setup sub-optimal? I'm just thinking it would be nice to be able to build the projects individually instead of from the top level in the event that we start to develop more and more executables that use this common library, but perhaps this is something I shouldn't be concerned about.
When you setup your build environment, you should put some thought into the following three topics (beside others, but for this discussion/answer I reduced it to the three I feel are relevant here):
Dependecies / Coupling
Deployment
Teams
"Strong Coupling"
I came to think of the add_subdirectory() command as supporting "strong coupling" and your current setup implicitly supports:
A frequently changing common library
A single deployment (process and timing) for all your apps
A single team working on the complete source base
An IDE would show it all in one solution
You generate one build environment for everything
"Loose Coupling"
If you want a more "loose coupling" you could make use of external scripts in other languages or the use of CMake's ExternalProject_Add() macro. So if you setup the common library (maybe even including "binary delivery") and each app as a separate project you do support:
A less often changing common library
Probably with its own release cycles
An independent development/deployment cycle for each app
A team of different developers working on each app
A Mixture of Both
So as you can see there a lot of things to consider and CMake can give you support for all kind of approaches. Considering that your project might be in the early stages, you probably go by a mixed approach (not right away de-coupling the common library):
CMakeLists.txt
project(toplevel)
cmake_minimum_required(VERSION 3.1)
include(ExternalProject)
ExternalProject_Add(
app1
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/app1"
PREFIX app1
INSTALL_COMMAND ""
)
ExternalProject_Add(
app2
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/app2"
PREFIX app2
INSTALL_COMMAND ""
)
app1/CMakeLists.txt
project(app1)
cmake_minimum_required(VERSION 3.1)
add_subdirectory(../common common)
add_executable(${PROJECT_NAME} src/main.cpp)
target_link_libraries(${PROJECT_NAME} common)
This will actually generate three build environments. One directly in your binary output directory and one each in app1 and app2 sub-directories.
And in such approaches you may want to think about common CMake toolchain files.
References
Use CMake-enabled libraries in your CMake project (II)
CMake: How to setup Source, Library and CMakeLists.txt dependencies?
You should use project() command in subdirectories only if this subproject is intended to be built both as standalone and as a part of toplevel project. This is the case for LLVM and Clang, for example: Clang can be compiled separately, but when LLVM build system detects Clang source, it includes its targets too.
In your case you don't need subprojects. To compile only app1 or app2 target issue make app1/make app2 in projects build dir.
I have a huge project managed with CMake and this project has hundreds of components each of them having own source files and each of them linking to a list of libraries, specified with target_link_libraries(${project} some_libraries, some_other_libraries)
Now, what I am aiming for is that:
Without actually modifying any of the CMakeLists.txt I want ALL the projects's target executable to link to some specific libraries.
Is there a way of achieving this? Since this is a one time trial, I don't want to manually hunt down all the CMakeLists.txt files and modify them (yes, this is the other alternative). Just a note, I compile the entire project from command line, using cmake (no cmake gui).
This is kind of a hack, but for a C++ project, you can use CMAKE_CXX_STANDARD_LIBRARIES. For a C project, I think you would use CMAKE_C_STANDARD_LIRBARIES.
Example for C++ that links to libbar and libfoo:
cmake ... -DCMAKE_CXX_STANDARD_LIBRARIES="-lbar -lfoo"
See the documentation here:
https://cmake.org/cmake/help/v3.6/variable/CMAKE_LANG_STANDARD_LIBRARIES.html
This won't be available for older versions of CMake; it was added some time after version 3.0.
This is a dirty, dirty hack, so please only use it for testing.
You can actually overload the add_executable command by defining a function of the same name. Do this close to the top of the top-level CMakeLists.txt:
function (add_executable name)
message("Added executable: " ${name})
_add_executable(${name} ${ARGN})
target_link_libraries(${name$} your_additional_lib)
endfunction()
Note that _add_executable is an internal CMake name that may break in future CMake versions. As of now (version 3.0) it seems to work with all versions though.
You can overload add_library the same way if required.
For more fine-grained control over what is linked, instead of calling target_link_libraries you can also mess with the LINK_LIBRARIES and INTERFACE_LINK_LIBRARIES target properties directly.
I have the following directory layout:
main_folder
+ static_lib1
+ executable
Both 'static_lib1' and 'executable' have a full CMakeLists so that they can be
built independently.
The 'executable' depends on 'static_lib1'. It uses find_package() to locate 'static_lib1'.
The main_folder contains a CMakeLists that includes both 'static_lib1' and 'executable' via add_subdirectory for conveniently building the whole project in one go.
Everything works fine if I manually build 'static_lib1' and then 'executable'. But when running the CMakeLists from the main folder, I get an error because find_package is unable to find the library files from 'static_lib1' which have not yet been built.
How can I resolve this while keeping the CMakeLists files separate (i.e. without including the static_lib's CMakeLists from the executable's CMakeLists)?
In executable's CMakeLists.txt you can check if you are building stand-alone or as part of project:
if( CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR )
# stand-alone build
find_package(static_lib1)
else()
include_directories(../static_lib1)
link_directories(../static_lib1)
...
target_link_libraries(executable static_lib1)
endif()
Switch from a file-based approach to a target-based approach for handling the dependency from executable to static_lib1.
The original problem occurred because executable called find_package for locating static_lib1, which then attempted to fill a variable like STATIC_LIB1_LIBRARY with the paths to the library files by calling find_library. executable then consumes the content of that variable in a target_link_libraries(executable ${STATIC_LIB1_LIBRARY}) call. The problem here is, since those library files only get generated as part of the build, that call to find_library will not be able to find anything.
Building executable needs to support two scenarios here:
Building standalone, where a pre-compiled version of static_lib1 is located somewhere on the disc.
Building from main_folder, where both executable and static_lib1 are part of the same build.
The approach from the question supports scenario 1, but not scenario 2.
Instead of using using a variable to communicate a dependency between the two builds, use a target. The CMakeLists.txt for static_lib1 likely creates a library target like add_library(static_lib1 [...]). In executable we now simply do target_link_libraries(executable PUBLIC static_lib1). This is sufficient to support scenario 2.
To also allow for scenario 1 at the same time, we look at the call to find_package(static_lib1) in the CMakeLists.txt for executable. Instead of providing a variable like before, this call now needs to provide a target static_lib1 for consumption.
So we adapt the find script for static_lib1 to the following behavior:
If a target static_lib1 already exists, there's nothing to be done and the find script can just return (this is scenario 2).
Otherwise, we call find_library to locate the library file on disc (as before in the original approach) and then create a new imported target: add_library(static_lib1 STATIC IMPORTED). We then configure all relevant properties of the static library to that target. For instance, to add the location of the library file, we could do
set_target_properties(static_lib1 PROPERTIES
IMPORTED_LINK_INTERFACE_LANGUAGES "CXX"
IMPORTED_LOCATION ${STATIC_LIB1_LIBRARY}
)
To support multi-config generators like MSVC, instead of setting IMPORTED_LOCATION and IMPORTED_LINK_INTERFACE_LANGUAGES, you will want to set the configuration specific properties like IMPORTED_LOCATION_DEBUG and IMPORTED_LOCATION_RELEASE instead. Since this can get quite tedious to do manually, you can have CMake generate this information (and a bunch of other convenient stuff) for you in a package script. The find mechanism for package scripts works slightly different under the hood, but the code in the CMakeLists.txt for executable will look just the same, a simple call to find_package(static_lib1). The main difference is that this call will then not dispatch to a hand-written find script, but to a package script that was automatically generated by CMake as part of the build process of static_lib1.
I guess I will leave this answer for posterity since only recently I have searched for a solution to this problem and found out that...
Since CMake 3.24 it is possible!
It is possible to override subsequent calls to find_package() with FetchContent_Declare() flag OVERRIDE_FIND_PACKAGE.
Your
add_subdirectory("path/to/static_lib1")
call has to be replaced in main_folder/CMakeLists.txt with:
include(FetchContent)
FetchContent_Declare(
static_lib1
SOURCE_DIR "path/to/static_lib1"
OVERRIDE_FIND_PACKAGE
)
Any calls to find_package(static_lib1) will call FetchContent_MakeAvailable() for you, virtually making it identical to add_subdirectory() call.
You can read more about OVERRIDE_FIND_PACKAGE in CMake documentation.