CMake is awesome, especially with lots of modules (FindOOXX). However, when it comes to write a FindXXX module, a library XXX which your project depends, it's not that easy to handle for non-cmake-expert. I sometimes encounter a library without support to CMake, and I like to make one for it. I'm wondering if there is any interactive shell while writing/testing cmake modules?
Are you writing FindXXX for project "XXX" or is "XXX" a dependency of your project that you're trying to find? If the former, you should instead write a file called XXX-config.cmake (or XXXConfig.cmake) and install it into one of the directories mentioned in the docs for find_package. In general, XXX-config.cmake files are for projects which are expected to be found by CMake (and installed by the project itself) and FindXXX.cmake files are for projects which don't support CMake (and usually have to support finding any version of XXX).
That said, for FindXXX.cmake, usually you just need a few find_file (e.g., for headers), some find_library calls, or even just a single pkg_check_module from FindPkgConfig.cmake followed by a find_package_handle_standard_args call (use include(FindPackageHandleStandardArgs) to get it). FPHSA makes writing proper Find modules a breeze.
For XXX-config.cmake files, I have typically used configure_file to generate two versions: one for the install (which includes your install(EXPORT) file) and one for the build tree (generated by export() calls). Using this, other useful variables can be accurately set such as things like "which exact version of Boost was used" or "was Python support compiled in" so that dependent projects can get a better picture of what the dependency looks like.
I have also recently discovered that CMake ships with the CMakePackageConfigHelpers module which is supposed to help with making these files. There looks to be quite a bit of documentation for it.
Related
I'm trying to understand what some of the best practices are when using modern CMake (3.13+) with respect to building and including vendored or submoduled code.
Say I'm building a library MyLib. My file structure is something like this
MyLib
|-CMakeLists.txt
|-src
|-include
|-submodules
|-libgeos
In this example, I've included libgeos as a git submodule, because it's really convenient to be able to clone the project and immediately build and run tests because that dependency is present. This could also be solved by using FetchContent or something, and my question still stands; the important thing is that I do not want to rely on libgeos being installed in build environment.
Note I picked libgeos arbitrarily; I have no idea if libgeos is set up as a cmake project appropriately for this example, but this is all theoretical and I just needed some concrete library name. Please do not use the specific details of how libgeos is configured to answer this, unless libgeos is a good example of conventional cmake.
But now, there's some other project that wants to use my project, and it needs libgeos and doesn't want to depend on my project providing it.
OtherProject
|-CMakeLists.txt
|-src
|-include
|-submodules
|-libgeos
|-MyLib
|submodules
|-libgeos
When you clone OtherProject, you get two versions of libgeos, and maybe that's not great; but it's not a huge issue either. And maybe they're not the same version; say MyLib requires libgeos >= 2.0, so 2.0 is what MyLib includes, and OtherProject requires libgeos>=2.1 so OtherProject includes libgeos >= 2.1.
Now we potentially end up with some build issues. If we have the following line in OtherProject/CMakeLists.txt
add_subdirectory(submodules/libgeos)
and then again, that same line within MyLib/CMakeLists.txt, we end up with cmake errors because libgeos as a target is defined twice in the build. This can be solved a couple of ways.
Check if geos exists before adding it
if(NOT TARGET geos)
add_subdirectory(submodules/libgeos)
endif()
But this case has some issues; if that blob is in OtherProject at the top, it's fine and both projects use libgeos 2.1. But if it's in OtherProject after add_subdirectory(submodules/MyLib), then the geos 2.0 version gets added to the build, which may or may not fail loudly (Hopefully it would).
This could also be solved with find_package. Both projects include cmake/FindGeos.cmake which use that blurb above (if(NOT TARGET...)) to add geos the build and then the top project cmake files can do this
list(APPEND CMAKE_MODULE_PATH cmake)
find_package(geos 2) # (or 2.1)
then it doesn't matter what order they try to include geos, because they will both defer to FindGeos.cmake in OtherProject because it's first in the module path.
But now there's a new issue, some ThirdProject wants to use MyLib also, but ThirdProject wants to depend on libgeos which is in the system environment. It uses find_package(geos 2.1 CONFIG) to use the installed GeosConfig.cmake file, which adds geos::geos to the build and sets geos_FOUND. Suddenly, MyLib fails to build, because geos_FOUND was set, but I'm doing target_link_library(mylib PUBLIC geos).
So this could be solved by adding add_library(geos::geos ALIAS geos) in both custom FindGeos.cmake files, then it doesn't matter if geos was built from source or using the installed version, the target names are the same either way.
Now we get to my actual questions:
Lets start with
Am I crazy, no one does this, and my team is trying to use cmake all wrong?
Is there some feature of cmake that I've just completely missed that solves all these problems?
I suspect there's a good few books or presentations that cover this topic, but I just don't know where to look because there's so many; what should I be looking at? I've seen the CMake Packages page, which looks like it solves the problem when you're using all projects which are configured according to that page; but it doesn't really answer how to bridge the gap between older and newer projects.
If I'm not crazy and there's no straightforward answer or presentation that I can look at, then
What should the cmake configuration for both MyLib and libgeos look like so that these cases work?
MyLib is built alone
MyLib is built as part of a larger project which provides a different version of geos
MyLib is built as part of a larger project which depends on a different version of geos in the environment
I understand that cmake provides helpers that could be used to produce MyLibConfig.cmake if I wanted to install it in the environment. I also see that the export() function exists, which could be used to save those files in the build tree somewhere and then find them with find_package in config mode. But this feels a bit odd to me to do because it's not a multi-stage build, it's just one invocation of cmake then make.
But lets say that's the right answer and the CMake for libgeos doesn't follow it. Would it be appropriate to have FindGeos.cmake do something like this?
if(NOT geos_FOUND)
add_subdirectory(submodules/libgeos)
export(geos NAMESPACE geos)
find_package(geos CONFIG)
endif()
I'm trying to use find_package to include libraries in CMake.
This question talks about how to tell CMake to link to the GMP library (external). I am trying to follow the steps of the answer there but do not have any of the <name>Config.cmake or <name>-config.cmake files, as mentioned by some of the comments, which appears to be the default. The answer does not mention any solution for when you don't know how to get/find these files. The comments to that answer link to an old website (external) with a lot of broken links, that describes a list of Load Modules. It's unclear to me where these modules come from and how to get them.
According to the official CMake documentation (external), if the configuration files are not found, find_package falls back from "Module Mode" to "Config Mode". I don't understand what this means and in what cases this would be relevant, especially since the documentation discourages reading about "Config Mode".
The documentation says that
The file is first searched in the CMAKE_MODULE_PATH, then among the Find Modules provided by the CMake installation.
I am still confused about whether these configuration files are supposed to come with CMake or with the library in question and where they are supposed to be located. Probably both are possible but how does one know in a specific case?
Example code, trying to follow modern best practices:
# CMakeLists.txt (not working)
cmake_minimum_required(VERSION 3.2) # I have no idea what version I actually need
project (GMP_demo_project)
# Enable C++17 standard
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
find_package(GMP REQUIRED)
# Create the executable from sources
add_executable(GMP_demo GMP_demo.cpp)
target_link_libraries(GMP_demo gmp gmpxx)
The code outputs an error message along the lines of
CMake Error at CMakeLists.txt:10 (find_package):
By not providing "FindGMP.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "GMP", but
CMake did not find one.
Could not find a package configuration file provided by "GMP" with any of
the following names:
GMPConfig.cmake
gmp-config.cmake
Add the installation prefix of "GMP" to CMAKE_PREFIX_PATH or set "GMP_DIR"
to a directory containing one of the above files. If "GMP" provides a
separate development package or SDK, be sure it has been installed.
Question: How does one, in general, obtain and organize these configuration files (CMake Load Modules)? How can one expect another user to have these files on his system? My question is intended to be general and only use GMP as an example (although I am in fact interested in being able to use it).
Just as an aside, I can compile, link and execute my demo code just fine using gcc GMP_demo.cpp -lstdc++ -lgmp after having installed GMP as suggested by the library documentation. The problem is just getting CMake to do it. I can also just give CMake the absolute path of the library, which would of course be much easier but not portable (assuming one can get find_package to actually work and be portable with reasonable amounts of work).
How does one, in general, obtain and organize these configuration files (CMake Load Modules)?
Broadly speaking, there are three buckets these fall into:
Files provided directly by the package. This is the ideal solution, and would be what CMake calls Config mode. There would be a file called GMPConfig.cmake which cmake could find by searching preconfigured paths, or by providing a specific path at configuration time (cmake -DGMP_Dir=/path/to/GMP/install/root). The advantages of this approach are that generation of GMPConfig.cmake is mostly automatic, and the libraries can include things like installation paths and compilation flags. The disadvantage is that the library develops have to actually go to the effort of leveraging modern CMake, and not everybody does this.
Files provided directly by CMake. For common packages (e.g., boost) CMake ships FindXXX.cmake files that search well-known paths and take care of this for you. These work identically to the above from an end-user perspective, but which Find modules are available depends on the version of CMake you have installed.
Files provided by some random person that are copy/pasted into projects. How these works depends on the person who wrote it, so you'll have to read their documentation. Use your favorite search engine and try to find FindGMP.cmake, then drop it in a module folder somewhere and update CMAKE_MODULE_PATH appropriately.
How can one expect another user to have these files on his system?
It's your job to install whatever dependencies a package requires. Anything using modern CMake (bullet 1 listed above) should install the XXXConfig.cmake file as part of its installation. If a library is built by something other than CMake, you'd have to either hope for bullet #2, or find/write your own FindXXX.cmake file (bullet #3).
For your specific case, you might be better off with find_library, since your sample compilation line looks like it just needs to link.
I'm attempting to make a CMake package for Crypto++ inclusion in CMake projects, this will end up in the noloader/cryptopp-cmake repo if it gets done.
The ultimate goal is to come up with a working cross-platform FindCryptoPP.cmake file which can be dropped in the Crypto++ source directory to do things like:
find_package(CryptoPP REQUIRED)
target_link_libraries(libbiocoin cryptopp-static)
Or:
find_package(CryptoPP REQUIRED)
target_link_libraries(libbiocoin cryptopp-shared)
In a finished application and have it "just work."
My current best solution within a CMake application is to build Crypto++ for the platform, stick the resulting archive or library in a lib directory, reference that within the CMakeLists.txt and pull it in that way, but of course that requires packaging a binary distribution of the compiled Crypto++ for every platform targeted by the application, which would be nasty to maintain and generally bad even if it weren't crypto code.
It's better to provide a CMake configuration file. find_package will look for a configuration file if no FindFoo.cmake find script is provided. One advantage over a find script is that you won't end with different, maybe conflicting versions of the find script.
See https://cmake.org/cmake/help/latest/manual/cmake-packages.7.html, especially the section Create Layout.
I'm not asking this for just myself. I hope this question will be a reference for the many newbies who like me, found it utterly perplexing about what exactly what was going on behind the scenes when for such a small CMakeLists.txt file
cmake_minimum_required (VERSION 2.6)
project(Tutorial)
add_executable(Tutorial tutorial.cpp)
and such a small tutorial.cpp
int main() { return 0; }
there are so many files generated
CMakeCache.txt cmake_install.cmake Makefile
CMakeLists.txt tutorial.cpp
and a CMakeFiles folder with so many files and folders
CMakeCCompiler.cmake CMakeOutput.log Makefile.cmake
cmake.check_cache CMakeSystem.cmake progress.marks
CMakeCXXCompiler.cmake CMakeTmp TargetDirectories.txt
CMakeDetermineCompilerABI_C.bin CompilerIdC Tutorial.dir
CMakeDetermineCompilerABI_CXX.bin CompilerIdCXX
CMakeDirectoryInformation.cmake Makefile2
Not understanding what was going on behind the scenes (i.e: why so may files had to be generated and what their purpose was), was the biggest obstacle in being able to learn CMake.
If anyone knows, could you please explain it for the sake of posterity? What is the purpose of these files, and when I type cmake ., what exactly is cmake configuring and generating before it builds the project?
The secret is that you don't have to understand what the generated files do.
CMake introduces a lot of complexity into the build system, most of which only pays off if you use it for building complex software projects.
The good news is that CMake does a good job of keeping a lot of this messiness away from you: Use out-of-source builds and you don't even have to look at the generated files. If you didn't do this so far (which I guess is the case, since you wrote cmake .), please check them out before proceeding. Mixing the build and source directory is really painful with CMake and is not how the system is supposed to be used.
In a nutshell: Instead of
cd <source_dir>
cmake .
always use
cd <build_dir_different_from_source_dir>
cmake <source_dir>
I usually use an empty subfolder build inside my source directory as build directory.
To ease your pain, let me give a quick overview of the relevant files which CMake generates:
Project files/Makefiles - What you are actually interested in: The files required to build your project under the selected generator. This can be anything from a Unix Makefile to a Visual Studio solution.
CMakeCache.txt - This is a persistent key/value string storage which is used to cache value between runs. Values stored in here can be paths to library dependencies or whether an optional component is to be built at all. The list of variables is mostly identical to the one you see when running ccmake or cmake-gui. This can be useful to look at from time to time, but I would recommend to use the aforementioned tools for changing any of the values if possible.
Generated files - This can be anything from autogenerated source files to export macros that help you re-integrate your built project with other CMake projects. Most of these are only generated on demand and will not appear in a simple project such as the one from your question.
Anything else is pretty much noise to keep the build system happy. In particular, I never needed to care about anything that is going on inside the CMakeFiles subdirectory.
In general you should not mess with any of the files that CMake generates for you. All problems can be solved from within CMakeLists.txt in one way or the other. As long as the result builds your project as expected, you are probably fine. Do not worry too much about the gory details - as this is what CMake was trying to spare you of in the first place.
As stated on its website:
Cmake is cross-platform, open-source build system for managing the build process of software using a compiler-independent method
In most cases it is used to generate project/make files - in your example it has produced Makefile which are used to build your software (mostly on Linux/Unix platform).
Cmake allows to provide cross platform build files that would generate platform specific project/make files for particular compilation/platform.
For instance you may to try to compile your software on Windows with Visual Studio then with proper syntax in your CMakeLists.txt file you can launch
cmake .
inside your project's directory on Windows platform,Cmake will generate all the necessary project/solution files (.sln etc.).
If you would like to build your software on Linux/Unix platform you would simply go to source directory where you have your CMakeLists.txt file and trigger the same cmake . and it will generate all files necessary for you to build software via simple make or make all.
Here you have some very good presentation about key Cmake functionalities http://www.elpauer.org/stuff/learning_cmake.pdf
EDIT
If you'd like to make platform dependent library includes / variable definitions etc. you can use this syntax in CMakeLists.txt file
IF(WIN32)
...do something...
ELSE(WIN32)
...do something else...
ENDIF(WIN32)
There is also a lot of commands with use of which you are able to prevent the build from failing and in place Cmake will notify you that for instance you do not have boost libraries filesystem and regex installed on your system. To do that you can use the following syntax:
find_package(Boost 1.45.0 COMPONENTS filesystem regex)
Having checked that it will generate the makefiles for appropriate system/IDE/compiler.
Exactly how CMake works is a question for the developers, so this question can't be answered here.
However we can give a touch of useful guidance as far as when you should use CMake and when you therefore need to worry about how it works. I'm not a fan of "oh it just works" answers either - because, especially in software, NOTHING ever "just works" and you ALWAYS have to get into the nitty-gritty details at some point.
CMake is an industrial-strength tool. It automates several VERY complex process and takes into account many variables of which you may not be aware, especially as a fairly new developer, probably working with limited knowledge of all the operating systems and build tools CMake can handle. The reason so many files are generated and why things seem so complex is because all of those other systems are complex and must be accounted for and automated. Additionally there are the issues of "caching" and other time-saving features of the tool To understand everything in CMake would mean understanding everything in these build tools and OS's and all the possible combinations of these variables, which as you can imagine is impossible.
It's important to note that if you're not in charge of managing a large cross-platform build system, and your code base is a few KLOC, maybe up to 100KLOG, using CMake seems a little bit like using a 100,000 dollar forestry tree removal machine to remove weeds from your 2 foot by 2 foot flower garden. (By the way, if you've never seen such a machine, you should look for one on youtube, they're amazing)
If your build system is small and simple it's likely to be better to just write your own makefiles by hand or script them yourself. When your makefiles become unwieldy or you need to build a version of your system on another platform, then you can switch over to CMake. At that point, you'll have lots of problems to solve and you can ask more focused questions about it. In the meantime, check out some of the great books that have been written about CMake, or even better, write one yourself! 8)
I have a specific question which serves as context for a more general question.
There is a scientific package called LAMMPS, and it is usually used as an executable. However, it supports use as a "library". To try to do things right, I put it in /usr/local/lib/lammps. It contains a lammps/src/ directory, which has around 40 source files. Using the instructions provided, I compiled lammps as a .so file in lammps/src/liblammps_serial.so.
I also have separate code in "~/code/ljtube/". This uses cmake to try to find the library. Thus, I wrote a FindLAMMPS.txt so that I could use
FIND_PACKAGE (lammps)
in my CMakeLists. I modified the libtool config file to search in /usr/local/ successfully. I found that it searches in /usr/local/lib/ for a .so file and in /usr/local/include/ for a .h file. So I made a dynamic link to the .so file in /usr/local/lib/, and I copied the .h file from the lammps/src/ to /usr/local/include/.
CMake can now find those two files, but it cannot link to anything else in lammps/src/. It seems absurd to need to make a separate FIND_PACKAGE for each of the .h's I want to include (group.h, fix.h, force.h, pair.h, etc.). It also seems ridiculous to dump the whole package of .h files into the /usr/local/include/ directory. I will be using this code both locally and on a cluster, and possibly distributing it to other group members.
How can I make CMake find what I want to find without hard coding in the location of /usr/local/lib/lammps/src/? Phrased more generically, how should I manage large packages like these to make them easy to link to in the code I write, even if the original developer did not use the best conventions?
(As a side note, I am using a shared library because it seems like the right choice, but I'm not especially married to it. Should I be using a static library? Is there a way for CMake to find an already-compiled library relative to the current source directory, and might that be a better way to implement this? I know that I will be using LAMMPS in multiple projects, so having a local shared copy superficially seems to make the most sense.)
Normally a find_package call yields a variable specifying the path to the "includes" folder of the package. This would then be added in the caller's CMakeLists.txt via include_directories.
For example, to use find_package for boost, you could do:
find_package(Boost) # sets ${Boost_INCLUDE_DIRS} and ${Boost_LIBRARIES}
if(Boost_FOUND)
include_directories(${Boost_INCLUDE_DIRS})
add_executable(foo foo.cc)
target_link_libraries(foo ${Boost_LIBRARIES})
endif()
Regarding your side note, you could use find_library and/or find_path to find the library and its headers given a known location.
Both these commands can be invoked in such a way as to avoid searching in common locations, e.g. by setting PATHS to the known location and using NO_DEFAULT_PATH in the find commands.
Another alternative is for your projects to make use of the ExternalProject_Add function which is described in more detail in this article. From this article:
The ExternalProject_Add function makes it possible to say “download this project from the internet, run its configure step, build it and install it”
A downside to this approach is that each of your projects would end up with its own copy of the third party sources and lib.