this is not a request about header-only packages. Those are straightforward
I've got a cross-platform library which I'd like to not package with any .a (or similar) prebuilt binaries but rather indicate its .cpp must be built along with the consuming application (add_subdirectory style).
The only way I see to do this are:
conan install -build style
Set up conan build profiles
Both of those two are reasonable, yet seem to be more effort than needed for a consumer who "just wants to recompile the C++" with whatever toolchain their top level CMake is pointed towards.
So, in other words, can conan serve similarly as a delivery mechanism for a git retrieve/add_subdirectory and present it as a CONAN_PKG?
Related
Background:
I have a Visual Studio solution(s) with multiple (50+) projects (libraries static/dynamic and final executables). There is internal Visual Studio reference mechanism used to comsume required libraries for particular executables. Of course each project uses external packages, there are "duplicates" like boost, gtest, there are also some "unique" references for only one or few projects.
What's more, libraries are used in other solutions (project sharing) to deploy other executables.
This is my general project structure:
MainDir
|
- DebugDlls (build output)
- Debug64Dlls (build output)
- ReleaseDlls (build output)
- Release64Dlls (build output)
- Libraries
|
- lib1
- lib2
- ...
- Executables
|
- exe1
- exe2
...
I'm about to migrate from NuGet to conan as a dependency manager for external libraries since there are more ready to use conan packages that NuGet one and it's cross-platform. I'd like to do it project by project, dependency by dependency.
One global conan file to rule them all is not an option since each library has to be as standalone as possible so I'm able to simply grab one and use for new executable. What's more it would be impossible to track dependencies of particular library or executable.
My idea is to put a separate conanfile in each project and define dependencies.
Here is the first issue: I need some global/automatic management of common libraries like boost to not mix versions/variants and spare some time on version updates.
this one may be handled by a global file which defines reusable depndencies
is there something ready to use in conan, like template?
Second issue is to copy dlls from dependencies into proper build output so I'm able to execute the binaries.
this one should be fixable also by some global file with proper defines.
Third one is to execute conan install in each project
once again, a hand crafted script will do the job.
I was digging across the conan documentation but it's not very well organized and I was unable to find proper solution in my case. Maybe I missed something?
What would be the best approach here? Is there any build in conan mechanism for that (like CMake add_subdirectory). I would not like to reinvent the well if one already exists :)
I'm about to use conan 1.x
Bret Brown in his talk Modern CMake Modules recommends using Conan (or other package manager) to deliver reusable CMake code.
As instructed by Brett I've created a Conan package that delivers a MyHelpersConfig.cmake CMake file.
(The MyHelpersConfig.cmake file is the content of the package; it is not part of the package build system.)
My Conan package delivers only this one file.
Unfortunately I don't know how to make this line in CMake actually work:
find_package(MyHelpers)
Brett mentions, that when using Conan you need to manually override CMAKE_PREFIX_PATH, but he doesn't go into more detail (link to the relevant portion of his talk: Delivering CMake modules).
Does anyone know what needs to go into the Conan recipe, and how to use the package from CMake, to make it work?
EDIT:
From what I was able to figure out cmake_multi (generator I use when consuming packages) will update CMAKE_PREFIX_PATH, but only if CMAKE_BUILD_TYPE is set (which is rarely the case for multi configuration projects):
if(${CMAKE_BUILD_TYPE} MATCHES "Debug")
set(CMAKE_PREFIX_PATH ${CONAN_CMAKE_MODULE_PATH_DEBUG} ${CMAKE_PREFIX_PATH})
...
We would need to add something like this to CMake (pseudocode):
set(CMAKE_PREFIX_PATH ${CONAN_CMAKE_MODULE_PATH_$<CONFIG>} ${CMAKE_PREFIX_PATH})
But that is impossible.
So my conclusion would be that it should work out of the box for non-multi configuration projects, and can not possibly work for multi configuration projects.
The problem I had was that when consuming a package from CMake Conan was not updating CMAKE_PREFIX_PATH, and therefore MyHelpersConfig.cmake was not found.
This happened when using a cmake_multi generator for the consuming project.
Single-configuration generators should not have this problem, or could be solved easily by adding something like:
set(CMAKE_PREFIX_PATH ${CONAN_CMAKE_MODULE_PATH_<BUILD-MODE-HERE>} ${CMAKE_PREFIX_PATH})
To solve it for multi-config generators you can add the following to CMake in the consuming project:
set(CMAKE_PREFIX_PATH ${CONAN_<YOUR-PACKAGE-NAME>_ROOT_RELEASE} ${CMAKE_PREFIX_PATH})
This will work only under assumption that CMake files you deliver in your Conan package are the same for all build types (Debug, Release...). So it is a viable solution for general-purpose utility functions.
I don't think it is possible solve this situation when CMake files differ between build modes, simply because in multi-config projects build type is known only after all find_package() calls were already evaluated.
I am working on a project that needs some external libraries. Since it is meant to be cross platform, I am using cmake.
What is the preferred way when distributing such projects? Should I supply the external libraries (such as zlib) with their own CMakeLists.txt or should I signal the dependency by simply supplying find_packages()?
the former provides all things needed. while the latter let's the developer decide how to supply the dependency (vcpkg for example)
Althoug there is no universally preferred approach, I absolutely believe you should stick to find_package. Declare your dependencies like this:
find_package(Pkg [version] REQUIRED [components])
Include [version] and [components] only if you know Pkg itself provides first-party CMake package configuration files. If you are writing and distributing a library, you will include equivalent find_dependency calls in your MyProjConfig.cmake file.
If some dependency does not have a standard CMake find module or provide its own CMake package configuration file, you should write your own in ./cmake and add list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake") to the root CMakeLists.txt, before any find_package call. You will install your find modules, too, and include the same addition to the module path in your config files.
Inside the find module, you can use whatever approach you want to create some imported targets for your dependencies. Using PkgConfig is a good approach here.
Going through find_package instantly works with a number of dependency providers: vcpkg, the cmake_paths Conan generator, Linux distro system packages, and so on.
The primary alternative to doing this is to vendor the code, meaning including your dependencies in your build directly, whether through copy/paste into your source tree, a git submodule, or by build-time download from the internet (FetchContent).
The mechanism used to build these is nearly always add_subdirectory in the end, which pulls your dependencies' CMake builds into yours.
Perhaps the biggest issue with this is that most projects' CMake code is totally unprepared to be used in this way. It might trample your cache variables, inject invalid flags into your targets, overwrite your generated headers, and so on. Integration is a nightmare.
Also, from a software distribution standpoint, doing this ties your code to particular versions of your dependencies and takes control away from others who might want to package your code. For instance, Debian packages are not allowed to bundle their dependencies... if libA depends on libB, then each gets its own package. With find_package, it is trivial for a maintainer to inject the appropriate dependencies into your build. Without, it typically involves a difficult-to-maintain patch.
I'm missing something in my understanding of CMake+vcpkg, and I'm also missing proper keywords to search for a solution. (Plus I'm new to both CMake and vcpkg, unfortunately.)
I want to have a public repo for a C++ project that uses CMake as its build system and vcpkg as its package manager.
At my currently level of understanding the user needs to have CMake and vcpkg already installed before he can type cmake and build the repo. I'd like to make it as simple as possible to build the repo and not have a bunch of instructions telling him how to get set up even before he can build.
Is this right?
I'd like a one-step solution: After cloning the repo user types ... something ... and the repo gets built.
I am willing in this day-and-age to assume he's got CMake installed ... plus that it can find the right toolchain. So maybe all he needs to type is 'cmake' ...
Is it a reasonable assumption that the user has CMake installed and configured with his preferred toolchain?
I am not willing to assume he's got vcpkg installed.
Is it a reasonable assumption that the user does not have vcpkg installed and configured?
(TBH, I don't even know if it is CMake or vcpkg that configures the toolchain - I assumed CMake but one of the suggested questions suggests it is vcpkg ...)
What are the reasonable assumptions today, and what is the minimal-step solution?
There's nothing wrong in assuming that the user has certain tools installed.
Let's say you are developing libfoo which depends on libbar and you want to make it as easy as possible for your users to install libfoo.
With a package manager
If libfoo and libbar are available via the same package manager all your users have to do is:
vcpkg install libbar libfoo
You don't have to do anything special in libfoo for this, just instruct the user to install all dependencies in your readme.
It doesn't really matter what package manager is used.
Without a package manager
You will still want to make it easy for people to build and install your project directly. It may seem that invoking a package manager during the build or configuration phase of your project and solving all dependencies is user friendly because the user no longer has to deal with installing those, but it isn't for a number of reasons, including:
you or someone else may want to add your project to another package manager (like conan, spack, etc)
someone may want to consume libfoo with FetchContent, CPM, directly with add_subdirectory, etc
someone may not be a user of vcpkg - there's no need to force them to use it, if possible
you may want to add another dependency, libbaz, which is not available on vcpkg
a user may have the right version of libbar already installed (not necessarily through vcpkg)
This list is not exhaustive. If you're not writing a library some points don't really apply.
This means that someone who has all the dependencies already installed should be able to use libfoo like this:
git clone your-repo
cd your-repo
cmake -Bbuild
cmake --build build
cmake --install build
Resolving dependencies without a package manager
However, it may be desirable to solve dependencies automatically. If your dependencies are using CMake the easiest way of doing this is with FetchContent. For some of the reasons outlined above you should provide an escape hatch so people can still use the already installed dependencies. This can be done with an option. For example, something like FOO_USE_EXTERNAL_BAR. This can be set either to yes or no by default, there's no right answer. As long as the user can control this I don't think it matters that much. You should namespace your options to avoid possible conflicts with options used by other projects.
In this case your build script could do this:
if (FOO_USE_EXTERNAL_BAR)
find_package(bar REQUIRED)
else ()
FetchContent_Declare(
bar
GIT_REPOSITORY bar-repo
GIT_TAG release-tag
)
FetchContent_MakeAvailable(bar)
endif ()
target_link_libraries(foo PRIVATE bar::bar)
Depending on how libbar's CMakeLists.txt is written and organized the if and else branches may get more complicated. See Effective CMake for some details and tips.
Now I can either let libfoo resolve the libbar by setting FOO_USE_EXTERNAL_BAR to ON when I configure your project, or I can set it to OFF to have more control over how it is resolved. I may even use libfoo as a dependency for a project that already depends on libbar. If you always pull it in I can't avoid conflicts in this case.
Using CMake to update dependencies
You may still find it easy for you to be able to update all the project's dependencies using CMake without downloading them via FetchContent. While this will probably raise some eyebrows you could add a custom target for solving dependencies with a package manager. This should also be controllable by an option. Unlike in the above case I strongly believe that if you do this the option should be set to off by default:
if (FOO_AUTO_USE_VCPKG)
add_custom_target(
update_deps
COMMAND vcpkg install libbar
)
add_dependencies(foo update_deps)
endif ()
This will invoke vcpkg every time you build foo so it will make your builds slower. If you remove the add_dependencies call you would have to manually run the update_deps target whenever you need to (which shouldn't be that often anyway).
Notes
Using options is a great way of providing options to your users. It should be noted that they increase the cognitive load, so picking strong defaults can help with that.
FetchContent is a nice way of taking the care away from the user, but at the same time multiple projects that use it will end up re-downloading the same libraries over and over again. It is still more user friendly than invoking a package manager at build time and as long as the users can disable this behavior there's nothing to worry about.
Some parts of this answer may be regarded more as opinion and less as facts. As I said, there is no one right way of doing this, different people will have different ways of solving this problem. Different projects and different environments will have different constraints.
I already recommended the Effective CMake talk above, other useful recourses are available here. If you're a library author you may also want to take a look at Deep CMake for Library Authors.
I had this same question. For my part, I am not willing to assume that the user has either CMake or vcpkg preinstalled.
Here is my solution so far, as a Windows batch file:
#REM Bootstrap...
set VCKPG_PARENT_DIR=C:\Projects
set CMAKE_VERSION="3.20.2"
mkdir "%VCKPG_PARENT_DIR%"
pushd "%VCKPG_PARENT_DIR%"
git clone https://github.com/Microsoft/vcpkg.git
.\vcpkg\bootstrap-vcpkg.bat -disableMetrics
set PATH=%PATH%;%VCKPG_PARENT_DIR%\vcpkg\downloads\tools\cmake-%CMAKE_VERSION%-windows\cmake-%CMAKE_VERSION%-windows-i386\bin
set VCPKG_DEFAULT_TRIPLET=x64-windows
set PYTHONHOME=%VCKPG_PARENT_DIR%\vcpkg\packages\python3_x64-windows\tools\python3
popd
#REM Build the project...
cmake -B build -S .\engine\ -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%\scripts\buildsystems\vcpkg.cmake -DCMAKE_BUILD_TYPE=Release -DUSE_PYTHON_3=ON
cmake --build .\build\ --config Release
mkdir bin
xcopy .\build\Release\*.* .\bin\
xcopy .\build\objconv\Release\*.* .\bin\
xcopy .\build\setup\Release\*.* .\bin\
It could use some improvement, but hopefully this gives you an idea of one route you could take.
A quote from the official documentation:
"Specify rules to run at install time."
What exactly is install time?
The problem for me: I am on Linux, software is installed from packages that are just dependencies and data. There is no CMake that can do anything here. So installation time of software is out of scope from CMake. So what exactly do they mean?
Building a CMake project can roughly be divided into three phases:
Configure time. This includes everything that happens while running cmake itself. This phase is concerned with inspecting certain properties of the host system and generating the specific build files for that platform under the selected configuration.
Build time. This includes everything that happens while actually building your project from the files generated by CMake (like, when running cmake --build or make). This is where all of the actual compilation and linking happens, so at the end of the build phase, you have a usable binary.
Install time. This includes everything that happens when running the INSTALL target generated by CMake (like, when running cmake --build --target install or make install). This takes care of copying the binaries that were generated into the build tree to a different directory. Note that the build tree contains a lot of stuff that is no longer needed once the build is completed if you are only interested in running the binary. Examples include all intermediate build artifacts, like the build files generated during the configure phase or the intermediate object files created during the build phase. Furthermore, the install phase might include additional steps to ensure that the binaries produced during the build are portable. For instance, on Linux systems you might want to remove the build directory from the shared library search path in the binary and replace it with a portable equivalent. So the install phase might do more than just copy all the important files to a new directory. It could also include additional steps that change the binaries to make them more portable.
Note that the last phase is optional. If you do not want to support calling make install but prefer another deployment mechanism, you simply don't use the install command in your CMake script and no INSTALL target will be generated.
I'd like to expand the answer, which ComicSansMS gave you, a little bit.
As he mentioned - CMake generates an extra target called install for the make tool (when you use a Makefile-based generator).
It may look weird for you as a package system is used for Linux. However the install target is still useful or even necessary:
When you develop your application you may need to install (move binaries and possibly some include files) to a certain location so some of your projects may see each other. For example, you may develop a library and a set of non-related applications which use it. Then this library must be installed somewhere to be visible. It doesn't mean you need to put it to the /usr directory; you may use your /home.
The process of Linux package preparation requires an install step. For example, the RPM packaging system does three main steps when the rpm package file is being built: the project is configured, then is compiled and linked and finally is being installed to a certain location. All files from this location are being packed to the rpm file.