Python library and CMake target with the same name - cmake

I'm constructing a library "mylib" that is C++ header-only and has a Python API using pybind11.
I want to use "mylib" both as CMake target, containing compile instructions, and as name of the Python API. However, this leads to a name conflict.
Problem description
Consider the following file structure:
CMakeLists.txt
include/mylib.hpp
python_api.cpp
In reality there are also tests and examples, each with their own CMakeLists.txt, but for the purpose of this example the only thing that matters is:
In the (main) CMakeLists.txt I am defining a CMake target "mylib" that has the include path to the header(s), but also 'links' the targets of dependencies. So that the user (or tests, examples, or build of the Python API) only has to 'link' the target and be good to go. (Finally, I'm also installing the target in mylibTargets.cmake when I install the headers such that there is CMake support for the end user).
Now the problem: My Python package should have the same name, "mylib". However, if I call pybind11_add_module with "mylib", CMake complains that
CMake Error at .../share/cmake/pybind11/pybind11Tools.cmake:166 (add_library):
add_library cannot create target "mylib" because another target with the
same name already exists. The existing target is an interface library
created in source directory "..".
See documentation for policy CMP0002 for more details.
It has the right to complain. At the same time I cannot use a different name for either the CMake target (since I want to install and use it using the only logical name, "mylib") or the pybind11 target (since it has to encode "mylib").
So: how do I solve this?
(The only solution I found was to rename one of targets, but as described I don't want to do this)
Detailed example
Consider the simplified, single, CMakeLists.txt:
cmake_minimum_required(VERSION 3.1..3.19)
# configure target
project(mylib)
find_package(xtensor REQUIRED)
add_library(${PROJECT_NAME} INTERFACE)
target_include_directories(${PROJECT_NAME} INTERFACE
$<INSTALL_INTERFACE:include>
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>)
target_link_libraries(${PROJECT_NAME} INTERFACE xtensor)
# installation of headers and of CMake target
include(CMakePackageConfigHelpers)
include(GNUInstallDirs)
install(DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/include/" DESTINATION include)
install(TARGETS ${PROJECT_NAME} EXPORT ${PROJECT_NAME}-targets)
install(
EXPORT ${PROJECT_NAME}-targets
FILE "${PROJECT_NAME}Targets.cmake"
DESTINATION "${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME}")
# Build Python module
find_package(pybind11 CONFIG REQUIRED)
pybind11_add_module(${PROJECT_NAME} python_api.cpp) # <- target name conflict
target_link_libraries(example PUBLIC pybind11::module)
Too limited work around
I could entirely split building (and later install) the Python API to an independent CMakeLists.txt. However, I want to use the target "mylib", that I already equipped with everything it needs, to build the Python API. Since I want to do this without being forced to install the library forced, I don't know how to do this in a 'single' CMakeLists.txt

pybind11_add_module is just a wrapper around add_library, this is explicitely written in the documentation for that function. So, most of the "tricks", which works for the common libraries, works for python modules too.
That is, if you want resulted file to be named as mylib.so but cannot afford you to use mylib as a target name, then you could use any other name for the target but adjust OUTPUT_NAME property for that target. For example:
# Python library target has suffix '_python'
pybind11_add_module(mylib_python ...)
# But name of the library file doesn't have this suffix
set_target_properties(mylib_python PROPERTIES OUTPUT_NAME mylib)

Related

Cmake and vcpkg - find the correct library name

I have troubles finding out the right "library target name" to be used in a cmake file, for packages installed using vcpkg.
In example, I installed the gtest package using vcpkg install gtest. My sample cmake file looks like:
#CMakeLists.txt
cmake_minimum_required(VERSION 3.0)
project(example)
add_executable(main main.cpp)
find_package(gtest REQUIRED)
target_link_libraries(main gtest) # here, "gtest" is not the right name!
Running cmake, a solution for Visual Studio is generated, but after running cmake --build ., I get the error:
../use-cmake-vcpkg\main.cpp(1): fatal error C1083: Cannot open include file: 'gtest/gtest.h': No such file or directory ..
Turns out the line: target_link_libraries(main gtest) isn't correct, and I need to use another "name" to include/link the gtest package.
Is there a way (using cmake or vcpkg) to find out what is the correct target name to be used? (for gtest in this case, but also for any other pacakage?)
When use find_package(XXX), it can work in two modes: MODULE and CONFIG. And resulted variables and targets of this call depend on the mode.
If FindXXX.cmake file exists (and can be found), the MODULE mode is used and given file is processed. Otherwise, if the package is shipped with XXXConfig.cmake file, CONFIG mode is used and given file is processed. If none of this file exists, CMake emits an error (if called with REQUIRED keyword) or a warning (without REQUIRED keyword).
In case of gtest package, CMake is shipped with FindXXX.cmake script, so this script is processed in MODULE mode. You may find description of this script in the documentation, which tells that you need to use GTest::GTest target for link with gtest:
target_link_libraries(main GTest::GTest)
Not all packages provide a CMake library definition. If you're lucky, then vcpkg install will show you the name:
$ ./vcpkg install openssl
The package openssl is compatible with built-in CMake targets:
find_package(OpenSSL REQUIRED)
target_link_libraries(main PRIVATE OpenSSL::SSL OpenSSL::Crypto)
This will work even if you've already installed the package, so you can use it anytime to query the package name.
On the other hand, if vcpkg install <pkg> doesn't say anything about CMake, then you need to include it manually in your CMake file, by finding the include path and the library files.
Here is an example of such a case, here for the live555 library:
# Use one of the headers to locate the include location
find_path(LIVE555_INCLUDE_DIR liveMedia.hh)
# Find the libraries
find_library(LIVE555_LIBRARY1 liveMedia)
find_library(LIVE555_LIBRARY2 groupsock)
find_library(LIVE555_LIBRARY3 BasicUsageEnvironment)
find_library(LIVE555_LIBRARY4 UsageEnvironment)
add_executable(rtsp testRTSPClient.cpp)
target_include_directories(rtsp PRIVATE ${LIVE555_INCLUDE_DIR})
target_link_libraries(rtsp PRIVATE ${LIVE555_LIBRARY1} ${LIVE555_LIBRARY2} ${LIVE555_LIBRARY3} ${LIVE555_LIBRARY4})

With 'find_package': how do I know which variables have to be used for linking?

I use find_package() to find some library. How do I know which variables it defines, so I could use them for linking with that library?
For example, the following CMakeLists.txt successfully is run by CMake, but then make fails at the linking stage, because ${DOCOPT_LIBRARIES} was empty.
cmake_minimum_required(VERSION 3.1)
project(test)
set(PROJECT_LIBS "")
find_package(docopt REQUIRED)
include_directories(${DOCOPT_INCLUDE_DIRS})
set(PROJECT_LIBS ${PROJECT_LIBS} ${DOCOPT_LIBRARIES})
add_executable(${PROJECT_NAME} main.cpp)
target_link_libraries(${PROJECT_NAME} ${PROJECT_LIBS})
But I have no idea what else to try?
The usage of results of find_package(XXX) depends on the script which is actually used by this command:
FindXXX.cmake.
This file is either shipped with the project which wants to use it, or with CMake. Usually, such "Find" scripts start with some description about their usage.
XXXConfig.cmake or xxx-config.cmake.
"Config" scripts are usually installed with the package. Normally, they are generated by CMake using specific CMake commands. You need to consult the package's documentation for use such scripts properly.
If the package doesn't document the usage of its "Config" script, then you may only guess about its usage.
Reading a "Config" script itself could be difficult: CMake includes into it a lot of "sugar", which helps portability and compatibility.
Normally, a "Config" script provides some IMPORTED library target(s), which can be used with target_link_libraries call.
One may scan the project's CMakeLists.txt for a line like
install(TARGETS <targets...> EXPORT ...)
Here <targets...> denotes targets, which can be accessed after find_package() call.
Also scan for a line like
install(EXPORT ...)
if it has NAMESPACE option, then its value prepends all exported targets.
Example:
For docopt package, its CMakeLists.txt contains a line
install(TARGETS docopt EXPORT ${export_name} DESTINATION ${CMAKE_INSTALL_LIBDIR})
thus its docopt target is exported.
As installation of the export file
install(EXPORT ${export_name} DESTINATION "${CMAKE_INSTALL_LIBDIR}/cmake/docopt")
doesn't contain NAMESPACE option, the export target is accessible with its original name:
find_package(docopt REQUIRED)
...
add_executable(${PROJECT_NAME} main.cpp)
# Use *IMPORTED* 'docopt' target
target_link_libraries(${PROJECT_NAME} docopt)

Library depend on a header file

Let's assume I have a project with a series of libraries. I also need to generate a header file that will be used by all of these projects. So I created a CMake file, like this:
project(main)
add_subdirectory(sub_1)
add_subdirectory(sub_2)
# ...
add_subdirectory(sub_n)
add_custom_command(
OUTPUT CustomHeader.h
COMMENT "Generating custom header for all the libraries"
COMMAND ...)
add_library(${PROJECT_NAME} STATIC ${OBJECT_LIST})
The problem is, that I don't know how to tell CMake to run my custom command (that generates this CustomHeader.h) before it would try to build the libraries in the subfolders.
I tried add_custom_target(TARGET MyPrebuild PRE_BUILD ...) but I'm running on Linux, and this option only works on Windows platform according to the documentation.
add_dependencies only work between targets, and not a target and a single file.
I could, in theory, add the header to be among the source files of the individual libraries (in the sub_1, .., sub_n folders) but it feels wrong, as the header is not required to be part of those libraries.
So I just have no idea how I can make a library depend on an include file, that is not part of it.
Any tips how I can overcome this problem?
For make header file (re)built before a library in subdirectory is compiled, you may create target, which builds the file, and make the library dependent from the target:
# *CMakeLists.txt*
# ...
add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h ...)
add_custom_target(generate_custom_header DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h)
# *sub/CMakeLists.txt*
# ...
add_library(libA ...)
add_dependencies(libA generate_custom_header)
Instead of using add_dependencies, you may create header-only library which "implements" you header and link with it:
# *CMakeLists.txt*
# ...
add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h ...)
add_custom_target(generate_custom_header DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h)
add_library(libCustom INTERFACE) # Header only library
add_dependencies(libCustom generate_custom_header) # which depends on generated header
# You may even assign include directories for the header-only library
target_include_directories(libCustom INTERFACE ${CMAKE_CURRENT_BINARY_DIR})
# *sub/CMakeLists.txt*
# ...
add_library(libA ...)
target_link_libraries(libA libCustom) # Common linking with a header-only library.
Note, that INTERFACE library is a "fake" - it is never created by itself. Instead, all "features" of INTERFACE library are just propagated to its users.
I would suggest to add another library target that will both keep track of the generated headers and will help to properly configure other libraries to know where to find them (i.e. target_include_directories).
cmake_minimum_required(VERSION 3.0)
project(testable)
set(CustomHeaderInPath ${CMAKE_CURRENT_SOURCE_DIR}/CustomHeader.example)
set(CustomHeaderPath ${CMAKE_CURRENT_BINARY_DIR}/CustomHeader.h)
add_custom_command(
OUTPUT ${CustomHeaderPath}
COMMAND cp ${CustomHeaderInPath} ${CustomHeaderPath}
COMMENT "Generated file"
DEPENDS ${CustomHeaderInPath})
add_library(CustomHeaderLibrary ${CustomHeaderPath})
target_include_directories(CustomHeaderLibrary PUBLIC ${CMAKE_CURRENT_BINARY_DIR})
set_target_properties(CustomHeaderLibrary PROPERTIES LINKER_LANGUAGE C)
add_library(LibA a.c)
target_link_libraries(LibA CustomHeaderLibrary)
add_library(LibB b.c)
target_link_libraries(LibB CustomHeaderLibrary)
add_executable(${PROJECT_NAME} main.c)
target_link_libraries(${PROJECT_NAME} PUBLIC LibA LibB)
Note that I had to explicitly set the LINKER_LANGUAGE of the new target as cmake won't be able to deduce it properly if no c or cpp files are added to the library.

Making a CMake library accessible by other CMake packages automatically

I have one project that produces a library:
project (myCoolLibrary)
ADD_LIBRARY(my_cool_library SHARED ${mysources_SRC})
And another project that should be using this library:
find_package (myCoolLibrary REQUIRED)
INCLUDE_DIRECTORIES("${myCoolLibrary_INCLUDE_DIRS}" )
add_executable(myCoolExe ${my_sources_SRC} )
TARGET_LINK_LIBRARIES(myCoolExe ${myCoolLibrary_LIBRARIES} )
Is there a way that I can change the first file so that the second file works automatically? That by running CMake on the first file and then running make on the output, then running CMake on the second file, CMake is able to find the package?
An answer where I just give the address of where the first project is built to the second package is also acceptable.
Taking the code found in a blog post by #daniperez - Use CMake-enabled libraries in your CMake project (III) - I've come up with the following minimal solution:
myCoolLibrary/CMakeLists.txt
cmake_minimum_required(VERSION 3.3)
project(myCoolLibrary)
function(my_export_target _target _include_dir)
file(
WRITE "${CMAKE_CURRENT_BINARY_DIR}/${_target}Config.cmake"
"
include(\"\$\{CMAKE_CURRENT_LIST_DIR\}/${_target}Targets.cmake\")
set_property(
TARGET ${_target}
APPEND PROPERTY
INTERFACE_INCLUDE_DIRECTORIES \"${_include_dir}\"
)
"
)
export(TARGETS ${_target} FILE "${CMAKE_CURRENT_BINARY_DIR}/${_target}Targets.cmake")
# NOTE: The following call can pollute your PC's CMake package registry
# See comments/alternatives below
export(PACKAGE ${_target})
endfunction(my_export_target)
...
add_library(${PROJECT_NAME} SHARED ${mysources_SRC})
my_export_target(${PROJECT_NAME} "${CMAKE_CURRENT_SOURCE_DIR}")
myCoolExe/CMakeLists.txt
cmake_minimum_required(VERSION 3.3)
project(myCoolExe)
find_package(myCoolLibrary REQUIRED)
...
add_executable(${PROJECT_NAME} ${my_sources_SRC})
target_link_libraries(${PROJECT_NAME} myCoolLibrary)
To make it reusable I have packed everything into my_export_target(). And I'm friend of self-propagating properties like INTERFACE_INCLUDE_DIRECTORIES.
As commented by #ruslo, using export(PACKAGE ...) can pollute your package registry. So alternatively you can:
Write the target configuration files directly to some dedicated place specific for a certain toolchain
See e.g. How to install your custom CMake-Find module and 0003659: FIND_PACKAGE command improvements.
Set CMAKE_MODULE_PATH via the second project's CMake command line (injecting the search path(s) from the outside). If you are building the two projects anyway with a build script, then this is the most direct way to propagate the module search path(s).
Additional References
export()
CMake/Tutorials/Package Registry
Unable to find Eigen3 with CMake
How to instruct CMake to use the build architecture compiler

How to use CMake to find and link to a library using install-export and find_package?

You have a CMake-enabled library project. You need to use it in another library or executable. How to use CMake to find and link to the library? You may have the following preferences:
write the least possible amount of boiler-plate code
decouple the internal details of the linked library from the consuming target
Ideally, the usage of the library should look like this:
add_executable(myexe ...)
target_link_libraries(myexe mylib::mylib)
Let me demonstrate a possible solution on a concrete example:
The myapp project
We have an executable target myapp. We're linking it with mylib, which is built in its own build tree. In the CMakeLists.txt of myapp we find and specify mylib as a dependency of myexe:
find_package(mylib REQUIRED)
...
add_executable(myexe ...)
target_link_libraries(myexe mylib::mylib)
Let's see how to set up mylib and the build of myexe to make this work.
The mylib project
The directory layout of mylib:
mylib
- CMakeLists.txt
- mylib.c
+ include
- mylib.h # single public header
In the CMakeLists.txt of mylib we need to create the target and specify its source files:
add_library(mylib mylib.c include/mylib.h)
The public header mylib.h will be included as #include "mylib.h" both by mylib and the clients of mylib:
mylib itself and other targets built in mylib's CMake project (for example tests) need to find include/mylib.h from the mylib source tree
clients of mylib built in their own projects (like myexe) need to find include/mylib.h at its installed location
CMake allows us to specify both include paths for mylib:
target_include_directories(mylib PUBLIC
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>)
We're using the PUBLIC option here since this header is needed on the public interface of mylib. Use PRIVATE for include paths internal to mylib.
The INSTALL_INTERFACE specifies a path relative to the install root, that is, CMAKE_INSTALL_PREFIX. To actually install the public header:
set_target_properties(mylib PROPERTIES PUBLIC_HEADER include/mylib.h")
We also need to install the library itself and the so-called config-module and related files. The config-module is the file which will be used by consuming projects, like myapp to find mylib and get all the parameters needed to link with it. It is similar to the pkg-config's .pc files.
We need two, related install commands. The first one:
install(TARGETS mylib
EXPORT mylib-targets
PUBLIC_HEADER DESTINATION include
ARCHIVE DESTINATION lib
LIBRARY DESTINATION lib
RUNTIME DESTINATION bin)
The list of destinations needed to cover all the standard install locations of static libraries, dll's and so's. If you're sure your library will be built exclusively as a static lib, a single DESTINATION lib would make it.
The interesting part is the EXPORT option. It assigns the list of targets (currently, it's only mylib) to the identifier mylib-targets. This identifier will be used in the next command to generate and install some special files which make find_package(mylib) work in the consuming projects:
install(EXPORT mylib-targets
NAMESPACE mylib::
FILE mylib-config.cmake
DESTINATION lib/cmake/mylib)
This command generates multiple files:
one file for each build configuration (Debug, Release, etc..) which describes the library file and configuration-dependent parameters
a file which describes the configuration-agnostic parameters and also includes all the config-dependent files. Since this file can also be used as a config-module on its own we simply rename it as mylib-config.cmake
The files will be installed into ${CMAKE_INSTALL_PREFIX}/lib/cmake/mylib which is one of the many standard locations the find_package(mylib) command will search for mylib-config.cmake.
Building mylib
We need to specify an install location in the variable CMAKE_INSTALL_PREFIX:
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$PWD/../out ../mylib
and build and install the library:
cmake --build . --target install
Building myexe
myexe needs to know where to look for mylib. The variable CMAKE_PREFIX_PATH can be a list of paths. We need to specify the previous install location:
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=$PWD/../out ../myexe
cmake --build .
A note on building multiple configurations
Usually we need to build multiple configurations (Debug, Release). A critical issue is to specify configuration-dependent filenames or install locations. For example, you can set the default value of the DEBUG_POSTFIX property for the library project:
set(CMAKE_DEBUG_POSTFIX d)
The debug version of the mylib library file will be named libmylibd.lib (or mylibd.lib on Windows). The generated EXPORT files will contain the modified filenames.
If you're using makefile-style CMake generators you can control the build configuration by setting the CMAKE_BUILD_TYPE variable:
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/../out ../mylib
cmake --build . --target install
You may need seperate build directories for each configuration or you can re-use the same build dir. In that case, to play it safe it's best to explicitly clean before build:
cmake --build . --target install --clean-first
If you're using a multiconfig IDE generator, like Xcode or Visual Studio, you need to specify the configuration in build time:
cmake -DCMAKE_INSTALL_PREFIX=$PWD/../out ../mylib
cmake --build . --target install --config Release
References
You can clone and build this repository which contains the mylib and myexe projects (tested on Windows and Linux).
Check out the CMake documentation. The most important related commands are:
add_library
target_link_libraries
find_package
install
target_include_directories
target_compile_definitions
and two detailed articles:
Build System
Packages