I've got a project managed by CMake with multiple libraries that have link time dependencies between them, but each of the libraries can be compiled independently of each other. How do I express this to CMake so that I can build all the libraries concurrently?
For example, I tried this CMakeLists.txt:
cmake_minimum_required(VERSION 2.6)
project (test)
add_library(lib1 STATIC lib1.cpp)
add_library(lib2 STATIC lib2.cpp)
add_library(lib3 STATIC lib3.cpp)
add_executable(main main.cpp)
target_link_libraries(lib2 lib1)
target_link_libraries(lib3 lib2)
target_link_libraries(main lib3)
Each file just defines a different empty function like:
void f1() {}
When I type cmake . && make -j4, I see this:
[ 25%] Building CXX object CMakeFiles/lib1.dir/lib1.cpp.o
Linking CXX static library liblib1.a
[ 25%] Built target lib1
[ 50%] Building CXX object CMakeFiles/lib2.dir/lib2.cpp.o
Linking CXX static library liblib2.a
[ 50%] Built target lib2
[ 75%] Building CXX object CMakeFiles/lib3.dir/lib3.cpp.o
Linking CXX static library liblib3.a
[ 75%] Built target lib3
[100%] Building CXX object CMakeFiles/main.dir/main.cpp.o
Linking CXX executable main
[100%] Built target main
Even though I've specified -j4 and compiling each .cpp file should never depend on any .a files, it's waiting on the previous compile and link to finish begin the next one. I'd rather see something like:
Building CXX object CMakeFiles/lib1.dir/lib1.cpp.o
Building CXX object CMakeFiles/lib2.dir/lib2.cpp.o
Building CXX object CMakeFiles/lib3.dir/lib3.cpp.o
Building CXX object CMakeFiles/main.dir/main.cpp.o
Linking CXX static library liblib1.a
Built target lib1
Linking CXX static library liblib2.a
Built target lib2
Linking CXX static library liblib3.a
Built target lib3
Linking CXX executable main
Built target main
Is it possible to tell CMake that it can build all the .o files concurrently?
In reality, I'm doing this in a million-line project with about 20 CPU cores at my disposal (with distcc), so this is a huge bottleneck on my build times.
The sequential execution is probably a consequence of the link dependencies established between the static libraries lib1, lib2 and lib3.
One work-around is to get rid of these static library link dependencies. Since you are building static libraries anyway, removing the dependencies will not prevent them from being linked successfully. The executable main needs to depend on all libraries then:
cmake_minimum_required(VERSION 2.6)
project (test)
add_library(lib1 STATIC lib1.cpp)
add_library(lib2 STATIC lib2.cpp)
add_library(lib3 STATIC lib3.cpp)
add_executable(main main.cpp)
target_link_libraries(main lib1 lib2 lib3)
Organized this way make -j builds the libraries in parallel.
If getting rid of the link dependencies is not an option, you can apply the principle "Any problem in computer science can be solved with another layer of indirection":
cmake_minimum_required(VERSION 2.6)
project (test)
add_library(lib1_objects STATIC lib1.cpp)
add_library(lib2_objects STATIC lib2.cpp)
add_library(lib3_objects STATIC lib3.cpp)
add_executable(main main.cpp)
add_library(lib1 STATIC empty.cpp)
add_library(lib2 STATIC empty.cpp)
add_library(lib3 STATIC empty.cpp)
target_link_libraries(lib1 lib1_objects)
target_link_libraries(lib2 lib2_objects lib1)
target_link_libraries(lib3 lib3_objects lib2)
target_link_libraries(main lib3)
This sets up helper libraries (e.g., lib1_objects), which have no dependencies and can thus be built in parallel. The original libraries link to these helper libraries and also have the required link dependencies set up. empty.cpp is just an empty dummy CPP source file.
Related
I have a problem with Connector/C++.
I'm using CLion as IDE and want to create a c++ program to interact with mysql database.
this is my CMakeList.txt file which i include c++/connector static and dynamic libraries in it:
cmake_minimum_required(VERSION 3.15)
project(cpp_programming)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_INCLUDE_CURRENT_DIR ON)
set(MYSQL_CPPCONN_DIR "C:/Program Files/MySQL/MySQL Connector C++ 8.0")
include_directories(${PROJECT_NAME} PUBLIC ${MYSQL_CPPCONN_DIR}/include)
add_executable(${PROJECT_NAME} main.cpp)
# Static Libraries
target_link_libraries(${PROJECT_NAME} ${MYSQL_CPPCONN_DIR}/lib64/vs14/libcrypto.lib)
target_link_libraries(${PROJECT_NAME} ${MYSQL_CPPCONN_DIR}/lib64/vs14/libssl.lib)
target_link_libraries(${PROJECT_NAME} ${MYSQL_CPPCONN_DIR}/lib64/vs14/mysqlcppconn.lib)
target_link_libraries(${PROJECT_NAME} ${MYSQL_CPPCONN_DIR}/lib64/vs14/mysqlcppconn8.lib)
target_link_libraries(${PROJECT_NAME} ${MYSQL_CPPCONN_DIR}/lib64/vs14/mysqlcppconn8-static.lib)
target_link_libraries(${PROJECT_NAME} ${MYSQL_CPPCONN_DIR}/lib64/vs14/mysqlcppconn-static.lib)
# Dynamic Link Libraries
target_link_libraries(${PROJECT_NAME} ${PROJECT_SOURCE_DIR}/libcrypto-1_1-x64.dll)
target_link_libraries(${PROJECT_NAME} ${PROJECT_SOURCE_DIR}/libssl-1_1-x64.dll)
target_link_libraries(${PROJECT_NAME} ${PROJECT_SOURCE_DIR}/mysqlcppconn-7-vs14.dll)
target_link_libraries(${PROJECT_NAME} ${PROJECT_SOURCE_DIR}/mysqlcppconn8-2-vs14.dll)
And i just include xdevapi.h header in my c++ source file like this:
#include <iostream>
#include <mysqlx/xdevapi.h>
using namespace std;
int main()
{
return 0;
}
And i run file in Release mode in clion and i receive these errors:
Error message i see in the clion console
====================[ Build | cpp_programming | Release ]=======================
"C:\Program Files\JetBrains\CLion 2019.3.2\bin\cmake\win\bin\cmake.exe" --build C:\Users\Kianoush\CLionProjects\cpp_programming\cmake-build-release --target cpp_programming -- -j 2
Scanning dependencies of target cpp_programming
[ 50%] Building CXX object CMakeFiles/cpp_programming.dir/main.cpp.obj
[100%] Linking CXX executable cpp_programming.exe
CMakeFiles\cpp_programming.dir/objects.a(main.cpp.obj):main.cpp:(.text$_ZNK6mysqlx4abi22r05Value5printERSo[_ZNK6mysqlx4abi22r05Value5printERSo]+0x21): undefined reference to `mysqlx::abi2::r0::DbDoc::print(std::ostream&) const'
CMakeFiles\cpp_programming.dir/objects.a(main.cpp.obj):main.cpp:(.text$_ZNK6mysqlx4abi22r05Value5printERSo[_ZNK6mysqlx4abi22r05Value5printERSo]+0x2c): undefined reference to `mysqlx::abi2::r0::common::Value::print(std::ostream&) const'
CMakeFiles\cpp_programming.dir/objects.a(main.cpp.obj):main.cpp:(.text$_ZNK6mysqlx4abi22r08internal14Warning_detail5printERSo[_ZNK6mysqlx4abi22r08internal14Warning_detail5printERSo]+0x87): undefined reference to `mysqlx::abi2::r0::string::Impl::to_utf8[abi:cxx11](mysqlx::abi2::r0::string const&)'
CMakeFiles\cpp_programming.dir/objects.a(main.cpp.obj):main.cpp:(.rdata$_ZTCN6mysqlx4abi22r05ValueE0_NS1_6common5ValueE[_ZTCN6mysqlx4abi22r05ValueE0_NS1_6common5ValueE]+0x20): undefined reference to `mysqlx::abi2::r0::common::Value::print(std::ostream&) const'
CMakeFiles\cpp_programming.dir/objects.a(main.cpp.obj):main.cpp:(.rdata$.refptr._ZTVN6mysqlx4abi22r05DbDocE[.refptr._ZTVN6mysqlx4abi22r05DbDocE]+0x0): undefined reference to `vtable for mysqlx::abi2::r0::DbDoc'
collect2.exe: error: ld returned 1 exit status
CMakeFiles\cpp_programming.dir\build.make:97: recipe for target 'cpp_programming.exe' failed
CMakeFiles\Makefile2:74: recipe for target 'CMakeFiles/cpp_programming.dir/all' failed
CMakeFiles\Makefile2:81: recipe for target 'CMakeFiles/cpp_programming.dir/rule' failed
mingw32-make.exe[3]: *** [cpp_programming.exe] Error 1
mingw32-make.exe[2]: *** [CMakeFiles/cpp_programming.dir/all] Error 2
mingw32-make.exe[1]: *** [CMakeFiles/cpp_programming.dir/rule] Error 2
mingw32-make.exe: *** [cpp_programming] Error 2
Makefile:117: recipe for target 'cpp_programming' failed
Do i make mistakes in linking dll files or static files ?
What solution do you suggest ?
please help me, this take me in trouble for many days.
Full screen image
There are a couple of issues here.
You do not need to link all of these different libraries. You really only should require one mysqlcppconn library to be linked. To locate the library, try using find_library(), and use it for linking instead. Your CMake file should reduce to something like this:
cmake_minimum_required(VERSION 3.15)
project(cpp_programming)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_INCLUDE_CURRENT_DIR ON)
set(MYSQL_CPPCONN_DIR "C:/Program Files/MySQL/MySQL Connector C++ 8.0")
include_directories(${PROJECT_NAME} PUBLIC ${MYSQL_CPPCONN_DIR}/include)
add_executable(${PROJECT_NAME} main.cpp)
# Find the mysqlcppconn library.
find_library(mysqlcppconn_LIB
mysqlcppconn8
HINTS ${MYSQL_CPPCONN_DIR}/lib/vs14
# Static Libraries
target_link_libraries(${PROJECT_NAME} PUBLIC ${mysqlcppconn_LIB})
I'm not sure if you are using the ssl and crypto libraries, so add those back in if needed.
Your libraries (VisualC++) do not match the compiler (MinGW) you are using. To my knowledge, the MySQL Connector C++ downloads do not provide a MinGW set of libraries; they only provide libraries that are built with the Visual Studio compiler. Thus, you need to switch to use the VisualC++ compiler to use these libraries. Another option would be to download the MySQL source and try to build it with MinGW, but that may be more difficult.
Hope this helps!
I have a library that gets build and depends on other libraries:
add_library(library1 STATIC main.c)
target_link_libraries(library1 m;pthread)
If I want to now use the just declared library1 in other library I would expect that libm and libpthread to be passed in the linker to build library2
add_library(library2 STATIC main2.c)
target_link_libraries(library2 library1) # I would expect m and pthread to be linked to library2 but don't
add_executable(binary library2)
Is the normal behavior pass libm and libpthread to the compiler to build binary?
Thanks
I have a big Fortran program which contains many directories. Each directory is compiled separately in a pseudo-library, but there is still an interdependency mess, so at the end all pseudo-libraries are combined in a single usable library. I'd like to use Fortran modules, but it's very fragile, since I cannot rely on automatic dependency checking, and compilation may fail depending on the order.
For instance, consider the following CMakeLists.txt file:
project (test Fortran)
add_library (lib1 dir1/lib1.f90)
add_library (lib2 dir2/lib2.f90 dir2/mod.f90)
add_executable (exe dir3/exe.f90)
target_link_libraries (exe lib1 lib2)
With the sources:
dir1/lib1.f90:
subroutine bar
use foo, only: foofoo
implicit none
write(6,*) foofoo
end subroutine bar
dir2/lib2.f90:
subroutine bar2
use foo, only: foofoo
implicit none
write(6,*) foofoo,' again'
end subroutine bar2
dir2/mod.f90:
module foo
implicit none
integer :: foofoo=3
end module foo
dir3/exe.f90:
program meh
implicit none
call bar()
call bar2()
end program meh
Compiling from scratch fails:
$ make
[ 25%] Building Fortran object CMakeFiles/lib1.dir/dir1/lib1.f90.o
/home/user/cmake/dir1/lib1.f90:2.4:
use foo, only: foofoo
1
Fatal Error: Can't open module file 'foo.mod' for reading at (1): No such file or directory
make[2]: *** [CMakeFiles/lib1.dir/dir1/lib1.f90.o] Error 1
make[1]: *** [CMakeFiles/lib1.dir/all] Error 2
make: *** [all] Error 2
but doing it in the right order works:
$ make lib2
Scanning dependencies of target lib2
[ 50%] Building Fortran object CMakeFiles/lib2.dir/dir2/mod.f90.o
[100%] Building Fortran object CMakeFiles/lib2.dir/dir2/lib2.f90.o
Linking Fortran static library liblib2.a
[100%] Built target lib2
$ make
[ 25%] Building Fortran object CMakeFiles/lib1.dir/dir1/lib1.f90.o
Linking Fortran static library liblib1.a
[ 25%] Built target lib1
[ 75%] Built target lib2
Scanning dependencies of target exe
[100%] Building Fortran object CMakeFiles/exe.dir/dir3/exe.f90.o
Linking Fortran executable exe
[100%] Built target exe
Is there any way CMake can figure out the dependency and compile lib2 (or at least mod.f90) before lib1?
ETA: A robust solution should work regardless of the order in which lib1 and lib2 are defined in the CMakeLists.txt file and, once the program has been compiled, after running rm foo.mod ; touch ../dir1/lib1.f90.
The upcoming ninja build system version (1.10.0) has support for dynamic dependencies, which will resolve modules compilation order correctly.
To use it with CMake, you have to specify the Ninja generator:
cmake .. -DCMAKE_GENERATOR=Ninja # generate project
ninja # build the project
Here problem is that lib1 target required object file for mod.f90. But there is no rule mention in CMakeLists.txt to create mod.f90.o while creating liblib1.a. For lib2 target mod.f90.o is created.
There can be two possible solution as below.
Solution-1
Add mod.f90 to both library.
project (test Fortran)
add_library (lib1 dir1/lib1.f90 dir2/mod.f90)
add_library (lib2 dir2/lib2.f90 dir2/mod.f90)
add_executable (exe dir3/exe.f90)
target_link_libraries (exe lib1 lib2)
Solution-2
Link library with mod.f90.o to other one.
project (test Fortran)
add_library (lib1 dir1/lib1.f90)
add_library (lib2 dir2/lib2.f90 dir2/mod.f90)
add_dependencies(lib1 lib2)
add_executable (exe dir3/exe.f90)
target_link_libraries (exe lib1 lib2)
Given this cmake sample project:
lib1/CMakeLists.txt:
add_library(lib1 src1.cpp)
target_include_directories(lib1 PUBLIC include)
lib2/CMakeLists.txt:
add_library(lib2 src2.cpp)
target_include_directories(lib2 PUBLIC include)
target_link_libraries(lib2 PRIVATE lib1)
lib3/CMakeLists.txt:
add_library(lib3 src3.cpp)
target_include_directories(lib3 PUBLIC include)
target_link_libraries(lib3 PRIVATE lib2)
All .cpp and .h files are dummy files with no dependency between them.
From what I understood from the documentation, adding a "PRIVATE" target library from lib2 to lib1 means that lib1 include directory should not be added when compiling lib3. However, when launching compiling this using cmake (3.3.2) and "make VERBOSE=1", command line to compile "src3.cpp" contains "-I/.../lib1/include":
c++ -I/.../lib3/include -I/.../lib2/include -I/.../lib1/include -o .../src3.cpp.o -c /.../lib3/src3.cpp
What did I get wrong ?
From what I understood from the documentation, adding a "PRIVATE"
target library from lib2 to lib1 means that lib1 include directory
should not be added when compiling lib3.
Your assumptions are correct.
However, due to legacy reasons in how properties propagate between dependent targets, this correct behavior is deactivated when specifying an old version in cmake_minimum_required.
By changing the version to 3.0 or later, you will get the correct behavior:
cmake_minimum_required(VERSION 3.0)
...
See also this thread on the CMake mailing list.
I have a CMake project that builds a static library which depends on another static library. I would like to turn this static library into an object libraries. When I do that I get a compiler error and I imagine that there is something I don't get about object libraries.
Here is an example of what I am trying to achieve. MyLib and MyLib2 are both static libraries and MyLib uses a function defined and declared in MyLib2.
MyLib
CMakeList.txt
MyLib.h
MyLib.cpp
MyLib2
CMakeList.txt
MyLib2.h
MyLib2.cpp
MyLib2/CMakeLists.txt:
cmake_minimum_required(VERSION 3.0)
project (MyLib2)
add_library(${PROJECT_NAME} OBJECT MyLib2.cpp)
target_include_directories(${PROJECT_NAME} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
MyLib/CMakeLists.txt:
cmake_minimum_required(VERSION 3.0)
project (MyLib)
add_subdirectory(MyLib2)
add_library(${PROJECT_NAME} STATIC MyLib.cpp MyLib.h)
target_link_libraries(${PROJECT_NAME} MyLib2)
MyLib.h includes MyLib2.h to use the function it declares.
#ifndef MyLib
#define MyLib
#include "MyLib2.h"
#endif
When MyLib2 is built as a static library I can build the code without any problems (I am using make and clang on Mac). However when I turn MyLib2 into an object library I get a compile error saying that MyLib2.h cannot be found.
MyLib.h:4:10: fatal error: 'MyLib2.h'file not found
Here are the content of the CMake files when MyLib2 is an object library.
MyLib2/CMakeLists.txt:
cmake_minimum_required(VERSION 3.0)
project (MyLib2)
add_library(${PROJECT_NAME} OBJECT MyLib2.cpp)
target_include_directories(${PROJECT_NAME} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
MyLib/CMakeLists.txt:
cmake_minimum_required(VERSION 3.0)
project (MyLib)
add_subdirectory(MyLib2)
add_library(${PROJECT_NAME} STATIC MyLib.cpp $<TARGET_OBJECTS:MyLib2>)
I do not understand why MyLib can no longer MyLib2.h when MyLib2 is an object library. Maybe there is something wrong with the way is use target_include_directories.
This was answered in the comments:
#Clem: When you link with libary target, you consume its INTERFACE_INCLUDE_LIBRARIES property, which contains include directory Mylib2. When you use object library via $, you don't use any target, so don't consume any property. – Tsyvarev