I have a service - say ServiceA. It is mainly kotlin, with a mix of java.
It has a dependency on another lib - LibA. It is written in Java.
Both of these are in separate repos, with their own .gitlab pipelines.
Now, in the gradle of ServiceA, we consume the LibA with the fully specified version string: like <group>:<artifact>:<version>.
But in intellij, I want to use the local checkout of LibA. So whenever I build/debug/run ServiceA, I want LibA to build.
In the past, I used to achieve this by editing the dependencies in Idea for ServiceA. I would remove the explicitly specified artifact <group>.<artifact>.<version> and add a "Module Dependency" for LibA. I made sure to import LibA as a module in my Idea project.
All of this used to work in the past. But now, it doesnt work.
When I rebuild project, it first build LibA and then ServiceA
But if I do a Debug or Run, it does not. So it ends up using the wrong version of LibA and then my breakpoints in LibA dont hit anymore.
Any ideas what I am doing wrong?
Related
I'm trying to understand what some of the best practices are when using modern CMake (3.13+) with respect to building and including vendored or submoduled code.
Say I'm building a library MyLib. My file structure is something like this
MyLib
|-CMakeLists.txt
|-src
|-include
|-submodules
|-libgeos
In this example, I've included libgeos as a git submodule, because it's really convenient to be able to clone the project and immediately build and run tests because that dependency is present. This could also be solved by using FetchContent or something, and my question still stands; the important thing is that I do not want to rely on libgeos being installed in build environment.
Note I picked libgeos arbitrarily; I have no idea if libgeos is set up as a cmake project appropriately for this example, but this is all theoretical and I just needed some concrete library name. Please do not use the specific details of how libgeos is configured to answer this, unless libgeos is a good example of conventional cmake.
But now, there's some other project that wants to use my project, and it needs libgeos and doesn't want to depend on my project providing it.
OtherProject
|-CMakeLists.txt
|-src
|-include
|-submodules
|-libgeos
|-MyLib
|submodules
|-libgeos
When you clone OtherProject, you get two versions of libgeos, and maybe that's not great; but it's not a huge issue either. And maybe they're not the same version; say MyLib requires libgeos >= 2.0, so 2.0 is what MyLib includes, and OtherProject requires libgeos>=2.1 so OtherProject includes libgeos >= 2.1.
Now we potentially end up with some build issues. If we have the following line in OtherProject/CMakeLists.txt
add_subdirectory(submodules/libgeos)
and then again, that same line within MyLib/CMakeLists.txt, we end up with cmake errors because libgeos as a target is defined twice in the build. This can be solved a couple of ways.
Check if geos exists before adding it
if(NOT TARGET geos)
add_subdirectory(submodules/libgeos)
endif()
But this case has some issues; if that blob is in OtherProject at the top, it's fine and both projects use libgeos 2.1. But if it's in OtherProject after add_subdirectory(submodules/MyLib), then the geos 2.0 version gets added to the build, which may or may not fail loudly (Hopefully it would).
This could also be solved with find_package. Both projects include cmake/FindGeos.cmake which use that blurb above (if(NOT TARGET...)) to add geos the build and then the top project cmake files can do this
list(APPEND CMAKE_MODULE_PATH cmake)
find_package(geos 2) # (or 2.1)
then it doesn't matter what order they try to include geos, because they will both defer to FindGeos.cmake in OtherProject because it's first in the module path.
But now there's a new issue, some ThirdProject wants to use MyLib also, but ThirdProject wants to depend on libgeos which is in the system environment. It uses find_package(geos 2.1 CONFIG) to use the installed GeosConfig.cmake file, which adds geos::geos to the build and sets geos_FOUND. Suddenly, MyLib fails to build, because geos_FOUND was set, but I'm doing target_link_library(mylib PUBLIC geos).
So this could be solved by adding add_library(geos::geos ALIAS geos) in both custom FindGeos.cmake files, then it doesn't matter if geos was built from source or using the installed version, the target names are the same either way.
Now we get to my actual questions:
Lets start with
Am I crazy, no one does this, and my team is trying to use cmake all wrong?
Is there some feature of cmake that I've just completely missed that solves all these problems?
I suspect there's a good few books or presentations that cover this topic, but I just don't know where to look because there's so many; what should I be looking at? I've seen the CMake Packages page, which looks like it solves the problem when you're using all projects which are configured according to that page; but it doesn't really answer how to bridge the gap between older and newer projects.
If I'm not crazy and there's no straightforward answer or presentation that I can look at, then
What should the cmake configuration for both MyLib and libgeos look like so that these cases work?
MyLib is built alone
MyLib is built as part of a larger project which provides a different version of geos
MyLib is built as part of a larger project which depends on a different version of geos in the environment
I understand that cmake provides helpers that could be used to produce MyLibConfig.cmake if I wanted to install it in the environment. I also see that the export() function exists, which could be used to save those files in the build tree somewhere and then find them with find_package in config mode. But this feels a bit odd to me to do because it's not a multi-stage build, it's just one invocation of cmake then make.
But lets say that's the right answer and the CMake for libgeos doesn't follow it. Would it be appropriate to have FindGeos.cmake do something like this?
if(NOT geos_FOUND)
add_subdirectory(submodules/libgeos)
export(geos NAMESPACE geos)
find_package(geos CONFIG)
endif()
Long story short, I rewrote the Godot build system to cmake (only windows part), mostly because I wanted to learn it, but I have trouble compiling Godot with mingw. When I'm trying to compile it, at first everything goes fine, up until the point of linking final exe, where I get a lot of "undefined reference to" errors. It looks like main libraries (core/scene/editor/..) can't see functions from each other. MSVC build is working fine, and scons version is also compiling under mingw, so I clearly just missed something in my cmake version.. I tried to remove some compile/linking options throughout my cmake scripts as a test, but nothing changed. I don't really know how even debug this problem, so if someone could kick me in the right direction I would be really glad for it.
Ok, I finally got time to came back to this.
The problem
So basically the problem was in circular dependency on godot libs. I didn't thought this was the problem because I'm spoiled by MSVC (which doesn't depend on link order). Also, I tried to replicate circular dependency in mingw on a test project with much smaller scale. The project was a 30 libs with two functions in each, first function printing string, and another calling all first functions from all 30 libs, so there is 30 libs of circular dependency. Strangely enough, the project linked no problems, and printed 30^2 strings..
The solution
The solution is to use -Wl,--start-group/-Wl,--end-group linking flags around all libraries. There is two ways you can do it.
First way, is to add all your libraries to the some list of sorts. This could be global property, or property on some target (not just a simple variable), so it could be accessed from other subdirectories. After you formed your list of libraries you simply link it to the executable as follows
# getting all your libs from the global property..
get_property(__LIBS_LIST GLOBAL PROPERTY EXE_LIBS_LIST)
# linking all libraries to the exe..
target_link_libraries(my-exe PRIVATE -Wl,--start-group ${__LIBS_LIST} -Wl,--end-group)
This is the easiest solution, but be cautious about dependencies on libraries which you link to your exe, because it seems that when CMake creates link line for your exe, it first lists all libraries which are linked directly to your exe, and only after it places libraries which came from dependencies of libraries linked directly. Basically if your target dependency tree looks something like this:
exe // your main exe file
- lib_A // lib A linked directly to the main exe
- lib_AA // lib AA linked to the lib_A
- lib_AAA // lib AAA linked to the lib_AA
- lib_B // lib B linked directly to the main exe
- lib_BB // lib BB linked to the lib_B
- lib_BBB // lib BBB linked to the lib_BB
your link order for the exe will look something like this:
// first libs linked directly to the exe
lib_A
lib_B
// only after recursively initial libs dependencies
lib_AA
lib_AAA
lib_BB
lib_BBB
That's meen, that if you will link your libs like target_link_libraries(my-exe PRIVATE -Wl,--start-group ${__LIBS_LIST} -Wl,--end-group), --start-group and --end-group will guard only the libraries linked directly to your exe. I didn't found this described in documentation, but I found SO question which talks pretty much about same behaviour (CMake library linking order). Also, as I tested this on mingw, it didn't mattered how exactly libs lib_AA/lib_AAA/lib_BB/lib_BBB were linked, via PRIVATE or via INTERFACE, the results were the same.
Second way, is to exploit recursivity of link expanding for dependencies of libraries linked directly to the exe. From my example you can see, that dependencies of lib_A (lib_AA/lib_AAA) weren't mixed with dependencies of lib_B (lib_BB/lib_BBB). So basiacaly what we can do, is to create INTERFACE library and connect to it -Wl,--start-group immediately after that. Then add any number of libraries to it's interface and link global-libs to your exe (order does not matter). And in very end, close the group in your global-libs library
add_library(global-libs INTERFACE)
target_link_libraries(global-libs INTERFACE -Wl,--start-group)
# ...
# linking another libs, and linking global-libs to exe
# ...
target_link_libraries(global-libs INTERFACE -Wl,--end-group)
This will ensure, that all libraries connected to global-libs will be surrounded by -Wl,--start-group/-Wl,--end-group.
Now, theoretically, CMake should handle circular dependency by itself, by placing libraries in link line multiple times (how many times controlled by LINK_INTERFACE_MULTIPLICITY). But this method didn't worked for me (mb I just missed something..). Plus, you need declare dependencies between cmake targets, and with -Wl,--start-group/-Wl,--end-group you can just set one specific interface library as a holder for all libs with circular dependencies..
I'm currently writing a library (let's call it mylib) using modern cmake (i.e. with proper targets and properties). It's was all good until I encountered a 3rdparty dependency (let's call it libfoo) that doesn't define imported targets in its FooConfig.cmake.
For this, I decided to write my own custom FindFoo.cmake module where I define the target Foo::Foo of the dependency myself. This works fine for building mylib, but fails when attempting to link an executable against it (after installing it).
Here is a simplified version of MyLibConfig.cmake:
include(CMakeFindDependencyMacro)
find_dependency(Foo 1.0 REQUIRED)
include(${CMAKE_CURRENT_LIST_DIR}/MyLibTargets.cmake)
Now, when I try to link against mylib from an executable (let's call it myexec), I get the following error:
Target "myexec" links to target "Foo::Foo" but the target was
not found. Perhaps a find_package() call is missing for an IMPORTED
target, or an ALIAS target is missing?
I understand why this happens: the target Foo::Foo was defined in my custom FindFoo.cmake module (and it's a public dependency of mylib). However, this module is not available to myexec when it loads MyLibConfig.cmake.
The question now is, what's the best possible solution to this problem? One idea that I saw was to install FindFoo.cmake with mylib and add it to CMAKE_MODULE_PATH when running MyLibConfig.cmake, bus this seems wrong to me. How can I make transitive targets available to users?
With a growing codebase, it makes sense to organize it between separate repositories, each repo being a separate CMake-managed project.
Because of modularity, this usually means you end up in a situation where a CMake-managed project Application depends on another CMake-managed project Library, while both are internal code (i.e., code owned and maintained by your structure).
The automatic dependency recompilation issue
Then, if some sources in Library are modified, it needs to be recompiled in order to build Application. The question being:
Is it possible to have the "build Application" command (a button in an IDE, or a call to make on the command line) to first rebuild Library if Library files changed ?
I'd suggest to use the ExternalProject_Add command.
The documentation has slightly changed for the different versions:
CMake v2.8.9 ExternalProject
CMake v3.0. ExternalProject
CMake v3.3 ExternalProject
In case you encounter problems with getting the dependencies right, this thread might help you.
By looking at how the OpenChemistry parent-project does it, and with the confirmation by normanius's answer, it turns out this can be achieved with relatively few CMake script code.
It turns out that CMake CLI is offering an abstraction over the "build" action of the targeted build systems. See --build option.
ExternalProject_Add can be seen as a wrapper to use this CLI interface directly from CMake scripts.
Imagine there is a CMake-managed repository, building libuseful, and a separate CMake-managed repo, building appawesome with a dependency on libuseful.
find_package(libuseful CONFIG) # The usual way to find a dependency
# appawesome is the executable we are building, it depends on libuseful
add_executable(appawesome main.cpp)
target_link_libraries(appawesome libuseful)
Adding automatic rebuild
Then it is possible to make building appawesome systematically first try to rebuild libuseful with some code looking like:
ExternalProject_Add(EP_libuseful)
SOURCE_DIR <libuseful_sourcedir> # containing libuseful's root CMakeLists.txt
BINARY_DIR <libuseful_binarydir> # containing libuseful's CMakeCache.txt
BUILD_ALWAYS 1 # Always rebuild libuseful
)
add_dependencies(libuseful EP_libuseful)
The last line is quite important: find_package() in config mode should make a libuseful imported targed available. The call to ExternalProject_Add made a build target EP_libuseful available (which is a custom build step, building libuseful). The last line just makes sure that libuseful depends on its build step.
I have the following scenario:
mylib is a library (for which I have the sources, so I'd like to put them into a Maven project mylib:mylib for example). This library has a jar dependency for which I only have the jar, and it is not to be found in the Maven repository (and I do NOT want to install it there either). To make it compile, something like this would work: add the jar file to the mylib project in a "lib" folder, e.g. "lib/thirdpartylib.jar" and in mylib's pom.xml, add a dependency with self-chosen group/artifact/version and a "<scope>system</scope><systemPath>${project.basedir}/lib/thirdpartylib.jar</systemPath>" entry. The mylib project will compile fine.
Note that mylib also has a runtime dependency to a dll file, say thirdparty.dll. But for compilation this is not important.
However, now what I am wondering how to achieve the following:
Any other projects, e.g. project "X", that uses mylib, will need the
- mylib.jar
- thirdpartylib.jar
- thirdpartylib.dll
,
and it'll have to set the java.library.path to the directory (e.g. ".") such that the thirdparty jar and dll are found by the executing VM.
My concern is this: I would like the thirdparty jar/dll things to be the responsibility of the mylib project. I.e. I want to define the knowledge, that you need to copy the thirdparty jar and dll to the target folder and that java.library.path refers to them, to be part of the mylib project (the mylib pom knows how the thing is to be used in other projects). Yet, I want this knowledge (i.e. copy instructions, regardless how they are exactly done in Maven) to be transitively handed over to any other project using mylib, like X for example. Is that somehow possible?
[My hack solution for now would be that I have a copy of the thirdparty things in X, but even then I dunno how to copy/deal with the dll file, so I'd have to write a readme saying that the dll file has to be copied to the bin folder of the executing VM).
Any suggestions are appreciated!
The base idea is the following:
Maven is good in handling with one result per Maven POM.
It is possible to have libraries and dependencies to these libraries in your local repositories only.
So you have to do the following steps:
Define for the additional library a separate project (or a module in your project) and define the library as the result.
Change your POM so that this POM has now a dependency to the new project.
Do the same steps for your DLL (see the post Managing DLL dependencies with Maven) how to do that).
Deploy your additional library and your DLL to your local repository.
Now you should be able to use Maven for the build process again, and by using additional plugins like the maven-assembly-plugin, you are able to pack all together.