I have 10 targets in cmake. If i give
gmake -j4
it is compiling all the targets paralllely. I want all the cores to be work on single target and after finishing that it should go for next target. I know that it can be achieved with adding dependency between targets. but i don't want to do that because sometimes i want to compile only one target. if dependency is there, it will compile all the dependency targets before compiling the required target.
how to make the compilation faster but in the sequential manner ?
Related
I have a big CMakeLists with a lot of targets. One of them (specifically one C++ file in one library) is taking a lot of memory to compile and is making my CI pipeline run out of memory.
I would like the compilation to reduce to one simultaneous job when compiling this file, then resume parallel compilation when finished.
I looked at Ninja job pools, but it will only allow me to compile a specific target (e.g. the library) in one job, without constraining the other targets.
Is there a way to do so, if possible without manually adding dependencies to this library?
Let's call the target that you want to build without object-level parallelism "heavy_target". If the problem is specifically with heavy_target and not with the combination of itself and its dependencies, then you can do the following: build all the dependencies of heavy_target first with whatever parallelism you want, then build heavy_target with no parallelism, then build everything else (or build everything and let the buildsystem detect that heavy_target and all its dependencies have been built).
cmake --build <binary_dir> --target <dep1> <dep2> <dep3> <...> <other args>
cmake --build <binary_dir> --target heavy_target --parallel 1 <other args>
cmake --build <binary_dir> <other args>
When building heavy_target's dependencies, you should only need to list the direct dependencies, and the generated buildsystem will know about the transitive ones.
<other args> might be things like --config <config>, --parallel <jobs>, etc. see the docs on arguments to the cmake command.
If it's inconvenient to read through the cmake files to find out all the dependencies of heavy_target, you can get a list of the link-libraries of heavy_target by reading the LINK_LIBRARIES target property:
get_property(heavy_target_link_libraries TARGET heavy_target PROPERTY LINK_LIBRARIES)
message("direct deps of heavy_target: ${heavy_target_link_libraries}")
Inlining the asker's comment: "It should also be possible to create a custom phony target depending on all heavy_target's dependencies."
I have an External project called messages. I am using ExternalProject_Add in order to fetch and build the project.
If i use find_package(messages REQUIRED) in top level CMakeLists.txt the cmake .. fails because it couldn't find the package installation files, which is logical as they are only build during make command invocation.
I am not sure, if there is way use find_package() on ExternalProjects. If so, please point me to an example.
Thanks
BhanuKiran
You have misunderstood how ExternalProject is supposed to work. You cannot find_package(messages REQUIRED) because it hasn't been built yet. ExternalProject merely creates the build steps necessary to build the subproject.
You have two options:
Use add_subdirectory or FetchContent in place of ExternalProject. In this case, you don't need a find_package call. This effectively adds the sub-project to the main build and imports the subproject's targets.
Use two ExternalProject calls: one for messages and another for main_project, which depends on messages. If messages uses the export(EXPORT) function, you can point CMAKE_PREFIX_PATH or messages_ROOT to the build directory. Otherwise you'll need to run the install step for messages and set up an install prefix inside your build directory. Then the find_project(messages REQUIRED) call inside main_project will succeed. This will likely require re-structuring your build.
Generally speaking, ExternalProject is only useful for defining super-builds, which are chains of CMake builds that depend on one another. And super builds are only useful when you need completely different configure-time options, like different toolchains (eg. you're cross compiling, but need a code generator to run on the build machine). If that's not the case, prefer FetchContent or add_subdirectory with a git submodule.
It is best to use FetchContent with CMake 3.14+ since it adds the FetchContent_MakeAvailable macro that cuts down on boilerplate.
Docs:
https://cmake.org/cmake/help/latest/module/ExternalProject.html
https://cmake.org/cmake/help/latest/module/FetchContent.html
Since I like keeping my CMake file agnostic on how I get my packages.
I was using FetchContent and added this file (Findalib.cmake):
if(NOT alib_POPULATED)
set(alib_BUILD_TESTS OFF CACHE BOOL INTERNAL)
set(alib_BUILD_EXAMPLES OFF CACHE BOOL INTERNAL)
set(alib_BUILD_DOCS OFF CACHE BOOL INTERNAL)
FetchContent_MakeAvailable(alib)
endif()
set(alib_FOUND TRUE)
Then, in my CMake files:
find_package(alib REQUIRED)
target_link_libraries(my-executable PUBLIC alib::alib)
That way, packages are only declared in my file in which I declare dependencies, and I fetch them only if I try to find them.
I have a project that needs to be built with an LLVM pass. Here is the project structure:
proj/
instrumentation/
CMakeLists.txt
Instrumentation.cpp
[My project files]
CMakeLists.txt
instrumentation/CMakeLists.txt has a target to build the LLVM pass called MyPass.
I've added the following the CMakeLists.txt:
add_subdirectory(instrumentation)
add_compile_options(
"SHELL:-Xclang -load"
"SHELL:-Xclang $<TARGET_FILE:MyPass>")
Now I just need to add MyPass as a dependency for all the targets in my project. I was wondering if there was a way to force MyPass to build first so that when compiling other files, the LLVM pass would be present. Maybe there a way to add a target compile_options dependencies?
Note: I'd rather not add the dependency manually to every target because there many targets in the project.
My solution to this is to create another project to wrap the compiler executable and automatically build with the LLVM pass. Then rebuild the original project with the wrapped compiler.
AFL does this exact thing when building with a clang compiler.
I'm using CMake for a project where I want to bundle Clang. I use ExternalProject_Add to build clang from source. However, since Clang and LLVM is huge, a make with nothing changed takes 45 seconds.
Is there a way to make CMake just build the ExternalProject once, and then not even check if anything has changed on subsequent runs if it has already been built successfully?
The best way to use ExternalProject_Add() is to structure your project as a superbuild. This means that your top-level project (the "superbuild") does not build any actual code and instead consists only of ExternalProject_Add calls. Your "real" project is added as one of these "external" projects. This allows you to set up the superbuild with all dependencies, ordering, etc.
The workflow is then as follows:
Generate the superbuild project.
Build the superbuild project. This will build and install all dependencies, and also generate (and build) your real project.
Switch to the buildsystem generated for your real project and start doing further development using that. Your dependencies are already correctly set up and installed by the build of the superbuild project in the previous step, and will never be checked for out-of-dateness by the build.
If you ever need to change the setup of your dependencies, do it and build the superbuild again.
With a growing codebase, it makes sense to organize it between separate repositories, each repo being a separate CMake-managed project.
Because of modularity, this usually means you end up in a situation where a CMake-managed project Application depends on another CMake-managed project Library, while both are internal code (i.e., code owned and maintained by your structure).
The automatic dependency recompilation issue
Then, if some sources in Library are modified, it needs to be recompiled in order to build Application. The question being:
Is it possible to have the "build Application" command (a button in an IDE, or a call to make on the command line) to first rebuild Library if Library files changed ?
I'd suggest to use the ExternalProject_Add command.
The documentation has slightly changed for the different versions:
CMake v2.8.9 ExternalProject
CMake v3.0. ExternalProject
CMake v3.3 ExternalProject
In case you encounter problems with getting the dependencies right, this thread might help you.
By looking at how the OpenChemistry parent-project does it, and with the confirmation by normanius's answer, it turns out this can be achieved with relatively few CMake script code.
It turns out that CMake CLI is offering an abstraction over the "build" action of the targeted build systems. See --build option.
ExternalProject_Add can be seen as a wrapper to use this CLI interface directly from CMake scripts.
Imagine there is a CMake-managed repository, building libuseful, and a separate CMake-managed repo, building appawesome with a dependency on libuseful.
find_package(libuseful CONFIG) # The usual way to find a dependency
# appawesome is the executable we are building, it depends on libuseful
add_executable(appawesome main.cpp)
target_link_libraries(appawesome libuseful)
Adding automatic rebuild
Then it is possible to make building appawesome systematically first try to rebuild libuseful with some code looking like:
ExternalProject_Add(EP_libuseful)
SOURCE_DIR <libuseful_sourcedir> # containing libuseful's root CMakeLists.txt
BINARY_DIR <libuseful_binarydir> # containing libuseful's CMakeCache.txt
BUILD_ALWAYS 1 # Always rebuild libuseful
)
add_dependencies(libuseful EP_libuseful)
The last line is quite important: find_package() in config mode should make a libuseful imported targed available. The call to ExternalProject_Add made a build target EP_libuseful available (which is a custom build step, building libuseful). The last line just makes sure that libuseful depends on its build step.