Our large repository has a lot of different unit tests executables.
If I build our repo incremetally, cmake only builds those libraries and executables, which need to be build.
If I now start the tests via ctest all tests are executed. Also those tests, which have not changed and thus will give the same result as last time, will be started.
Is there a way to only run tests that changed?
Related
I know that the cmake build stage can be done in parallel, but the initial configuration stage takes too long for large packages. Can that be parallelized?
No, the configuration stage cannot be parallelized.
Nonetheless, here are some things you could do:
Profile the Configuration Phase
CMake has --profiling-output/--profiling-format options, which emits a file that can be opened e.g. in Chromium-based browsers using about:tracing. This visualizes all command that were processes by CMake, how long they took, how they are nested, etc. This information can be used to find bottlenecks in your configuration phase and possibly optimize them to run faster. This is especially useful if you have a lot of custom functions/macros in your CMake files. Example:
$ cmake -G Ninja --profiling-output ./profile.json --profiling-format google-trace ..
Note that the profiling itself poses some additional work on CMake. Therefore, while profiling the CMake run will take longer as usual.
Generate Less Configurations
Running CMake actually does two things: configure and generate. Configure means, all CMakeLists.txt and *.cmake files are read and processes. Generate means, the CMake-internal representation of the build is written to some buildsystem specific format, e.g. Makefile, ninja.build, Visual Studio projects, etc. In case you use a Multi-Config generator like "Ninja Multi-Config", Visual Studio or XCode, it might help to reduce the number of generated configurations. By default CMake generates build files for 4 configurations: Debug, Release, RelWithDebInfo and MinSizeRel. In case you only need a subset of those, you can specify the required ones using the CMAKE_CONFIGURATION_TYPES variable. From my experience (using Ninja Multi-Config) this can reduce the generation phase significantly for large projects.
Superbuild
Use a superbuild structure with ExternalProject. Here you basically have one build that orchestrates the configuration and building of several other projects. This way, the sub-builds including their configuration phases are run in parallel. Note however, that this has other issues like the targets of the other projects not being available at configure time. In my experience, superbuild are only a good choice for special use cases.
Concurrent execute_process()
Someone managed write a ray tracer using CMake. For the parallelization he exploited that execute_process() runs multiple COMMANDs concurrently. This could help, if you have to do some heavy preprocessing that cannot be deferred to the build stage. But then again, it runs as a separate process, so if you call CMake recursively with this you don't have access to targets, variables, etc. of that subprocess.
Do as much as possible in the Build Phase
I don't have insight into your build, but often it help to perform only the things necessary in the configuration phase. Especially for execute_process() calls, one should always evaluate if it could be replaced with add_custom_command()/add_custom_target(), because creating processes is relatively costly.
An example: I had once a large scale project at work where a lot of source files were generated from XML/XSLT files. Since CMake doesn't have any knowledge about the dependencies between XML/XSLT files, we had a script that figured out those dependencies by reading the files and following includes recursively. At first, this script was called using execute_process() at configure time in order to pass the output to the DEPENDS options of add_custom_command(). Later, I optimized this by doing the dependency evaluation at build time and generating a .d dependency file (Makefile syntax) that could be passed to the DEPFILE option of add_custom_command() instead. The speedup in configuration was enormous, while the build time suffered only slightly, because it in parallel.
The dependency between libraries and libraries+executable could be managed via target_link_libraries. So if some library fails build - all dependent targets will not build.
I have another task: there are exist the library target, the executable test target (that provide test for library) and the main executable target with main logic.
I need that main executable target should be build only if build of the executable test target is not fails.
In other word I need set up dependency between two executables.
How I could achieve it? Could somebody provide an example?
Testing is a different stage in the CMake/Ctest/CPack suite, so you’re unlikely to find a way to conditionally compile/link an executable based on test results.
You can use add_dependencies to ensure that the test executable builds first, but that’s not the same as only producing a build product of the test runs successfully.
I recommend running your build followed by the test. Based on the return code of the test, you can choose whether or not to package the final product.
I have a cmake based (C++) project which includes tests created via add_test().
The build process is basically:
cmake
make all testsuite
make test
This has worked well for some time.
I am now trying to get my build to generate a Test.xml to be submitted as part of a pipeline build. It is not working for this project.
Q1 Can I tell cmake that I want a Test.xml to be generated when it (make test) runs the tests?
Currently I believe that a Test.xml can only be created by running ctest.
Presumably ctest aggregates the test results whereas cmake just runs them blindly?
Can someone confirm or refute this?
So I am currently trying to run ctest using:
ctest -vv -debug --output-on-failure -T test
As this does create the Test.xml file I need.
I had to add include(CTest) & include(Dart) to fix "could not find DartConfiguration.tcl".
However, even if I have previously run "make test" which guarantees the build is up to date it still tries to build and I get:
Error: could not load cache
For tests that run a compiled test (script based tests work fine).
The command it is executing is:
/opt/cmake-3.18.1/bin/cmake "--build" "<projectDir>/test/<TestName>" --target "<test_exe>"
The CMakeCache.txt & CMakeLists.txt actually live in "<projectDir>/cmake" which is why the cache cannot be loaded.
The command should be:
/opt/cmake-3.18.1/bin/cmake "--build" "<projectDir>/cmake" --target "<test_exe>"
Q2 Is there some way to tell ctest it does not need to run cmake?
My suspicion is no. Does ctest have to invoke cmake to run the tests? Does it just monitor and aggregate the output?
If I look in DartConfiguration.tcl I have:
SourceDirectory: <projectDir/cmake
BuildDirectory: <projectDir/cmake
ConfigureCommand: "/opt/cmake-3.18.1/bin/cmake" "<projectDir/cmake"
MakeCommand: /opt/cmake-3.18.1/bin/cmake --build . --config "${CTEST_CONFIGURATION_TYPE}" -- -i
I am running ctest -T test rather than ctest -T build-and-test why is ctest trying to build at all?
I've tried setting Ctest_build_command but it seems to have no effect on the DartConfiguration.tcl generated.
Some clarifications:
ctest works for me for other projects which use out of source builds
This one cannot yet use an out of source build without massive refactoring.
The source root is actually "<projectDir>"
The CMakeLists.txt lives in "<projectDir>/cmake"
Program source code lives in "<projectDir>/src"
Test scripts and code live in "<projectDir>/test"
My build worked using:
enable_testing()
I never previously had to use:
include(CTest)
include(Dart)
One or other of these creates the DartConfiguration.tcl which is just a list of "key: value" pairs. I think this is used when ctest is invoked directly which was not the case here.
I'm not entirely clear on why and what are they actually for despite reading this.
Also posted on the cmake discourse channel - https://discourse.cmake.org/t/cmake-ctest-generate-test-xml-without-rebuilding-could-not-load-cache/2025
It turns out I had some pseudo tests used to build the test driver programs via cmake --build. These were explicitly and inexplicably using CMAKE_CURRENT_DIRECTORY instead of CMAKE_SOURCE_DIR. Oops.
I have projects with unit, fuzz, and integration tests. I only want unit tests failing customer builds. I use fuzz and integration for CI. What I’d love to have is generated targets such as:
make test
make fuzz
make integration
I’ve been scouring the cmake source code all morning and haven’t found a way to generate something other than:
make test
I have to be missing something obvious. Any ideas?
As a workaround you can add a new configuration type (in addition to the default Debug, Release, RelWithDebInfo, MinSizeRel) and then qualify the test with that configuration type.
So for example:
set(CMAKE_CONFIGURATION_TYPES
Debug
DebugFuzzer
Release
RelWithDebInfo
MinSizeRel)
# ...
add_test(NAME FuzzTest COMMAND FuzzTestDriver CONFIGURATIONS DebugFuzzer)
This will let you run the fuzz tests only when you build with -DCMAKE_BUILD_TYPE=DebugFuzzer
In my CMake project I have several targets which simply run a certain set of unit tests (for example, runTestsForA, runTestsForB and runTestsForC).
I also have a target, tests, that depends on all of these unit test targets, so I can run them with a single command.
I'm using CLion is my IDE, which tries to use parallel make builds by default (which I want and am also doing on the Continuous Integration server).
However, it looks like the tests are running in parallel too now and some tests are not made for this (they use a local loopback to do some magic with sockets), which causes them to fail.. sometimes.
That is why I would like to force serial execution for some/all of the dependencies of my tests target.
Unfortunately the CMake documentation did not help me, when I was searching information on how to do this.
Which brings me to my questions: is this at all possible and how can it be done if it is?
Instead of manual tests target declaration you can use CTest tool. Use add_test command to create test targets, then CMake will automatically create tests target that will run all tests:
enable_testing()
add_test(NAME TestsForA COMMAND <command>)
add_test(NAME TestsForB COMMAND <command>)
set_tests_properties(TestsForA TestsForB PROPERTIES RUN_SERIAL TRUE)
After that you can run make tests or ctest -j8 . in your build tree. The tests will be serialized.
More information can be found at:
http://www.cmake.org/cmake/help/v3.2/command/add_test.html
http://www.cmake.org/cmake/help/v3.2/command/enable_testing.html
http://www.cmake.org/cmake/help/v3.2/command/set_tests_properties.html
http://www.cmake.org/cmake/help/v3.2/manual/cmake-properties.7.html#properties-on-tests
http://www.cmake.org/cmake/help/v3.2/manual/ctest.1.html
Perhaps it's not the best solution for controlling test execution, but, generally I believe you can use the CMake JOB_POOLS feature to restrict the amount of parallelism for some of the CMake targets in a project.
First you declare named pool(s) and the number of parallel jobs each pool supports:
set_property(GLOBAL PROPERTY JOB_POOLS two_jobs=2 ten_jobs=10)
You can then assign targets to those pools:
set_property(TARGET myexe PROPERTY JOB_POOL_COMPILE ten_jobs)