In my CMake project I have several targets which simply run a certain set of unit tests (for example, runTestsForA, runTestsForB and runTestsForC).
I also have a target, tests, that depends on all of these unit test targets, so I can run them with a single command.
I'm using CLion is my IDE, which tries to use parallel make builds by default (which I want and am also doing on the Continuous Integration server).
However, it looks like the tests are running in parallel too now and some tests are not made for this (they use a local loopback to do some magic with sockets), which causes them to fail.. sometimes.
That is why I would like to force serial execution for some/all of the dependencies of my tests target.
Unfortunately the CMake documentation did not help me, when I was searching information on how to do this.
Which brings me to my questions: is this at all possible and how can it be done if it is?
Instead of manual tests target declaration you can use CTest tool. Use add_test command to create test targets, then CMake will automatically create tests target that will run all tests:
enable_testing()
add_test(NAME TestsForA COMMAND <command>)
add_test(NAME TestsForB COMMAND <command>)
set_tests_properties(TestsForA TestsForB PROPERTIES RUN_SERIAL TRUE)
After that you can run make tests or ctest -j8 . in your build tree. The tests will be serialized.
More information can be found at:
http://www.cmake.org/cmake/help/v3.2/command/add_test.html
http://www.cmake.org/cmake/help/v3.2/command/enable_testing.html
http://www.cmake.org/cmake/help/v3.2/command/set_tests_properties.html
http://www.cmake.org/cmake/help/v3.2/manual/cmake-properties.7.html#properties-on-tests
http://www.cmake.org/cmake/help/v3.2/manual/ctest.1.html
Perhaps it's not the best solution for controlling test execution, but, generally I believe you can use the CMake JOB_POOLS feature to restrict the amount of parallelism for some of the CMake targets in a project.
First you declare named pool(s) and the number of parallel jobs each pool supports:
set_property(GLOBAL PROPERTY JOB_POOLS two_jobs=2 ten_jobs=10)
You can then assign targets to those pools:
set_property(TARGET myexe PROPERTY JOB_POOL_COMPILE ten_jobs)
Related
I'm working on an embedded project. As a part of this project, I have unit-tests that use gcc and Gtest. The question what is the best to approach to incorporate these unit-tests. My current implementation is that I have a different build type called unittest. I have a clause of CMAKE_BUILD_TYPE and decide which sources to use which targets to create. I see this is not a good design and this screws up multiconfiguration gnerators. What could be the elegant solution for this?
Thanks in advance for answering.
Create separate executables for testing and use ctest:
add_test to add the combination of executable+command line parameters as a test ctest runs and enable_testing() in the toplevel CMakeLists.txt.
This allows you to simply run ctest in the build dir and you can pass a configuration to test using -C command line option.
add_executable(MyTest test.cpp test_helper.cpp test_helper.h)
target_include_directories(MyTest PRIVATE .)
target_link_libraries(MyTest PRIVATE theLibToTest)
add_test(NAME NameOfTest COMMAND MyTest --gtest_repeat=1000)
enable_testing()
Running ctest from the build directory runs a test named NameOfTest. For multi configuration generators you simply specify the configuration to test with the -C command line option
ctest -C Release
Of course you can use add_test multiple times to add different test executables or the same test executable with different command line options.
Furthermore I recommend figuring out a way of storing results in a file, since this makes via the test parameters, since ctest's output probably won't do the trick. I'm not familiar enough with gtest to give advice on this.
Btw: ctest treats exit code 0 as success and any other exit code as failure, but I guess gtest produces executables that satisfy this property.
If you don't necessarily want to build the tests at the same time as the rest, you could exclude them from all, possibly adding a custom target that depends on all of the unit tests and is also excluded from all to allow building all of them at once.
You could also use a cache variable to toggle testing on and off:
# enabled via -D TEST_MY_PROJECT:BOOL=1 parameter for cmake
set(TEST_MY_PROJECT 0 CACHE BOOL "enable tests for my project")
if (TEST_MY_PROJECT)
# testing setup goes here
endif()
I'm trying CTest in CMake in order to automatically run some of my tests using make test target. The problem is CMake does not "understand" that the test I'm willing to run has to be built since it is part of the project.
So I'm looking for a way to explicitly specify this dependency.
It is arguably a bug in CMake (previously tracked here) that this doesn't work out of the box. A workaround is to do the following:
add_test(TestName ExeName)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND}
DEPENDS ExeName)
Then you can run make check and it will compile and run the test. If you have several tests, then you would have to use DEPENDS exe1 exe2 exe3 ... in the above line.
There is actually a way to use make test. You need to define the build of the test executable as one of the tests and then add dependencies between the tests. That is:
ADD_TEST(ctest_build_test_code
"${CMAKE_COMMAND}" --build ${CMAKE_BINARY_DIR} --target test_code)
ADD_TEST(ctest_run_test_code test_code)
SET_TESTS_PROPERTIES(ctest_run_test_code
PROPERTIES DEPENDS ctest_build_test_code)
I use a variant of richq's answer. In the top-level CMakeLists.txt, I add a custom target, build_and_test, for building and running all tests:
find_package(GTest)
if (GTEST_FOUND)
enable_testing()
add_custom_target(build_and_test ${CMAKE_CTEST_COMMAND} -V)
add_subdirectory(test)
endif()
In the various sub-project CMakeLists.txt files under test/, I add each test executable as a dependency of build_and_test:
include_directories(${CMAKE_SOURCE_DIR}/src/proj1)
include_directories(${GTEST_INCLUDE_DIRS})
add_executable(proj1_test proj1_test.cpp)
target_link_libraries(proj1_test ${GTEST_BOTH_LIBRARIES} pthread)
add_test(proj1_test proj1_test)
add_dependencies(build_and_test proj1_test)
With this approach, I just need to make build_and_test instead of make test (or make all test), and it has the benefit of only building test code (and its dependencies). It's a shame I can't use the target name test. In my case, it's not so bad because I have a top-level script that does out-of-tree debug and release (and cross-compiled) builds by calling cmake and then make, and it translates test into build_and_test.
Obviously, the GTest stuff isn't required. I just happen to use/like Google Test, and wanted to share a complete example of using it with CMake/CTest. IMHO, this approach also has the benefit of allowing me to use ctest -V, which shows the Google Test output while the tests run:
1: Running main() from gtest_main.cc
1: [==========] Running 1 test from 1 test case.
1: [----------] Global test environment set-up.
1: [----------] 1 test from proj1
1: [ RUN ] proj1.dummy
1: [ OK ] proj1.dummy (0 ms)
1: [----------] 1 test from proj1 (1 ms total)
1:
1: [----------] Global test environment tear-down
1: [==========] 1 test from 1 test case ran. (1 ms total)
1: [ PASSED ] 1 test.
1/2 Test #1: proj1_test ....................... Passed 0.03 sec
If you are using CMake >= 3.7, then the recommended approach is to use fixtures:
add_executable(test test.cpp)
add_test(test_build
"${CMAKE_COMMAND}"
--build "${CMAKE_BINARY_DIR}"
--config "$<CONFIG>"
--target test
)
set_tests_properties(test_build PROPERTIES FIXTURES_SETUP test_fixture)
add_test(test test)
set_tests_properties(test PROPERTIES FIXTURES_REQUIRED test_fixture)
This does the following:
Adds a test executable target built from test.cpp
Adds a test_build "test" that runs Cmake to build target test
Marks the test_build test to be a setup task of fixture test_fixture
Add a test test that just runs the test executable
Marks the test test to need fixture test_fixture.
So, every time test test is to be run, it first runs test test_build, which builds the necessary executable.
If you are trying to emulate make check, you may find this wiki entry usefull :
http://www.cmake.org/Wiki/CMakeEmulateMakeCheck
I have just checked that is does what it says with success (CMake 2.8.10).
Save yourself the headache:
make all test
Works out of the box for me and will build dependencies before running the test. Given how simple this is, it almost makes the native make test functionality convenient because it gives you the option of running the last compiling tests even if your code is broken.
For CMake 3.10 or later, another option is to use the TEST_INCLUDE_FILES directory property to set up a script that triggers a build before a test is run. In your outermost CMakeLists.txt add the following code:
set_property(DIRECTORY APPEND
PROPERTY TEST_INCLUDE_FILES "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake"
"execute_process(COMMAND \"${CMAKE_COMMAND}\""
" --build \"${CMAKE_BINARY_DIR}\""
" --config \"\$ENV{CMAKE_CONFIG_TYPE}\")")
The actual test configuration is passed through to the build via the environment variable CMAKE_CONFIG_TYPE. Optionally you can add a --target option to only build targets required by the test.
This is what I hammered out and have been using:
set(${PROJECT_NAME}_TESTS a b c)
enable_testing()
add_custom_target(all_tests)
foreach(test ${${PROJECT_NAME}_TESTS})
add_executable(${test} EXCLUDE_FROM_ALL ${test}.cc)
add_test(NAME ${test} COMMAND $<TARGET_FILE:${test}>)
add_dependencies(all_tests ${test})
endforeach(test)
build_command(CTEST_CUSTOM_PRE_TEST TARGET all_tests)
string(CONFIGURE \"#CTEST_CUSTOM_PRE_TEST#\" CTEST_CUSTOM_PRE_TEST_QUOTED ESCAPE_QUOTES)
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake" "set(CTEST_CUSTOM_PRE_TEST ${CTEST_CUSTOM_PRE_TEST_QUOTED})" "\n")
YMMV
Derrick's answer, simplified and commented:
# It is impossible to make target "test" depend on "all":
# https://gitlab.kitware.com/cmake/cmake/-/issues/8774
# Set a magic variable in a magic file that tells ctest
# to invoke the generator once before running the tests:
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake"
"set(CTEST_CUSTOM_PRE_TEST ${CMAKE_MAKE_PROGRAM})\n"
)
It is not perfectly correct, as it does not solve the concurrency problem of running ninja all test, in case anyone does that. On the contrary, because now, you have two ninja processes.
(Ftr, I also shared this solution here.)
All above answers are perfect. But actually CMake use CTest as its testing tools, so the standard method (I think it is) to do the mission is:
enable_testing ()
add_test (TestName TestCommand)
add_test (TestName2 AnotherTestCommand)
Then run cmake and make to build the targets. After that, you can either run make test, or just run
ctest
you will get the result. This is tested under CMake 2.8 .
Check details at: http://cmake.org/Wiki/CMake/Testing_With_CTest#Simple_Testing
All the answers are good, but they imply a breach of tradition to run a test by command make test. I've done this trick:
add_test(NAME <mytest>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMAND sh -c "make <mytarget>; $<TARGET_FILE:<mytarget>>")
This means that the test consists of building (optionally) and running of executable target.
To register tests under CMake, we need
enable_testing()
or
include(CTest)
and then for each single test (name fooTest, executable foo)
add_executable(foo <foo_sources>)
add_test(fooTest foo)
Tests can then be run with the command ctest.
Additionally, we can run tests with the command make check, provided we add once
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND})
and for each test we extend the above by a keyword EXCLUDE_FROM_ALL and a command add_dependencies:
add_executable(foo EXCLUDE_FROM_ALL <foo_sources>)
add_test(fooTest foo)
add_dependencies(check foo)
Ideally, this would make make check an alias of ctest. It does not so for at least two reasons:
(1) make check is flawed because it does not pass options to ctest [2]. In particular, ctest -j4 will run 4 tests in parallel, whereas make -j4 check will work in one thread on target check, and the other three threads will remain idle.
(2) ctest is flawed [3,4] because all tests are build under the all target, i.e. along with the main application. This may be desired behavior in some situations, but in other situations it ought to be possible to postpone the build until the tests are to be run.
Does this correctly summarize the current state of affairs?
Is there any way around (to eat the cake and have it)?
[1] https://cmake.org/Wiki/CMakeEmulateMakeCheck
[2] http://comments.gmane.org/gmane.comp.programming.tools.cmake.user/47300
[3] CMake & CTest : make test doesn't build tests
[4] http://public.kitware.com/Bug/view.php?id=8774
First, let me remark that ctest and make test are only simple command line tools, for simple testing tasks. If you want a tool for serious testing, use CDash, Buildbot, Jenkins or whatever.
Concerning the flaws of CTest: It is intentional, that the call for CTest does not build the tests. It is a bad idea in several scenarios:
Compiling tests can take more resources then running the tests itself. This might be true with respect to memory consumption, read/writes to the hard disk or compilation time. So compiling and linking in parallel might be bad, but executing the tests in parallel might be beneficial.
How to handle compilation or linking failure? Report it as failing? Report is as not compiling? Continuing with compiling the other tests or aborting immediately?
Autotools did it the way you want it and people got used to it. But why should it be a unit? Why not having two commands? What's the benefit of mixing two tasks and making it more difficult for project with special needs?
I came to the conclusion, to create a target build-tests or similar, and follow the decision made by the CMake developers to decouple building test and executing tests. Then I can decide whether I want parallel builds, how to treat compilation failures (e.g., passing -k to make) and so on.
The only downside is, that this target is only present in the top level directory and cannot be used in sub-directories.
To get such a target built-in by CMake would be a good feature request. Ranting on SO does no good.
CTest is not flawed at all, but the way you use CMake and CTest seems "flawed". The invocation of the command line interface (CLI) tool ctest is in general not related to the invokation of a CMake build targets (with the exception of the target test).
In my opinion the custom check target solution described in the CMake Wiki should not be used, since it changes the default behavior of CMake and is not configurable.
Instead the following approach using the built-in option BUILD_TESTING should be used:
include(CTest)
if(BUILD_TESTING)
find_package(GTest MODULE REQUIRED)
add_executable(example_test example_test.cpp)
target_link_libraries(
example_test
PRIVATE
GTest::GTest
GTest::Main
)
add_test(NAME example_test COMMAND example_test)
endif()
include(CTest) defines in the option BUILD_TESTING, which allows to control whether to build all tests of the project or not.
Quote from the official documentation:
CMake will generate tests only if the enable_testing() command has been invoked. The CTest module invokes the command automatically when the BUILD_TESTING option is ON.
The above can be used on the CLI as follows:
Create tests (default):
cmake -Hexample-testing -B_builds/example-testing/release -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release
cmake --build _builds/example-testing/release --config Release
In this case the commands cd _builds/example-testing/release and ctest / cmake --build . --target test build and run the test(s).
Do not create tests, setting -DBUILD_TESTING=OFF:
cmake -Hexample-testing -B_builds/example-testing/release-no-tests -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF
cmake --build _builds/example-testing/release-no-tests --config Release
In this case the commands cd _builds/example-testing/release-no-tests and ctest run no test(s), since no test(s) have been built.
The command cmake --build . --target test fails since it has not been created during configure phase of CMake.
We are only scratching the surface here. Refer to ctest --help, e.g. there are a lot of --build-<...> options that allow finer control regarding testing/building, though I have not any experience with that.
I highly recommend reading the following:
CTest - CMake Documentation
CTest Commands - CMake Documentation
ctest(1) - CMake Documentation
add_test - CMake Documentation
If you really want to enable building of tests, but via a separate target that is not invoked by default and run the test not via CTest but directly you can do the following:
include(CTest)
if(BUILD_TESTING)
find_package(GTest MODULE REQUIRED)
option(
BUILD_TESTING_EXCLUDE_FROM_ALL
"Do not build the testing tree together with the default build target."
OFF
)
if(BUILD_TESTING_EXCLUDE_FROM_ALL)
set(add_executable_args_for_test EXCLUDE_FROM_ALL)
endif()
# The "build_test" target is used to build all test executables.
add_custom_target(
build_test
# Workaround for printing the COMMENT, it does not work without a NOOP
# COMMAND.
COMMAND ${CMAKE_COMMAND} -E echo
COMMENT "Building tests..."
VERBATIM
)
add_executable(example_test ${add_executable_args_for_test} example_test.cpp)
target_link_libraries(
example_test
PRIVATE
GTest::GTest
GTest::Main
)
add_test(NAME example_test COMMAND example_test)
add_dependencies(build_test example_test)
# The "check" target is used to build AND run all test executables.
add_custom_target(
check
# Either invoke the test(s) indirectly via "CTest" (commented) or directly.
# COMMAND ${CMAKE_CTEST_COMMAND}
COMMAND example_test
COMMENT "Building and running test..."
VERBATIM
)
# Alternative to the COMMAND in the add_custom_target. Leads to the same
# behavior as calling "CTest" directly.
# add_custom_command(
# TARGET check
# COMMAND ${CMAKE_COMMAND} ARGS --build ${CMAKE_BINARY_DIR} --target test
# VERBATIM
# )
add_dependencies(check build_test)
endif()
Note that the above code does not invoke CTest or the target test in order to run the test, but the test directly.
Please read the comments and the commented code for alternative approaches using CTest that are similar to the approach described in the question.
It's easy to enhance the above code to support more than one test executable.
IMHO, Kitware should remove the entire CMake Wiki, since the Wiki contains almost only out-of-date information for CMake versions < 3.0. Most information in it cannot be considered as Modern CMake.
In my top level CMakeLists.txt I include some external libraries. If I enable testing for my project and the CMakeLists of the external library also calls enable_testing(), those tests are added to my own projects tests, which then cannot be run because the tests of the lib are not built.
I don't want the tests to be built, and I don't want them to clutter my own projects tests. How can I do that?
In my CMakeLists.txt:
add_subdirectory(some_ext_lib EXCLUDE_FROM_ALL) ' some_ext_libs tests should not be built nor run
enable_testing()
add_subdirectory(my_own_stuff) ' here the tests should be run
[...]
When you use add_subdirectory, some_ext_lib is no longer external. You have embedded it into your cmake build. If this is what you want to do, then the simplest way might be to add a variable and conditional logic to control building of the tests in some_ext_lib. For example, use set(BUILD_TESTS_some_ext_lib 0) before the add_subdirectory command, and then modify the CMakeLists.txt files for some_ext_lib where the tests are inside if (BUILD_TESTS_some_ext_lib) logic.
Note that for simplicity one should generally avoid embedding a top-level CMakeLists.txt of one project in another project. That is, for any given cmake project, there should be only one CMakeLists.txt that makes the find_package calls and defines the initial option values. Now, with any generalization there are valid exceptions, but when you are just starting out,
I think you should master the simple ways first.
If some_ext_lib is really external, then you should have it install its cmake config file that other projects will use with find_package(some_ext_lib) commands. But this requires a little more sophistication with cmake.
I ran into this problem today and discovered a (slightly dirty) trick to exclude the subproject tests. This is assuming that your some_ext_lib CMake file has a project(some_ext_lib) directive.
add_subdirectory(some_ext_lib EXCLUDE_FROM_ALL)
add_custom_target(exclude_some_ext_lib_tests ALL
COMMAND rm -f "${some_ext_lib_BINARY_DIR}/CTestTestfile.cmake")
This effectively tells CMake that there are no tests to be executed in some_ext_lib. The advantage of this approach is that it requires no modifications to the some_ext_lib sources.
I have defined a custom target in cmake. I now want to ensure that this target is only build when the cmake target test was executed. How can I achieve this.
Lets say that I have a target make coverage which should depend on the target make test to be called before, or call make test before executing.
How can I define this behavior in cmake?
Here my code that did not work as expected. I want to achive that make coverage depend that make test has to be called before.
ADD_CUSTOM_TARGET(
coverage COMMAND /bin/bash ${LIBPIPE_BINARY_DIR}/cmake/scripts/coverage.sh
DEPENDS test
)
You cannot add a "DEPENDS test" clause. The pre-defined/built-in targets in CMake (all, install, package, test, clean) are not available as actual targets in a CMakeLists.txt file. So you cannot express a dependency on a built-in target.
There is an outstanding feature request in the CMake bug tracker for this feature, but it is not yet implemented. See http://public.kitware.com/Bug/view.php?id=8438
However, you could change your command for your custom target, though, such that it calls "make test" first, and then runs your coverage command.
The CMake FAQ states that the add_custom_command/add_custom_target commands, which define custom targets, have a DEPENDS parameter.
Edit
This will not work for built-in target test because of following feature request.
But you always can create custom target check or whatever as suggested in CMake FAQ