I'm working on an embedded project. As a part of this project, I have unit-tests that use gcc and Gtest. The question what is the best to approach to incorporate these unit-tests. My current implementation is that I have a different build type called unittest. I have a clause of CMAKE_BUILD_TYPE and decide which sources to use which targets to create. I see this is not a good design and this screws up multiconfiguration gnerators. What could be the elegant solution for this?
Thanks in advance for answering.
Create separate executables for testing and use ctest:
add_test to add the combination of executable+command line parameters as a test ctest runs and enable_testing() in the toplevel CMakeLists.txt.
This allows you to simply run ctest in the build dir and you can pass a configuration to test using -C command line option.
add_executable(MyTest test.cpp test_helper.cpp test_helper.h)
target_include_directories(MyTest PRIVATE .)
target_link_libraries(MyTest PRIVATE theLibToTest)
add_test(NAME NameOfTest COMMAND MyTest --gtest_repeat=1000)
enable_testing()
Running ctest from the build directory runs a test named NameOfTest. For multi configuration generators you simply specify the configuration to test with the -C command line option
ctest -C Release
Of course you can use add_test multiple times to add different test executables or the same test executable with different command line options.
Furthermore I recommend figuring out a way of storing results in a file, since this makes via the test parameters, since ctest's output probably won't do the trick. I'm not familiar enough with gtest to give advice on this.
Btw: ctest treats exit code 0 as success and any other exit code as failure, but I guess gtest produces executables that satisfy this property.
If you don't necessarily want to build the tests at the same time as the rest, you could exclude them from all, possibly adding a custom target that depends on all of the unit tests and is also excluded from all to allow building all of them at once.
You could also use a cache variable to toggle testing on and off:
# enabled via -D TEST_MY_PROJECT:BOOL=1 parameter for cmake
set(TEST_MY_PROJECT 0 CACHE BOOL "enable tests for my project")
if (TEST_MY_PROJECT)
# testing setup goes here
endif()
Related
I'm trying to use add_custom_command to generate a file during the build. The command never seemed to be run, so I made this test file.
cmake_minimum_required( VERSION 2.6 )
add_custom_command(
OUTPUT hello.txt
COMMAND touch hello.txt
DEPENDS hello.txt
)
I tried running:
cmake .
make
And hello.txt was not generated. What have I done wrong?
The add_custom_target(run ALL ... solution will work for simple cases when you only have one target you're building, but breaks down when you have multiple top level targets, e.g. app and tests.
I ran into this same problem when I was trying to package up some test data files into an object file so my unit tests wouldn't depend on anything external. I solved it using add_custom_command and some additional dependency magic with set_property.
add_custom_command(
OUTPUT testData.cpp
COMMAND reswrap
ARGS testData.src > testData.cpp
DEPENDS testData.src
)
set_property(SOURCE unit-tests.cpp APPEND PROPERTY OBJECT_DEPENDS testData.cpp)
add_executable(app main.cpp)
add_executable(tests unit-tests.cpp)
So now testData.cpp will generated before unit-tests.cpp is compiled, and any time testData.src changes. If the command you're calling is really slow you get the added bonus that when you build just the app target you won't have to wait around for that command (which only the tests executable needs) to finish.
It's not shown above, but careful application of ${PROJECT_BINARY_DIR}, ${PROJECT_SOURCE_DIR} and include_directories() will keep your source tree clean of generated files.
Add the following:
add_custom_target(run ALL
DEPENDS hello.txt)
If you're familiar with makefiles, this means:
all: run
run: hello.txt
The problem with two existing answers is that they either make the dependency global (add_custom_target(name ALL ...)), or they assign it to a specific, single file (set_property(...)) which gets obnoxious if you have many files that need it as a dependency. Instead what we want is a target that we can make a dependency of another target.
The way to do this is to use add_custom_command to define the rule, and then add_custom_target to define a new target based on that rule. Then you can add that target as a dependency of another target via add_dependencies.
# this defines the build rule for some_file
add_custom_command(
OUTPUT some_file
COMMAND ...
)
# create a target that includes some_file, this gives us a name that we can use later
add_custom_target(
some_target
DEPENDS some_file
)
# then let's suppose we're creating a library
add_library(some_library some_other_file.c)
# we can add the target as a dependency, and it will affect only this library
add_dependencies(some_library some_target)
The advantages of this approach:
some_target is not a dependency for ALL, which means you only build it when it's required by a specific target. (Whereas add_custom_target(name ALL ...) would build it unconditionally for all targets.)
Because some_target is a dependency for the library as a whole, it will get built before all of the files in that library. That means that if there are many files in the library, we don't have to do set_property on every single one of them.
If we add DEPENDS to add_custom_command then it will only get rebuilt when its inputs change. (Compare this to the approach that uses add_custom_target(name ALL ...) where the command gets run on every build regardless of whether it needs to or not.)
For more information on why things work this way, see this blog post: https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/
This question is pretty old, but even if I follow the suggested recommendations, it does not work for me (at least not every time).
I am using Android Studio and I need to call cMake to build C++ library. It works fine until I add the code to run my custom script (in fact, at the moment I try to run 'touch', as in the example above).
First of,
add_custom_command
does not work at all.
I tried
execute_process (
COMMAND touch hello.txt
)
it works, but not every time!
I tried to clean the project, remove the created file(s) manually, same thing.
Tried cMake versions:
3.10.2
3.18.1
3.22.1
when they work, they produce different results, depending on cMake version, one file or several. This is not that important as long as they work, but that's the issue.
Can somebody shed light on this mystery?
I have a cmake based (C++) project which includes tests created via add_test().
The build process is basically:
cmake
make all testsuite
make test
This has worked well for some time.
I am now trying to get my build to generate a Test.xml to be submitted as part of a pipeline build. It is not working for this project.
Q1 Can I tell cmake that I want a Test.xml to be generated when it (make test) runs the tests?
Currently I believe that a Test.xml can only be created by running ctest.
Presumably ctest aggregates the test results whereas cmake just runs them blindly?
Can someone confirm or refute this?
So I am currently trying to run ctest using:
ctest -vv -debug --output-on-failure -T test
As this does create the Test.xml file I need.
I had to add include(CTest) & include(Dart) to fix "could not find DartConfiguration.tcl".
However, even if I have previously run "make test" which guarantees the build is up to date it still tries to build and I get:
Error: could not load cache
For tests that run a compiled test (script based tests work fine).
The command it is executing is:
/opt/cmake-3.18.1/bin/cmake "--build" "<projectDir>/test/<TestName>" --target "<test_exe>"
The CMakeCache.txt & CMakeLists.txt actually live in "<projectDir>/cmake" which is why the cache cannot be loaded.
The command should be:
/opt/cmake-3.18.1/bin/cmake "--build" "<projectDir>/cmake" --target "<test_exe>"
Q2 Is there some way to tell ctest it does not need to run cmake?
My suspicion is no. Does ctest have to invoke cmake to run the tests? Does it just monitor and aggregate the output?
If I look in DartConfiguration.tcl I have:
SourceDirectory: <projectDir/cmake
BuildDirectory: <projectDir/cmake
ConfigureCommand: "/opt/cmake-3.18.1/bin/cmake" "<projectDir/cmake"
MakeCommand: /opt/cmake-3.18.1/bin/cmake --build . --config "${CTEST_CONFIGURATION_TYPE}" -- -i
I am running ctest -T test rather than ctest -T build-and-test why is ctest trying to build at all?
I've tried setting Ctest_build_command but it seems to have no effect on the DartConfiguration.tcl generated.
Some clarifications:
ctest works for me for other projects which use out of source builds
This one cannot yet use an out of source build without massive refactoring.
The source root is actually "<projectDir>"
The CMakeLists.txt lives in "<projectDir>/cmake"
Program source code lives in "<projectDir>/src"
Test scripts and code live in "<projectDir>/test"
My build worked using:
enable_testing()
I never previously had to use:
include(CTest)
include(Dart)
One or other of these creates the DartConfiguration.tcl which is just a list of "key: value" pairs. I think this is used when ctest is invoked directly which was not the case here.
I'm not entirely clear on why and what are they actually for despite reading this.
Also posted on the cmake discourse channel - https://discourse.cmake.org/t/cmake-ctest-generate-test-xml-without-rebuilding-could-not-load-cache/2025
It turns out I had some pseudo tests used to build the test driver programs via cmake --build. These were explicitly and inexplicably using CMAKE_CURRENT_DIRECTORY instead of CMAKE_SOURCE_DIR. Oops.
I'm trying CTest in CMake in order to automatically run some of my tests using make test target. The problem is CMake does not "understand" that the test I'm willing to run has to be built since it is part of the project.
So I'm looking for a way to explicitly specify this dependency.
It is arguably a bug in CMake (previously tracked here) that this doesn't work out of the box. A workaround is to do the following:
add_test(TestName ExeName)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND}
DEPENDS ExeName)
Then you can run make check and it will compile and run the test. If you have several tests, then you would have to use DEPENDS exe1 exe2 exe3 ... in the above line.
There is actually a way to use make test. You need to define the build of the test executable as one of the tests and then add dependencies between the tests. That is:
ADD_TEST(ctest_build_test_code
"${CMAKE_COMMAND}" --build ${CMAKE_BINARY_DIR} --target test_code)
ADD_TEST(ctest_run_test_code test_code)
SET_TESTS_PROPERTIES(ctest_run_test_code
PROPERTIES DEPENDS ctest_build_test_code)
I use a variant of richq's answer. In the top-level CMakeLists.txt, I add a custom target, build_and_test, for building and running all tests:
find_package(GTest)
if (GTEST_FOUND)
enable_testing()
add_custom_target(build_and_test ${CMAKE_CTEST_COMMAND} -V)
add_subdirectory(test)
endif()
In the various sub-project CMakeLists.txt files under test/, I add each test executable as a dependency of build_and_test:
include_directories(${CMAKE_SOURCE_DIR}/src/proj1)
include_directories(${GTEST_INCLUDE_DIRS})
add_executable(proj1_test proj1_test.cpp)
target_link_libraries(proj1_test ${GTEST_BOTH_LIBRARIES} pthread)
add_test(proj1_test proj1_test)
add_dependencies(build_and_test proj1_test)
With this approach, I just need to make build_and_test instead of make test (or make all test), and it has the benefit of only building test code (and its dependencies). It's a shame I can't use the target name test. In my case, it's not so bad because I have a top-level script that does out-of-tree debug and release (and cross-compiled) builds by calling cmake and then make, and it translates test into build_and_test.
Obviously, the GTest stuff isn't required. I just happen to use/like Google Test, and wanted to share a complete example of using it with CMake/CTest. IMHO, this approach also has the benefit of allowing me to use ctest -V, which shows the Google Test output while the tests run:
1: Running main() from gtest_main.cc
1: [==========] Running 1 test from 1 test case.
1: [----------] Global test environment set-up.
1: [----------] 1 test from proj1
1: [ RUN ] proj1.dummy
1: [ OK ] proj1.dummy (0 ms)
1: [----------] 1 test from proj1 (1 ms total)
1:
1: [----------] Global test environment tear-down
1: [==========] 1 test from 1 test case ran. (1 ms total)
1: [ PASSED ] 1 test.
1/2 Test #1: proj1_test ....................... Passed 0.03 sec
If you are using CMake >= 3.7, then the recommended approach is to use fixtures:
add_executable(test test.cpp)
add_test(test_build
"${CMAKE_COMMAND}"
--build "${CMAKE_BINARY_DIR}"
--config "$<CONFIG>"
--target test
)
set_tests_properties(test_build PROPERTIES FIXTURES_SETUP test_fixture)
add_test(test test)
set_tests_properties(test PROPERTIES FIXTURES_REQUIRED test_fixture)
This does the following:
Adds a test executable target built from test.cpp
Adds a test_build "test" that runs Cmake to build target test
Marks the test_build test to be a setup task of fixture test_fixture
Add a test test that just runs the test executable
Marks the test test to need fixture test_fixture.
So, every time test test is to be run, it first runs test test_build, which builds the necessary executable.
If you are trying to emulate make check, you may find this wiki entry usefull :
http://www.cmake.org/Wiki/CMakeEmulateMakeCheck
I have just checked that is does what it says with success (CMake 2.8.10).
Save yourself the headache:
make all test
Works out of the box for me and will build dependencies before running the test. Given how simple this is, it almost makes the native make test functionality convenient because it gives you the option of running the last compiling tests even if your code is broken.
For CMake 3.10 or later, another option is to use the TEST_INCLUDE_FILES directory property to set up a script that triggers a build before a test is run. In your outermost CMakeLists.txt add the following code:
set_property(DIRECTORY APPEND
PROPERTY TEST_INCLUDE_FILES "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake"
"execute_process(COMMAND \"${CMAKE_COMMAND}\""
" --build \"${CMAKE_BINARY_DIR}\""
" --config \"\$ENV{CMAKE_CONFIG_TYPE}\")")
The actual test configuration is passed through to the build via the environment variable CMAKE_CONFIG_TYPE. Optionally you can add a --target option to only build targets required by the test.
This is what I hammered out and have been using:
set(${PROJECT_NAME}_TESTS a b c)
enable_testing()
add_custom_target(all_tests)
foreach(test ${${PROJECT_NAME}_TESTS})
add_executable(${test} EXCLUDE_FROM_ALL ${test}.cc)
add_test(NAME ${test} COMMAND $<TARGET_FILE:${test}>)
add_dependencies(all_tests ${test})
endforeach(test)
build_command(CTEST_CUSTOM_PRE_TEST TARGET all_tests)
string(CONFIGURE \"#CTEST_CUSTOM_PRE_TEST#\" CTEST_CUSTOM_PRE_TEST_QUOTED ESCAPE_QUOTES)
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake" "set(CTEST_CUSTOM_PRE_TEST ${CTEST_CUSTOM_PRE_TEST_QUOTED})" "\n")
YMMV
Derrick's answer, simplified and commented:
# It is impossible to make target "test" depend on "all":
# https://gitlab.kitware.com/cmake/cmake/-/issues/8774
# Set a magic variable in a magic file that tells ctest
# to invoke the generator once before running the tests:
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake"
"set(CTEST_CUSTOM_PRE_TEST ${CMAKE_MAKE_PROGRAM})\n"
)
It is not perfectly correct, as it does not solve the concurrency problem of running ninja all test, in case anyone does that. On the contrary, because now, you have two ninja processes.
(Ftr, I also shared this solution here.)
All above answers are perfect. But actually CMake use CTest as its testing tools, so the standard method (I think it is) to do the mission is:
enable_testing ()
add_test (TestName TestCommand)
add_test (TestName2 AnotherTestCommand)
Then run cmake and make to build the targets. After that, you can either run make test, or just run
ctest
you will get the result. This is tested under CMake 2.8 .
Check details at: http://cmake.org/Wiki/CMake/Testing_With_CTest#Simple_Testing
All the answers are good, but they imply a breach of tradition to run a test by command make test. I've done this trick:
add_test(NAME <mytest>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMAND sh -c "make <mytarget>; $<TARGET_FILE:<mytarget>>")
This means that the test consists of building (optionally) and running of executable target.
To register tests under CMake, we need
enable_testing()
or
include(CTest)
and then for each single test (name fooTest, executable foo)
add_executable(foo <foo_sources>)
add_test(fooTest foo)
Tests can then be run with the command ctest.
Additionally, we can run tests with the command make check, provided we add once
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND})
and for each test we extend the above by a keyword EXCLUDE_FROM_ALL and a command add_dependencies:
add_executable(foo EXCLUDE_FROM_ALL <foo_sources>)
add_test(fooTest foo)
add_dependencies(check foo)
Ideally, this would make make check an alias of ctest. It does not so for at least two reasons:
(1) make check is flawed because it does not pass options to ctest [2]. In particular, ctest -j4 will run 4 tests in parallel, whereas make -j4 check will work in one thread on target check, and the other three threads will remain idle.
(2) ctest is flawed [3,4] because all tests are build under the all target, i.e. along with the main application. This may be desired behavior in some situations, but in other situations it ought to be possible to postpone the build until the tests are to be run.
Does this correctly summarize the current state of affairs?
Is there any way around (to eat the cake and have it)?
[1] https://cmake.org/Wiki/CMakeEmulateMakeCheck
[2] http://comments.gmane.org/gmane.comp.programming.tools.cmake.user/47300
[3] CMake & CTest : make test doesn't build tests
[4] http://public.kitware.com/Bug/view.php?id=8774
First, let me remark that ctest and make test are only simple command line tools, for simple testing tasks. If you want a tool for serious testing, use CDash, Buildbot, Jenkins or whatever.
Concerning the flaws of CTest: It is intentional, that the call for CTest does not build the tests. It is a bad idea in several scenarios:
Compiling tests can take more resources then running the tests itself. This might be true with respect to memory consumption, read/writes to the hard disk or compilation time. So compiling and linking in parallel might be bad, but executing the tests in parallel might be beneficial.
How to handle compilation or linking failure? Report it as failing? Report is as not compiling? Continuing with compiling the other tests or aborting immediately?
Autotools did it the way you want it and people got used to it. But why should it be a unit? Why not having two commands? What's the benefit of mixing two tasks and making it more difficult for project with special needs?
I came to the conclusion, to create a target build-tests or similar, and follow the decision made by the CMake developers to decouple building test and executing tests. Then I can decide whether I want parallel builds, how to treat compilation failures (e.g., passing -k to make) and so on.
The only downside is, that this target is only present in the top level directory and cannot be used in sub-directories.
To get such a target built-in by CMake would be a good feature request. Ranting on SO does no good.
CTest is not flawed at all, but the way you use CMake and CTest seems "flawed". The invocation of the command line interface (CLI) tool ctest is in general not related to the invokation of a CMake build targets (with the exception of the target test).
In my opinion the custom check target solution described in the CMake Wiki should not be used, since it changes the default behavior of CMake and is not configurable.
Instead the following approach using the built-in option BUILD_TESTING should be used:
include(CTest)
if(BUILD_TESTING)
find_package(GTest MODULE REQUIRED)
add_executable(example_test example_test.cpp)
target_link_libraries(
example_test
PRIVATE
GTest::GTest
GTest::Main
)
add_test(NAME example_test COMMAND example_test)
endif()
include(CTest) defines in the option BUILD_TESTING, which allows to control whether to build all tests of the project or not.
Quote from the official documentation:
CMake will generate tests only if the enable_testing() command has been invoked. The CTest module invokes the command automatically when the BUILD_TESTING option is ON.
The above can be used on the CLI as follows:
Create tests (default):
cmake -Hexample-testing -B_builds/example-testing/release -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release
cmake --build _builds/example-testing/release --config Release
In this case the commands cd _builds/example-testing/release and ctest / cmake --build . --target test build and run the test(s).
Do not create tests, setting -DBUILD_TESTING=OFF:
cmake -Hexample-testing -B_builds/example-testing/release-no-tests -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF
cmake --build _builds/example-testing/release-no-tests --config Release
In this case the commands cd _builds/example-testing/release-no-tests and ctest run no test(s), since no test(s) have been built.
The command cmake --build . --target test fails since it has not been created during configure phase of CMake.
We are only scratching the surface here. Refer to ctest --help, e.g. there are a lot of --build-<...> options that allow finer control regarding testing/building, though I have not any experience with that.
I highly recommend reading the following:
CTest - CMake Documentation
CTest Commands - CMake Documentation
ctest(1) - CMake Documentation
add_test - CMake Documentation
If you really want to enable building of tests, but via a separate target that is not invoked by default and run the test not via CTest but directly you can do the following:
include(CTest)
if(BUILD_TESTING)
find_package(GTest MODULE REQUIRED)
option(
BUILD_TESTING_EXCLUDE_FROM_ALL
"Do not build the testing tree together with the default build target."
OFF
)
if(BUILD_TESTING_EXCLUDE_FROM_ALL)
set(add_executable_args_for_test EXCLUDE_FROM_ALL)
endif()
# The "build_test" target is used to build all test executables.
add_custom_target(
build_test
# Workaround for printing the COMMENT, it does not work without a NOOP
# COMMAND.
COMMAND ${CMAKE_COMMAND} -E echo
COMMENT "Building tests..."
VERBATIM
)
add_executable(example_test ${add_executable_args_for_test} example_test.cpp)
target_link_libraries(
example_test
PRIVATE
GTest::GTest
GTest::Main
)
add_test(NAME example_test COMMAND example_test)
add_dependencies(build_test example_test)
# The "check" target is used to build AND run all test executables.
add_custom_target(
check
# Either invoke the test(s) indirectly via "CTest" (commented) or directly.
# COMMAND ${CMAKE_CTEST_COMMAND}
COMMAND example_test
COMMENT "Building and running test..."
VERBATIM
)
# Alternative to the COMMAND in the add_custom_target. Leads to the same
# behavior as calling "CTest" directly.
# add_custom_command(
# TARGET check
# COMMAND ${CMAKE_COMMAND} ARGS --build ${CMAKE_BINARY_DIR} --target test
# VERBATIM
# )
add_dependencies(check build_test)
endif()
Note that the above code does not invoke CTest or the target test in order to run the test, but the test directly.
Please read the comments and the commented code for alternative approaches using CTest that are similar to the approach described in the question.
It's easy to enhance the above code to support more than one test executable.
IMHO, Kitware should remove the entire CMake Wiki, since the Wiki contains almost only out-of-date information for CMake versions < 3.0. Most information in it cannot be considered as Modern CMake.
In my CMake project I have several targets which simply run a certain set of unit tests (for example, runTestsForA, runTestsForB and runTestsForC).
I also have a target, tests, that depends on all of these unit test targets, so I can run them with a single command.
I'm using CLion is my IDE, which tries to use parallel make builds by default (which I want and am also doing on the Continuous Integration server).
However, it looks like the tests are running in parallel too now and some tests are not made for this (they use a local loopback to do some magic with sockets), which causes them to fail.. sometimes.
That is why I would like to force serial execution for some/all of the dependencies of my tests target.
Unfortunately the CMake documentation did not help me, when I was searching information on how to do this.
Which brings me to my questions: is this at all possible and how can it be done if it is?
Instead of manual tests target declaration you can use CTest tool. Use add_test command to create test targets, then CMake will automatically create tests target that will run all tests:
enable_testing()
add_test(NAME TestsForA COMMAND <command>)
add_test(NAME TestsForB COMMAND <command>)
set_tests_properties(TestsForA TestsForB PROPERTIES RUN_SERIAL TRUE)
After that you can run make tests or ctest -j8 . in your build tree. The tests will be serialized.
More information can be found at:
http://www.cmake.org/cmake/help/v3.2/command/add_test.html
http://www.cmake.org/cmake/help/v3.2/command/enable_testing.html
http://www.cmake.org/cmake/help/v3.2/command/set_tests_properties.html
http://www.cmake.org/cmake/help/v3.2/manual/cmake-properties.7.html#properties-on-tests
http://www.cmake.org/cmake/help/v3.2/manual/ctest.1.html
Perhaps it's not the best solution for controlling test execution, but, generally I believe you can use the CMake JOB_POOLS feature to restrict the amount of parallelism for some of the CMake targets in a project.
First you declare named pool(s) and the number of parallel jobs each pool supports:
set_property(GLOBAL PROPERTY JOB_POOLS two_jobs=2 ten_jobs=10)
You can then assign targets to those pools:
set_property(TARGET myexe PROPERTY JOB_POOL_COMPILE ten_jobs)