Under cmake, the following commands in a CMakeList.txt
enable_testing()
add_test(<test_name> <test_command>)
suffice to create a test that can be executed through the shell command ctest.
Unfortunately, the cmake docs give not the least indication of what constitutes a valid test executable (<test_command>). Usually, test executables are generated using a framework like google-test. It's a bit involved, but there are good examples in the web that show how to make tests under google-test under cmake.
Now I want to extend my use of cmake/ctest to functional test scripts that need not to be compiled, and therefore cannot be run under google-test. Thence my question: what constitutes a valid hand-written test executable to be directly activated through add_test:
Shall my test executable generate output? Shall it write to stdout or stderr?
Shall my test executable return certain values to indicate success or failure?
Or where in the cmake docs can I find the answer?
Your executable should return zero if the test passes and nonzero if not. Outputs are ignored.
The documentation for add_test doesn't mention this.
Related
I'm working on an embedded project. As a part of this project, I have unit-tests that use gcc and Gtest. The question what is the best to approach to incorporate these unit-tests. My current implementation is that I have a different build type called unittest. I have a clause of CMAKE_BUILD_TYPE and decide which sources to use which targets to create. I see this is not a good design and this screws up multiconfiguration gnerators. What could be the elegant solution for this?
Thanks in advance for answering.
Create separate executables for testing and use ctest:
add_test to add the combination of executable+command line parameters as a test ctest runs and enable_testing() in the toplevel CMakeLists.txt.
This allows you to simply run ctest in the build dir and you can pass a configuration to test using -C command line option.
add_executable(MyTest test.cpp test_helper.cpp test_helper.h)
target_include_directories(MyTest PRIVATE .)
target_link_libraries(MyTest PRIVATE theLibToTest)
add_test(NAME NameOfTest COMMAND MyTest --gtest_repeat=1000)
enable_testing()
Running ctest from the build directory runs a test named NameOfTest. For multi configuration generators you simply specify the configuration to test with the -C command line option
ctest -C Release
Of course you can use add_test multiple times to add different test executables or the same test executable with different command line options.
Furthermore I recommend figuring out a way of storing results in a file, since this makes via the test parameters, since ctest's output probably won't do the trick. I'm not familiar enough with gtest to give advice on this.
Btw: ctest treats exit code 0 as success and any other exit code as failure, but I guess gtest produces executables that satisfy this property.
If you don't necessarily want to build the tests at the same time as the rest, you could exclude them from all, possibly adding a custom target that depends on all of the unit tests and is also excluded from all to allow building all of them at once.
You could also use a cache variable to toggle testing on and off:
# enabled via -D TEST_MY_PROJECT:BOOL=1 parameter for cmake
set(TEST_MY_PROJECT 0 CACHE BOOL "enable tests for my project")
if (TEST_MY_PROJECT)
# testing setup goes here
endif()
The dependency between libraries and libraries+executable could be managed via target_link_libraries. So if some library fails build - all dependent targets will not build.
I have another task: there are exist the library target, the executable test target (that provide test for library) and the main executable target with main logic.
I need that main executable target should be build only if build of the executable test target is not fails.
In other word I need set up dependency between two executables.
How I could achieve it? Could somebody provide an example?
Testing is a different stage in the CMake/Ctest/CPack suite, so you’re unlikely to find a way to conditionally compile/link an executable based on test results.
You can use add_dependencies to ensure that the test executable builds first, but that’s not the same as only producing a build product of the test runs successfully.
I recommend running your build followed by the test. Based on the return code of the test, you can choose whether or not to package the final product.
I have a cmake based (C++) project which includes tests created via add_test().
The build process is basically:
cmake
make all testsuite
make test
This has worked well for some time.
I am now trying to get my build to generate a Test.xml to be submitted as part of a pipeline build. It is not working for this project.
Q1 Can I tell cmake that I want a Test.xml to be generated when it (make test) runs the tests?
Currently I believe that a Test.xml can only be created by running ctest.
Presumably ctest aggregates the test results whereas cmake just runs them blindly?
Can someone confirm or refute this?
So I am currently trying to run ctest using:
ctest -vv -debug --output-on-failure -T test
As this does create the Test.xml file I need.
I had to add include(CTest) & include(Dart) to fix "could not find DartConfiguration.tcl".
However, even if I have previously run "make test" which guarantees the build is up to date it still tries to build and I get:
Error: could not load cache
For tests that run a compiled test (script based tests work fine).
The command it is executing is:
/opt/cmake-3.18.1/bin/cmake "--build" "<projectDir>/test/<TestName>" --target "<test_exe>"
The CMakeCache.txt & CMakeLists.txt actually live in "<projectDir>/cmake" which is why the cache cannot be loaded.
The command should be:
/opt/cmake-3.18.1/bin/cmake "--build" "<projectDir>/cmake" --target "<test_exe>"
Q2 Is there some way to tell ctest it does not need to run cmake?
My suspicion is no. Does ctest have to invoke cmake to run the tests? Does it just monitor and aggregate the output?
If I look in DartConfiguration.tcl I have:
SourceDirectory: <projectDir/cmake
BuildDirectory: <projectDir/cmake
ConfigureCommand: "/opt/cmake-3.18.1/bin/cmake" "<projectDir/cmake"
MakeCommand: /opt/cmake-3.18.1/bin/cmake --build . --config "${CTEST_CONFIGURATION_TYPE}" -- -i
I am running ctest -T test rather than ctest -T build-and-test why is ctest trying to build at all?
I've tried setting Ctest_build_command but it seems to have no effect on the DartConfiguration.tcl generated.
Some clarifications:
ctest works for me for other projects which use out of source builds
This one cannot yet use an out of source build without massive refactoring.
The source root is actually "<projectDir>"
The CMakeLists.txt lives in "<projectDir>/cmake"
Program source code lives in "<projectDir>/src"
Test scripts and code live in "<projectDir>/test"
My build worked using:
enable_testing()
I never previously had to use:
include(CTest)
include(Dart)
One or other of these creates the DartConfiguration.tcl which is just a list of "key: value" pairs. I think this is used when ctest is invoked directly which was not the case here.
I'm not entirely clear on why and what are they actually for despite reading this.
Also posted on the cmake discourse channel - https://discourse.cmake.org/t/cmake-ctest-generate-test-xml-without-rebuilding-could-not-load-cache/2025
It turns out I had some pseudo tests used to build the test driver programs via cmake --build. These were explicitly and inexplicably using CMAKE_CURRENT_DIRECTORY instead of CMAKE_SOURCE_DIR. Oops.
In my CMake project I have several targets which simply run a certain set of unit tests (for example, runTestsForA, runTestsForB and runTestsForC).
I also have a target, tests, that depends on all of these unit test targets, so I can run them with a single command.
I'm using CLion is my IDE, which tries to use parallel make builds by default (which I want and am also doing on the Continuous Integration server).
However, it looks like the tests are running in parallel too now and some tests are not made for this (they use a local loopback to do some magic with sockets), which causes them to fail.. sometimes.
That is why I would like to force serial execution for some/all of the dependencies of my tests target.
Unfortunately the CMake documentation did not help me, when I was searching information on how to do this.
Which brings me to my questions: is this at all possible and how can it be done if it is?
Instead of manual tests target declaration you can use CTest tool. Use add_test command to create test targets, then CMake will automatically create tests target that will run all tests:
enable_testing()
add_test(NAME TestsForA COMMAND <command>)
add_test(NAME TestsForB COMMAND <command>)
set_tests_properties(TestsForA TestsForB PROPERTIES RUN_SERIAL TRUE)
After that you can run make tests or ctest -j8 . in your build tree. The tests will be serialized.
More information can be found at:
http://www.cmake.org/cmake/help/v3.2/command/add_test.html
http://www.cmake.org/cmake/help/v3.2/command/enable_testing.html
http://www.cmake.org/cmake/help/v3.2/command/set_tests_properties.html
http://www.cmake.org/cmake/help/v3.2/manual/cmake-properties.7.html#properties-on-tests
http://www.cmake.org/cmake/help/v3.2/manual/ctest.1.html
Perhaps it's not the best solution for controlling test execution, but, generally I believe you can use the CMake JOB_POOLS feature to restrict the amount of parallelism for some of the CMake targets in a project.
First you declare named pool(s) and the number of parallel jobs each pool supports:
set_property(GLOBAL PROPERTY JOB_POOLS two_jobs=2 ten_jobs=10)
You can then assign targets to those pools:
set_property(TARGET myexe PROPERTY JOB_POOL_COMPILE ten_jobs)
I have defined a custom target in cmake. I now want to ensure that this target is only build when the cmake target test was executed. How can I achieve this.
Lets say that I have a target make coverage which should depend on the target make test to be called before, or call make test before executing.
How can I define this behavior in cmake?
Here my code that did not work as expected. I want to achive that make coverage depend that make test has to be called before.
ADD_CUSTOM_TARGET(
coverage COMMAND /bin/bash ${LIBPIPE_BINARY_DIR}/cmake/scripts/coverage.sh
DEPENDS test
)
You cannot add a "DEPENDS test" clause. The pre-defined/built-in targets in CMake (all, install, package, test, clean) are not available as actual targets in a CMakeLists.txt file. So you cannot express a dependency on a built-in target.
There is an outstanding feature request in the CMake bug tracker for this feature, but it is not yet implemented. See http://public.kitware.com/Bug/view.php?id=8438
However, you could change your command for your custom target, though, such that it calls "make test" first, and then runs your coverage command.
The CMake FAQ states that the add_custom_command/add_custom_target commands, which define custom targets, have a DEPENDS parameter.
Edit
This will not work for built-in target test because of following feature request.
But you always can create custom target check or whatever as suggested in CMake FAQ