CMake verbose output from tests - cmake

How do I get "make test" to display verbose output? I want "make test" to do the same thing as ctest -V through the command line.
I have tried adding the following to my CMakeLists.txt, nothing worked :(
set(ENV{CTEST_OUTPUT_ON_FAILURE} TRUE)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
add_custom_command(TARGET test PRE_BUILD
COMMAND ${CMAKE_CTEST_COMMAND} -V)
But I still get this when I run make test:
Start 1: unittest1
1/143 Test #1: unittest1 .................................... Passed 0.01 sec
Start 2: unittest2
2/143 Test #2: unittest2 ............................ Passed 0.03 sec
Start 3: unittest3
3/143 Test #3: unittest3 .................... Passed 0.02 sec
To clarify, I want to add something to my CMakeLists.txt to make this possible, I don't want a manual solution that requires me to append something to "make test" in the command line such as
CTEST_OUTPUT_ON_FAILURE=TRUE make test
or
ctest -V
My question is similar to Using cmake how do I get verbose output from ctest?.

From GNU Radio's wiki page on doing tests on out-of-tree modules (courtesy of Mr. Braun):
Run ctest -V from the build directory (usually), and it will give you verbose information. Add -R regex to execute only tests that match regex.
Now, re-running a test on failure does seem to make a whole lot of sense to me -- and not automatically doing that on every make test, too, as tests might be time-consuming, and shouldn't be repeated in a broken build environment, etc. by default.
How to add that behaviour to the default make test behaviour of course depends on your CMake infrastructure, and I can't answer that without reading through your code.

Related

Should I use ctest dashboard if I don't use CDash?

I'm redefining my project's cmake configuration based on what I have learned from Daniel Pfeifer's “Effective CMake" talks, but I still cannot get my head around ctest.
Currently I have the following setup for my tests:
enable_testing()
find_program(MEMCHECK_COMMAND valgrind)
# executable
add_executable(myTest ${SRC_LIST})
target_link_libraries(myTest ${LIB_LIST})
# test
add_test(NAME myTest_test
COMMAND ${CMAKE_CURRENT_BINARY_DIR}/myTest)
add_custom_target(myTest_test
COMMAND ctest -R myTest_test --output-on-failure
DEPENDS myTest)
# memcheck
add_test(NAME myTest_memcheck
COMMAND ${MEMCHECK_COMMAND} ${CMAKE_CURRENT_BINARY_DIR}/myTest)
add_custom_target(myTest_memcheck
COMMAND ctest -R myTest_memcheck --output-on-failure
DEPENDS ${TEST_NAME})
and I could run make myTest_test and make myTest_memcheck and know that I didn't break anything.
It pains me that I have to do this for every test executable (and project is big so I have dozens of them) and even extracted as a function it just feels wrong.
I have read ctest docs (https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Testing-With-CTest) and dashboards seems almost perfect to me - if I could just run ctest -R myTest -T test, ctest -R myTest -T memcheck and ctest -R myTest -T coverage I would be very happy :) (and even more happy if those commands will trigger build if needed).
The problem is that include(CTest) creates targets that I do not need (Continuous/Nightly dashboards with Start/Update/Configure/Submit steps). It will be used solely as experimental builds and executed by devs during coding. For CI I have Jenkins with it's own magic. I would like to end up with:
- Dashboard/Build
- Dashboard/Test
- Dashboard/MemCheck
- Dashboard/Coverage
What should I do? Should I create manually my own dashboard (if it is possible) or drop this idea and stay with my custom targets?

ctest cannot run with simple command

I have the ctest setting and work fine with all executable I've built for the project. The problem is I want to add some test with a system command, it has nothing to do with program in the solution, I cannot get it run by ctest.
I added a simple test as below, it just run the command echo:
add_test(NAME toto_test COMMAND echo bla bla bla)
Then the when I run ctest, I got the error
Start 1: toto_test
Could not find executable echo
Looked in the following places:
echo
echo.exe
Release/echo
Release/echo.exe
Release/echo
Release/echo.exe
Unable to find executable: echo
1/10 Test #1: toto_test ........................***Not Run 0.00 sec
The following tests FAILED:
1 - toto_test (Not Run)
Errors while running CTest
And when I run ctest with verbose option, I get
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: toto_test
Could not find executable echo
Looked in the following places:
echo
echo.exe
Release/echo
Release/echo.exe
Release/echo
Release/echo.exe
1: Test command: "bla" "bla" "bla"
Unable to find executable: echo
1/1 Test #1: toto_test ........................***Not Run 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.02 sec
The following tests FAILED:
1 - toto_test (Not Run)
Errors while running CTest
Can anyone come to help please.
Thanks a lot
I found the way to do it. For the general purpose of running a test with an external program, use add_custom_target, then use ${CMAKE_COMMAND} to build this target when running the tests :
add_custom_target(run_toto COMMAND echo bla bla bla)
add_test(NAME test_toto COMMAND ${CMAKE_COMMAND} --build . --target run_toto)

Replace ctest command with "ctest --output-on-failure" permanently for a specific project in CMakeLists.txt

I have found that generic ctest command doesn't give much information about the tests, so I would like to add ctest --output-on-failure but not have the users to worry about the flag. I want them just to cmake, make the project and run ctest and it should run ctest with the --output-on-failure flag. Is it possible to do that in CMakeLists.txt?
EDIT:
Output of env CTEST_OUTPUT_ON_FAILURE=1 make test
4/13 Test #4: TEST_SSSP ........................***Failed Required regular expression not found.Regex=[CORRECT
] 0.00 sec
Loading Matrix-market coordinate-formatted graph ...
Input graph file /home/muhammad/gunrock/dataset/small/chesapeake.mtx does not exis
Output of set_property(TEST TestName PROPERTY ENVIRONMENT "CTEST_OUTPUT_ON_FAILURE=1")
4/13 Test #4: TEST_SSSP ........................***Failed Required regular expression not found.Regex=[CORRECT
] 0.00 sec
The flag in set_property is not working.
In CMake 3.17 release notes, there's a new variable CMAKE_CTEST_ARGUMENTS that you can set to pass any command-line arguments to CTest, including --output-on-failure. In your specific case, you can now simply add this to your CMakeLists.txt:
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
I went with make check per solution: CMake: setting an environmental variable for ctest (or otherwise getting failed test output from ctest/make test automatically)
add_custom_target(check ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG> --verbose
WORKING_DIRECTORY ${CMAKE_BINARY_DIR})

Two sets of tests in Cmake

I have two sets of tests (functional and unit tests) and I want to be able to specify which set to run through cmake.
One set of tests are my unittests that I want to run by doing "make test".
Another set of tests are my functional tests that I would like to run by doing "make functionaltests".
Currently both are part of ctest, in that I run both suites through add_test. My CMakeLists.txt file is like this:
FOREACH(functional_test ${functional_tests})
ADD_TEST(NAME functional_test COMMAND f_test.sh functional_test)
ENDFOREACH(functional_test)
FOREACH(unit_test ${unit_tests})
ADD_TEST(NAME unit_test COMMAND u_test.sh unit_test)
ENDFOREACH(unit_test)
I want to leverage ctest for both suites because it gives me a nice, readable format for the test suite (which tests passed and which failed).
I would prefer to not have to create a custom executable, create a target called functionaltests for it, and try to mimic how ctest prints out test results.
When you run ctest you can give it a regular expression to choose what tests to run. So you can name the tests in your CMakeLists file according to a pattern which supports this:
set(functional_tests "test1" "test2" "test3")
set(unit_tests "test1" "test2" "test3")
FOREACH(test ${functional_tests})
ADD_TEST(NAME functional_${test} COMMAND f_test.sh ${test})
ENDFOREACH()
FOREACH(test ${unit_tests})
ADD_TEST(NAME unit_${test} COMMAND u_test.sh ${test})
ENDFOREACH()
Now you can run the functional tests:
ctest -R functional_
Or the unit tests:
ctest -R unit_
If you want to create targets so that you can execute these through make, you can do:
add_custom_target(unit_tests COMMAND ${CMAKE_CTEST_COMMAND} -R unit_)
add_custom_target(functional_tests COMMAND ${CMAKE_CTEST_COMMAND} -R functional_)
Then you can run:
make unit_tests
make functional_tests
You may want to add dependencies to the custom commands so that they cause whatever executable you're testing to rebuild if necessary.

Using CMake, how do I get verbose output from CTest?

I'm using CMake to build my project. I have added a unit test binary which is using the Boost unit testing framework. This one binary contains all of the unit tests. I've added that binary to be run by CTest:
ADD_EXECUTABLE( tftest test-main.cpp )
ENABLE_TESTING()
ADD_TEST( UnitTests tftest)
But the build output in Visual Studio only shows the result of running CTest:
Start 1: UnitTests
1/1 Test #1: UnitTests ................***Failed 0.05 sec
0% tests passed, 1 tests failed out of 1
This is not very helpful, because I can't see which test failed. If I run ctest manually from the command line with --verbose I get the output from a Boost unit test which tells what actually failed:
1: Test command: tftest.exe
1: Test timeout computed to be: 9.99988e+006
1: Running 4 test cases...
1: test-main.cpp(20): error in "sanity_check3": check 1 == 2 failed
1:
1: *** 1 failure detected in test suite "Master Test Suite"
1/1 Test #1: UnitTests ................***Failed 0.00 sec
So, what do I need to change in the CMakeLists.txt to have CTest run with --verbose at all times? Is there a better way to use Boost unit tests with CMake/CTest?
You can use the ctest --output-on-failure option, or set the environment variable CTEST_OUTPUT_ON_FAILURE, which will show you any output from the test program whenever the test fails. One way to do this when using Makefiles and the command line would be as follows:
env CTEST_OUTPUT_ON_FAILURE=1 make check
This Stack Overflow question and answer shows how to set environment variables in Visual Studio.
You could call ctest directly, after cmaking and making your project.
ctest --verbose
There is a very simple solution (which for some reason is difficult to find via Google Search):
ctest --output-on-failure
If you use CMake with Visual Studio's open folder function you can add the
"ctestCommandArgs": "--output-on-failure"
setting to your build configuration.
You can check the Testing/Temporary subfolder. It is automatically created after running make test. This folder contains two files: LastTest.log and LastTestsFailed.log. LastTest.log contains desired output for run tests. LastTestFailed.log contains names of failed tests. So you can check them manually after executing make test.
The second way is to get ctest to show you the content of log files after running tests:
place in build dir (from which you run make test) file CTestCustom.ctest (you can do it with configure file command, for example) with following contents
CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log")
Instead of cat you may use whatever Windows cmd command that does similar things.
run make test again and get profit!
additional info about customizing ctest you can find here. Just step to "Customizing cmake" section.
Good luck!
I had to add "check" target by myself. "make tests" does nothing by some reason. So what I did (as was suggest somewhere on stackoverflow) - I added this target manually. To get verbose output I just wrote it like:
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
make check CTEST_OUTPUT_ON_FAILURE=TRUE
This makes test output more verbose:
make test ARGS="-V"
My approach is a combination of the answers from ony, from zbyszek, and from tarc. I use the ${CMAKE_COMMAND} variable (which is set to the absolute path to the invoked cmake executable) with the -E env CTEST_OUTPUT_ON_FAILURE=1 argument to invoke the actual ctest command using ${CMAKE_CTEST_COMMAND} -C $<CONFIG>. To help clarify what is going on, I start with three cmake -E echo commands to show the current working directory and the ctest command to be invoked. Here is how I call add_custom_target.
add_custom_target(check
${CMAKE_COMMAND} -E echo CWD=${CMAKE_BINARY_DIR}
COMMAND ${CMAKE_COMMAND} -E echo CMD=${CMAKE_CTEST_COMMAND} -C $<CONFIG>
COMMAND ${CMAKE_COMMAND} -E echo ----------------------------------
COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS ALL_BUILD
)
This plays nice with the MSVC IDE where any test errors are shown as clickable compilation errors. See cmake -E env for documentation of the cmake -E portable command line tool mode. I also add a dependency on ALL_BUILD so that all projects will be built before invoking the check target. (On Linux builds, one may need to replace ALL_BUILD with ALL; I have not tested this on Linux yet.)
For people using Visual Studio, here another variation (hack) on the theme:
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build . --target RUN_TESTS
ctest -VV or ctest --extra-verbose
From documentation:
Enable more verbose output from tests.
Test output is normally suppressed and only summary information is
displayed. This option will show even more test output.
There's now a CMake variable that allows you to modify the behaviour of make test. CMAKE_CTEST_ARGUMENTS lets you set a list of arguments to pass to ctest when run via make test.
So adding this to your CMake file:
set(CMAKE_CTEST_ARGUMENTS "--verbose")
Means CTest will always run verbose. Or for just the output of the failed tests, use:
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
Edit:
As suggested by RobLoach, since it's a list of arguments, you'll want to append to the list instead.
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
to show the result with XML file you have to execute the test with the following command
~$ ctest -T Test
and we found the result in the Testing/1234123432/test.xml
and other files are generated too in Testing Folder