I have the ctest setting and work fine with all executable I've built for the project. The problem is I want to add some test with a system command, it has nothing to do with program in the solution, I cannot get it run by ctest.
I added a simple test as below, it just run the command echo:
add_test(NAME toto_test COMMAND echo bla bla bla)
Then the when I run ctest, I got the error
Start 1: toto_test
Could not find executable echo
Looked in the following places:
echo
echo.exe
Release/echo
Release/echo.exe
Release/echo
Release/echo.exe
Unable to find executable: echo
1/10 Test #1: toto_test ........................***Not Run 0.00 sec
The following tests FAILED:
1 - toto_test (Not Run)
Errors while running CTest
And when I run ctest with verbose option, I get
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: toto_test
Could not find executable echo
Looked in the following places:
echo
echo.exe
Release/echo
Release/echo.exe
Release/echo
Release/echo.exe
1: Test command: "bla" "bla" "bla"
Unable to find executable: echo
1/1 Test #1: toto_test ........................***Not Run 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.02 sec
The following tests FAILED:
1 - toto_test (Not Run)
Errors while running CTest
Can anyone come to help please.
Thanks a lot
I found the way to do it. For the general purpose of running a test with an external program, use add_custom_target, then use ${CMAKE_COMMAND} to build this target when running the tests :
add_custom_target(run_toto COMMAND echo bla bla bla)
add_test(NAME test_toto COMMAND ${CMAKE_COMMAND} --build . --target run_toto)
Related
I'd like to disable a set of tests by default, but be able to run them when explicitly targeted.
E.g. suppose I have a project with lots of fast-running unit tests that have been added via a add_test(NAME SomeNameHere COMMAND SomeCommandHere) command. And suppose further that there are also a bunch of long-running integration tests that should be excluded from the default ctest run, But it should be possible to run all integration tests with a single command when explicity targeted.
As a concrete example:
TestThatShouldAlwaysRun - I'd like this to run when a user runs $ ctest
FooIntegrationTest - I'd like this to run only when a user runs $ ctest --some-option-here integration
BarIntegrationTest - I'd like this to run only when a user runs $ ctest --some-option-here integration
Here's a my attempt to achieve this.
cmake_minimum_required (VERSION 3.1)
project (HOW_TO_RUN_DISABLED_TESTS NONE)
enable_testing()
# Add a fast running test
# This test should be run when a user runs: 'ctest'
add_test(NAME TestThatShouldAlwaysRun COMMAND echo 'I am always enabled')
# Add a long-running integration test
# This test should be run when a user runs: 'ctest --label-regex my_integration_tests'
add_test(NAME FooIntegrationTest COMMAND echo 'Foo integration test')
set_tests_properties(FooIntegrationTest PROPERTIES DISABLED True)
set_tests_properties(FooIntegrationTest PROPERTIES LABELS my_integration_tests)
# Add another long-running integration test
# This test should be run when a user runs: 'ctest --label-regex my_integration_tests'
add_test(NAME BarIntegrationTest COMMAND echo 'Bar integration test')
set_tests_properties(BarIntegrationTest PROPERTIES DISABLED True)
set_tests_properties(BarIntegrationTest PROPERTIES LABELS my_integration_tests)
Now, let's run this via
$ mkdir build
$ cd build/
$ cmake ..
$ ctest -V
As we can see in the output (posted below) ctest did exclude the integration tests (great!).
1: Test command: /bin/echo "'I" "am" "always" "enabled'"
1: Test timeout computed to be: 10000000
1: 'I am always enabled'
1/3 Test #1: TestThatShouldAlwaysRun .......... Passed 0.00 sec
test 2
Start 2: FooIntegrationTest
2/3 Test #2: FooIntegrationTest ...............***Not Run (Disabled) 0.00 sec
test 3
Start 3: BarIntegrationTest
3/3 Test #3: BarIntegrationTest ...............***Not Run (Disabled) 0.00 sec
100% tests passed, 0 tests failed out of 1
Label Time Summary:
my_integration_tests = 0.00 sec*proc (2 tests)
Total Test time (real) = 0.05 sec
The following tests did not run:
2 - FooIntegrationTest (Disabled)
3 - BarIntegrationTest (Disabled)
But, I can not figure out how I can now, through ctest, run tests labeled with my_integration_tests.
ctest --label-regex my_integration_tests
Test project /path/to/code/example_repo/build
Start 2: FooIntegrationTest
1/2 Test #2: FooIntegrationTest ...............***Not Run (Disabled) 0.00 sec
Start 3: BarIntegrationTest
2/2 Test #3: BarIntegrationTest ...............***Not Run (Disabled) 0.00 sec
No tests were found!!!
Is there a way to run disabled tests when explicitly targeted?
I have explored other ways like
Not disabeling the integration tests. Then, when a user wants to run the fast running tests they need to specify ctest --label-exclude my_integration_tests. Which isn't great. That puts a burdon on everyone to remember this. I'd like the options that's used the most (running only the fast-running tests) to be runnable via ctest.
Not disableing the integration tests but provide the argument CONFIGURATIONS my_integration_tests to the add_test call. But I think that way I am semantically misusing CONFIGURATIONS. Also this doesn't allow me to run only the integration test when targeted.
Add the integration tests via add_custom_target. Also here I think I am misusing the API. add_custom_target is meant for custom targets, not as a way to run test, right?
I believe a simpler way to do this is just via a cmake configuration option.
# cat CMakeLists.txt
cmake_minimum_required (VERSION 3.1)
project (HOW_TO_RUN_DISABLED_TESTS NONE)
enable_testing()
add_test(NAME TestThatShouldAlwaysRun COMMAND echo 'I am always enabled')
if(BUILD_INTEGRATION_TESTING)
add_test(NAME FooIntegrationTest COMMAND echo 'Foo integration test')
add_test(NAME BarIntegrationTest COMMAND echo 'Bar integration test')
endif()
This is simple and also is more flexible, ie. enables addition of test dependent targets to be build (and setup (and ctest fixtures to be used)) for those tests. (I usually in projects see variables named like SOMETHING_BUILD_TESTING so it's similar to BUILD_TESTING cmake variable.)
Another idea I have is to use an environment variable and wrap the test in a script with SKIP_RETURN_CODE (or even without SKIP_RETURN_CODE and just return success if integration tests shouldn't be run):
add_test(NAME FooIntegrationTest
COMMAND sh -c "if [ \"\${RUN_INTEGRATION:-}\" ]; then echo 'Foo integration test'; else exit 127; fi"
VERBATIM)
set_tests_properties(FooIntegrationTest PROPERTIES
SKIP_RETURN_CODE 127)
then:
$ ctest -V
....
2: Test command: /usr/bin/sh "-c" "if [ "${RUN_INTEGRATION:-}" ]; then echo 'Foo integration test'; else exit 127; fi" "VERBATIM"
2: Test timeout computed to be: 10000000
2/3 Test #2: FooIntegrationTest ...............***Skipped 0.01 sec
....
The following tests did not run:
2 - FooIntegrationTest (Skipped)
but:
$ RUN_INTEGRATION=TRUE ctest -V
...
2: Test command: /usr/bin/sh "-c" "if [ "${RUN_INTEGRATION:-}" ]; then echo 'Foo integration test'; else exit 127; fi" "VERBATIM"
2: Test timeout computed to be: 10000000
2: Foo integration test
2/3 Test #2: FooIntegrationTest ............... Passed 0.01 sec
...
You can do that with LABELS property.
Mark the tests that you always want to run as "GOLD" tests and always run ctest as ctest -L GOLD.
Other tests that you want to disable by default can be categorized with other labels such as "EXPFAIL", "DISABLED" etc. And when you want to run those, you can use appropriate -L label.
This doesn't even require you to re-configure the tests. Configure once, and run what you want to run.
The only downside is that by default we have to use ctest -L "GOLD" instead of ctest.
I have found that generic ctest command doesn't give much information about the tests, so I would like to add ctest --output-on-failure but not have the users to worry about the flag. I want them just to cmake, make the project and run ctest and it should run ctest with the --output-on-failure flag. Is it possible to do that in CMakeLists.txt?
EDIT:
Output of env CTEST_OUTPUT_ON_FAILURE=1 make test
4/13 Test #4: TEST_SSSP ........................***Failed Required regular expression not found.Regex=[CORRECT
] 0.00 sec
Loading Matrix-market coordinate-formatted graph ...
Input graph file /home/muhammad/gunrock/dataset/small/chesapeake.mtx does not exis
Output of set_property(TEST TestName PROPERTY ENVIRONMENT "CTEST_OUTPUT_ON_FAILURE=1")
4/13 Test #4: TEST_SSSP ........................***Failed Required regular expression not found.Regex=[CORRECT
] 0.00 sec
The flag in set_property is not working.
In CMake 3.17 release notes, there's a new variable CMAKE_CTEST_ARGUMENTS that you can set to pass any command-line arguments to CTest, including --output-on-failure. In your specific case, you can now simply add this to your CMakeLists.txt:
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
I went with make check per solution: CMake: setting an environmental variable for ctest (or otherwise getting failed test output from ctest/make test automatically)
add_custom_target(check ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG> --verbose
WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
I have a Travis CI project which builds an iOS app then starts Appium and runs tests with Appium/Mocha.
The problem is that even though the Mocha tests fail and throw exception, the shell script which runs them via Gulp still exits with 0 and the build is deemed passing.
How can I make the build break/fail when the Mocha tests fail?
Here is how I managed to make this work:
Instead of running the Mocha tests via Gulp, run them directly from the shell script
Save the output to mocha.log besides displaying on stdout
./node_modules/.bin/mocha --reporter spec "appium/hybrid/*uat.js" 2>&1 | tee mocha.log
Check mocha.log for the string " failing" and exit with 1 if found
.
if grep -q " failing" mocha.log; then
exit 1
fi
The exit 1 will make the Travis build fail.
How do I get "make test" to display verbose output? I want "make test" to do the same thing as ctest -V through the command line.
I have tried adding the following to my CMakeLists.txt, nothing worked :(
set(ENV{CTEST_OUTPUT_ON_FAILURE} TRUE)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
add_custom_command(TARGET test PRE_BUILD
COMMAND ${CMAKE_CTEST_COMMAND} -V)
But I still get this when I run make test:
Start 1: unittest1
1/143 Test #1: unittest1 .................................... Passed 0.01 sec
Start 2: unittest2
2/143 Test #2: unittest2 ............................ Passed 0.03 sec
Start 3: unittest3
3/143 Test #3: unittest3 .................... Passed 0.02 sec
To clarify, I want to add something to my CMakeLists.txt to make this possible, I don't want a manual solution that requires me to append something to "make test" in the command line such as
CTEST_OUTPUT_ON_FAILURE=TRUE make test
or
ctest -V
My question is similar to Using cmake how do I get verbose output from ctest?.
From GNU Radio's wiki page on doing tests on out-of-tree modules (courtesy of Mr. Braun):
Run ctest -V from the build directory (usually), and it will give you verbose information. Add -R regex to execute only tests that match regex.
Now, re-running a test on failure does seem to make a whole lot of sense to me -- and not automatically doing that on every make test, too, as tests might be time-consuming, and shouldn't be repeated in a broken build environment, etc. by default.
How to add that behaviour to the default make test behaviour of course depends on your CMake infrastructure, and I can't answer that without reading through your code.
I'm using CMake to build my project. I have added a unit test binary which is using the Boost unit testing framework. This one binary contains all of the unit tests. I've added that binary to be run by CTest:
ADD_EXECUTABLE( tftest test-main.cpp )
ENABLE_TESTING()
ADD_TEST( UnitTests tftest)
But the build output in Visual Studio only shows the result of running CTest:
Start 1: UnitTests
1/1 Test #1: UnitTests ................***Failed 0.05 sec
0% tests passed, 1 tests failed out of 1
This is not very helpful, because I can't see which test failed. If I run ctest manually from the command line with --verbose I get the output from a Boost unit test which tells what actually failed:
1: Test command: tftest.exe
1: Test timeout computed to be: 9.99988e+006
1: Running 4 test cases...
1: test-main.cpp(20): error in "sanity_check3": check 1 == 2 failed
1:
1: *** 1 failure detected in test suite "Master Test Suite"
1/1 Test #1: UnitTests ................***Failed 0.00 sec
So, what do I need to change in the CMakeLists.txt to have CTest run with --verbose at all times? Is there a better way to use Boost unit tests with CMake/CTest?
You can use the ctest --output-on-failure option, or set the environment variable CTEST_OUTPUT_ON_FAILURE, which will show you any output from the test program whenever the test fails. One way to do this when using Makefiles and the command line would be as follows:
env CTEST_OUTPUT_ON_FAILURE=1 make check
This Stack Overflow question and answer shows how to set environment variables in Visual Studio.
You could call ctest directly, after cmaking and making your project.
ctest --verbose
There is a very simple solution (which for some reason is difficult to find via Google Search):
ctest --output-on-failure
If you use CMake with Visual Studio's open folder function you can add the
"ctestCommandArgs": "--output-on-failure"
setting to your build configuration.
You can check the Testing/Temporary subfolder. It is automatically created after running make test. This folder contains two files: LastTest.log and LastTestsFailed.log. LastTest.log contains desired output for run tests. LastTestFailed.log contains names of failed tests. So you can check them manually after executing make test.
The second way is to get ctest to show you the content of log files after running tests:
place in build dir (from which you run make test) file CTestCustom.ctest (you can do it with configure file command, for example) with following contents
CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log")
Instead of cat you may use whatever Windows cmd command that does similar things.
run make test again and get profit!
additional info about customizing ctest you can find here. Just step to "Customizing cmake" section.
Good luck!
I had to add "check" target by myself. "make tests" does nothing by some reason. So what I did (as was suggest somewhere on stackoverflow) - I added this target manually. To get verbose output I just wrote it like:
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
make check CTEST_OUTPUT_ON_FAILURE=TRUE
This makes test output more verbose:
make test ARGS="-V"
My approach is a combination of the answers from ony, from zbyszek, and from tarc. I use the ${CMAKE_COMMAND} variable (which is set to the absolute path to the invoked cmake executable) with the -E env CTEST_OUTPUT_ON_FAILURE=1 argument to invoke the actual ctest command using ${CMAKE_CTEST_COMMAND} -C $<CONFIG>. To help clarify what is going on, I start with three cmake -E echo commands to show the current working directory and the ctest command to be invoked. Here is how I call add_custom_target.
add_custom_target(check
${CMAKE_COMMAND} -E echo CWD=${CMAKE_BINARY_DIR}
COMMAND ${CMAKE_COMMAND} -E echo CMD=${CMAKE_CTEST_COMMAND} -C $<CONFIG>
COMMAND ${CMAKE_COMMAND} -E echo ----------------------------------
COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS ALL_BUILD
)
This plays nice with the MSVC IDE where any test errors are shown as clickable compilation errors. See cmake -E env for documentation of the cmake -E portable command line tool mode. I also add a dependency on ALL_BUILD so that all projects will be built before invoking the check target. (On Linux builds, one may need to replace ALL_BUILD with ALL; I have not tested this on Linux yet.)
For people using Visual Studio, here another variation (hack) on the theme:
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build . --target RUN_TESTS
ctest -VV or ctest --extra-verbose
From documentation:
Enable more verbose output from tests.
Test output is normally suppressed and only summary information is
displayed. This option will show even more test output.
There's now a CMake variable that allows you to modify the behaviour of make test. CMAKE_CTEST_ARGUMENTS lets you set a list of arguments to pass to ctest when run via make test.
So adding this to your CMake file:
set(CMAKE_CTEST_ARGUMENTS "--verbose")
Means CTest will always run verbose. Or for just the output of the failed tests, use:
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
Edit:
As suggested by RobLoach, since it's a list of arguments, you'll want to append to the list instead.
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
to show the result with XML file you have to execute the test with the following command
~$ ctest -T Test
and we found the result in the Testing/1234123432/test.xml
and other files are generated too in Testing Folder