I have two sets of tests (functional and unit tests) and I want to be able to specify which set to run through cmake.
One set of tests are my unittests that I want to run by doing "make test".
Another set of tests are my functional tests that I would like to run by doing "make functionaltests".
Currently both are part of ctest, in that I run both suites through add_test. My CMakeLists.txt file is like this:
FOREACH(functional_test ${functional_tests})
ADD_TEST(NAME functional_test COMMAND f_test.sh functional_test)
ENDFOREACH(functional_test)
FOREACH(unit_test ${unit_tests})
ADD_TEST(NAME unit_test COMMAND u_test.sh unit_test)
ENDFOREACH(unit_test)
I want to leverage ctest for both suites because it gives me a nice, readable format for the test suite (which tests passed and which failed).
I would prefer to not have to create a custom executable, create a target called functionaltests for it, and try to mimic how ctest prints out test results.
When you run ctest you can give it a regular expression to choose what tests to run. So you can name the tests in your CMakeLists file according to a pattern which supports this:
set(functional_tests "test1" "test2" "test3")
set(unit_tests "test1" "test2" "test3")
FOREACH(test ${functional_tests})
ADD_TEST(NAME functional_${test} COMMAND f_test.sh ${test})
ENDFOREACH()
FOREACH(test ${unit_tests})
ADD_TEST(NAME unit_${test} COMMAND u_test.sh ${test})
ENDFOREACH()
Now you can run the functional tests:
ctest -R functional_
Or the unit tests:
ctest -R unit_
If you want to create targets so that you can execute these through make, you can do:
add_custom_target(unit_tests COMMAND ${CMAKE_CTEST_COMMAND} -R unit_)
add_custom_target(functional_tests COMMAND ${CMAKE_CTEST_COMMAND} -R functional_)
Then you can run:
make unit_tests
make functional_tests
You may want to add dependencies to the custom commands so that they cause whatever executable you're testing to rebuild if necessary.
Related
I have complicated project - a big tree, with CMakeLists.txt in each directory.
Once the cmake is done, the Makefile has test target which states
.PHONY test
test:
$(SHELL) -c
"$(CTEST) $(CTEST_FLAGS) --test-dir $(BUILD_PATH)"
The problem is that the tests requires some pre-setup (temp directories to be create, environment variables to set, etc).
So if I were working with a clean make, I would just modify the Makefile to this:
.PHONY test
test:
$(SHELL) -c
"source $(BUILD_PATH)/setup_tests.sh && \
$(CTEST) $(CTEST_FLAGS) --test-dir $(BUILD_PATH)"
But the Makefile is created by cmake and I have no idea where to define such changes.
The root CMakeLists.txt has a function call enable_tests(). The src/tests directory has its own CMakeLists.txt but it just defines a bunch of subdirectories with individual tests.
Could someone point me in the direction on how to customize the execution of specific target in the cmake? Or does the ctest has option to run a shell script before all the tests (The -S looks promising but it requires cmake's language script? It does not work with the .sh).
The typical CMake way would be to use test properties:
set_tests_properties(mytest PROPERTIES
ENVIRONMENT "VAR1=val1;VAR2=val2;VAR3=val3"
)
As with running a script to set the environment as opposed to the typical CMake way, you could simply set the command to run that script before the executable:
find_program(BASH_PROGRAM bash)
add_test(NAME mytest COMMAND
${BASH_PROGRAM} -c "source \"${CMAKE_BUILD_DIR}/setup_tests.sh\" && $<TARGET_FILE:myexec>"
)
The intended way to create this kind of logic would be to define a fixture test for the setup:
For convenience we're simply running the following cmake script allowing us to specify a message in addition to allowing us to specify, if the test should fail, but you could run your own bash script as test instead.
testscript.cmake
#[===[
Params:
MESSAGE: message to print
FAIL : fail the test?
]===]
if(FAIL)
message(FATAL_ERROR "${MESSAGE}")
else()
message(STATUS "${MESSAGE}")
endif()
CMakeLists.txt
enable_testing()
# the values of the FAIL parameter could be modified to to test different scenarios...
add_test(NAME Fixture COMMAND ${CMAKE_COMMAND} -D FAIL:BOOL=0 -D "MESSAGE:STRING=executing fixture" -P ${CMAKE_CURRENT_SOURCE_DIR}/testscript.cmake)
add_test(NAME Test COMMAND ${CMAKE_COMMAND} -D FAIL:BOOL=0 -D "MESSAGE:STRING=executing actual test" -P ${CMAKE_CURRENT_SOURCE_DIR}/testscript.cmake)
add_test(NAME Test2 COMMAND ${CMAKE_COMMAND} -D FAIL:BOOL=1 -D "MESSAGE:STRING=executing test without fixture" -P ${CMAKE_CURRENT_SOURCE_DIR}/testscript.cmake)
set_tests_properties(Fixture PROPERTIES FIXTURES_SETUP Setup)
set_tests_properties(Test PROPERTIES FIXTURES_REQUIRED Setup)
Now
ctest -R 'Test$' --verbose
Executes the Fixture test and if this test succeeds, the actual test requested (Test) is executed.
ctest -R Test2 --verbose
Just executes Test2.
You can add cleanup logic for tests in a similar manner.
some pre-setup (temp directories to be create, environment variables to set, etc).
Environment variables are set with set_tests_properties as in the other answer.
You can create a directory with FILE(MAKE_DIRECTORY from CMake
For an advanced journey, you may want to discover test fixtures, which are meant to "prepare" the environment (not environment variables) for tests, for example run a service that tests need.
how to customize the execution of specific target in the cmake?
add_test takes a command, you can run any command you want.
does the ctest has option to run a shell script before all the tests
Why all the tests? Just the tests that need to run that shell - fixtures sound like they are for that.
I'm redefining my project's cmake configuration based on what I have learned from Daniel Pfeifer's “Effective CMake" talks, but I still cannot get my head around ctest.
Currently I have the following setup for my tests:
enable_testing()
find_program(MEMCHECK_COMMAND valgrind)
# executable
add_executable(myTest ${SRC_LIST})
target_link_libraries(myTest ${LIB_LIST})
# test
add_test(NAME myTest_test
COMMAND ${CMAKE_CURRENT_BINARY_DIR}/myTest)
add_custom_target(myTest_test
COMMAND ctest -R myTest_test --output-on-failure
DEPENDS myTest)
# memcheck
add_test(NAME myTest_memcheck
COMMAND ${MEMCHECK_COMMAND} ${CMAKE_CURRENT_BINARY_DIR}/myTest)
add_custom_target(myTest_memcheck
COMMAND ctest -R myTest_memcheck --output-on-failure
DEPENDS ${TEST_NAME})
and I could run make myTest_test and make myTest_memcheck and know that I didn't break anything.
It pains me that I have to do this for every test executable (and project is big so I have dozens of them) and even extracted as a function it just feels wrong.
I have read ctest docs (https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Testing-With-CTest) and dashboards seems almost perfect to me - if I could just run ctest -R myTest -T test, ctest -R myTest -T memcheck and ctest -R myTest -T coverage I would be very happy :) (and even more happy if those commands will trigger build if needed).
The problem is that include(CTest) creates targets that I do not need (Continuous/Nightly dashboards with Start/Update/Configure/Submit steps). It will be used solely as experimental builds and executed by devs during coding. For CI I have Jenkins with it's own magic. I would like to end up with:
- Dashboard/Build
- Dashboard/Test
- Dashboard/MemCheck
- Dashboard/Coverage
What should I do? Should I create manually my own dashboard (if it is possible) or drop this idea and stay with my custom targets?
I want ctest to show me the failed tests output by default. That is, I want to run:
$ make all test
and see any output of failed tests without having to cat Testing/Temporary/LastTest.log.
It appears that there are two ways of doing this:
(1) Setting the CTEST_OUTPUT_ON_FAILURE environmental variable:
$ CTEST_OUTPUT_ON_FAILURE=1 make all test
$ # or CTEST_OUTPUT_ON_FAILURE=1 ctest
(2) Specifying the --output-on-failure flag to the ctest invocation:
$ ctest --output-on-failure
Is there a way to write a CMakeLists.txt file such that ctests dumps failed tests output by default on a normal "make all test" invocation WITHOUT resorting to exporting the environmental variable globally in the session or resorting to a custom target like make check (as described here)?
I am aware of the SET_TESTS_PROPERTIES() command, but trying it out like this:
CMAKE_MINIMUM_REQUIRED(VERSION 2.8)
PROJECT(testenv CXX)
ENABLE_TESTING()
ADD_EXECUTABLE(hello hello.cpp)
ADD_TEST(testhello hello)
# Following sets the environment variable for the shell in which the test
# progoram 'hello' is run, but not the shell in which ctest is run
SET_TESTS_PROPERTIES(testhello
PROPERTIES ENVIRONMENT "CTEST_OUTPUT_ON_FAILURE=1")
and experimenting shows that the environmental variable is set in the shell that the test program is executed in, but not in the shell that ctest is executed in.
The built-in test target cannot be modified, but you can add a custom check target which invokes ctest with the --output-on-failure switch in the following way:
if (CMAKE_CONFIGURATION_TYPES)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND}
--force-new-ctest-process --output-on-failure
--build-config "$<CONFIGURATION>")
else()
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND}
--force-new-ctest-process --output-on-failure)
endif()
The custom target has to be set up differently for single build type and multi-configuration builds. In the latter case, the active build configuration has to be passed on to the ctest invocation using the --build-config flag. The --force-new-ctest-process is used by the built-in test target by default.
There is now the list variable CMAKE_CTEST_ARGUMENTS in which you can set ctest arguments.
https://cmake.org/cmake/help/latest/variable/CMAKE_CTEST_ARGUMENTS.html
I'm using CMake to build my project. I have added a unit test binary which is using the Boost unit testing framework. This one binary contains all of the unit tests. I've added that binary to be run by CTest:
ADD_EXECUTABLE( tftest test-main.cpp )
ENABLE_TESTING()
ADD_TEST( UnitTests tftest)
But the build output in Visual Studio only shows the result of running CTest:
Start 1: UnitTests
1/1 Test #1: UnitTests ................***Failed 0.05 sec
0% tests passed, 1 tests failed out of 1
This is not very helpful, because I can't see which test failed. If I run ctest manually from the command line with --verbose I get the output from a Boost unit test which tells what actually failed:
1: Test command: tftest.exe
1: Test timeout computed to be: 9.99988e+006
1: Running 4 test cases...
1: test-main.cpp(20): error in "sanity_check3": check 1 == 2 failed
1:
1: *** 1 failure detected in test suite "Master Test Suite"
1/1 Test #1: UnitTests ................***Failed 0.00 sec
So, what do I need to change in the CMakeLists.txt to have CTest run with --verbose at all times? Is there a better way to use Boost unit tests with CMake/CTest?
You can use the ctest --output-on-failure option, or set the environment variable CTEST_OUTPUT_ON_FAILURE, which will show you any output from the test program whenever the test fails. One way to do this when using Makefiles and the command line would be as follows:
env CTEST_OUTPUT_ON_FAILURE=1 make check
This Stack Overflow question and answer shows how to set environment variables in Visual Studio.
You could call ctest directly, after cmaking and making your project.
ctest --verbose
There is a very simple solution (which for some reason is difficult to find via Google Search):
ctest --output-on-failure
If you use CMake with Visual Studio's open folder function you can add the
"ctestCommandArgs": "--output-on-failure"
setting to your build configuration.
You can check the Testing/Temporary subfolder. It is automatically created after running make test. This folder contains two files: LastTest.log and LastTestsFailed.log. LastTest.log contains desired output for run tests. LastTestFailed.log contains names of failed tests. So you can check them manually after executing make test.
The second way is to get ctest to show you the content of log files after running tests:
place in build dir (from which you run make test) file CTestCustom.ctest (you can do it with configure file command, for example) with following contents
CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log")
Instead of cat you may use whatever Windows cmd command that does similar things.
run make test again and get profit!
additional info about customizing ctest you can find here. Just step to "Customizing cmake" section.
Good luck!
I had to add "check" target by myself. "make tests" does nothing by some reason. So what I did (as was suggest somewhere on stackoverflow) - I added this target manually. To get verbose output I just wrote it like:
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
make check CTEST_OUTPUT_ON_FAILURE=TRUE
This makes test output more verbose:
make test ARGS="-V"
My approach is a combination of the answers from ony, from zbyszek, and from tarc. I use the ${CMAKE_COMMAND} variable (which is set to the absolute path to the invoked cmake executable) with the -E env CTEST_OUTPUT_ON_FAILURE=1 argument to invoke the actual ctest command using ${CMAKE_CTEST_COMMAND} -C $<CONFIG>. To help clarify what is going on, I start with three cmake -E echo commands to show the current working directory and the ctest command to be invoked. Here is how I call add_custom_target.
add_custom_target(check
${CMAKE_COMMAND} -E echo CWD=${CMAKE_BINARY_DIR}
COMMAND ${CMAKE_COMMAND} -E echo CMD=${CMAKE_CTEST_COMMAND} -C $<CONFIG>
COMMAND ${CMAKE_COMMAND} -E echo ----------------------------------
COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS ALL_BUILD
)
This plays nice with the MSVC IDE where any test errors are shown as clickable compilation errors. See cmake -E env for documentation of the cmake -E portable command line tool mode. I also add a dependency on ALL_BUILD so that all projects will be built before invoking the check target. (On Linux builds, one may need to replace ALL_BUILD with ALL; I have not tested this on Linux yet.)
For people using Visual Studio, here another variation (hack) on the theme:
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build . --target RUN_TESTS
ctest -VV or ctest --extra-verbose
From documentation:
Enable more verbose output from tests.
Test output is normally suppressed and only summary information is
displayed. This option will show even more test output.
There's now a CMake variable that allows you to modify the behaviour of make test. CMAKE_CTEST_ARGUMENTS lets you set a list of arguments to pass to ctest when run via make test.
So adding this to your CMake file:
set(CMAKE_CTEST_ARGUMENTS "--verbose")
Means CTest will always run verbose. Or for just the output of the failed tests, use:
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
Edit:
As suggested by RobLoach, since it's a list of arguments, you'll want to append to the list instead.
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
to show the result with XML file you have to execute the test with the following command
~$ ctest -T Test
and we found the result in the Testing/1234123432/test.xml
and other files are generated too in Testing Folder
I'm building some tests using CTest. Usually, I can set up the test by simply the line:
ADD_TEST(Test_Name executable args)
However, I've run into a problem, I have some tests that require two commands to be run in order for it to work, is there any way I can run two programs within a single ctest, or am I required to create a new test for each?
Thank you.
The add_test command only accepts one executable, but you can run any executable that is really a script. To do this in a cross platform way, write the script in CMake itself. CMake has the -P option for running arbitrary chunks of CMake scripting language when you run make or make test, rather than at Makefile generation time.
Sadly you can't pass arguments to such a script. But you can set variables to values, which is just as good.
This script you can call runtests.cmake, it runs the commands CMD1 and CMD2 and checks each for a non-zero return code, returning out of CMake itself with an error if that happens:
macro(EXEC_CHECK CMD)
execute_process(COMMAND ${CMD} RESULT_VARIABLE CMD_RESULT)
if(CMD_RESULT)
message(FATAL_ERROR "Error running ${CMD}")
endif()
endmacro()
exec_check(${CMD1})
exec_check(${CMD2})
... and then add your test cases like so:
add_executable(test1 test1.c)
add_executable(test2 test2.c)
add_test(NAME test
COMMAND ${CMAKE_COMMAND}
-DCMD1=$<TARGET_FILE:test1>
-DCMD2=$<TARGET_FILE:test2>
-P ${CMAKE_CURRENT_SOURCE_DIR}/runtests.cmake)
$<TARGET_FILE:test1> gets expanded to the full path to the executable at build-file generation time. When you run make test or equivalent this will run "cmake -P runtests.cmake" setting the CMD1 and CMD2 variables to the appropriate test programs. The script will then execute your 2 programs in sequence. If either of the test programs fail, the whole test fails. If you need to see the output of the test, you can run make test ARGS=-V
There is a simple, although not cross platform, way to achieve this.
In Linux you can use bash to execute multiple commands:
add_test(
NAME
TestName
COMMAND
bash -c "COMMAND1 ; \
COMMAND2 ; \
${CMAKE_CURRENT_BINARY_DIR}/testExecutable"
)