I am trying to get CMake/CTest to work. My problem is that CTest does not seem to pick up a test executable unless it is named tests.
In my minimal example project I have a single CMakeLists.txt file:
cmake_minimum_required(VERSION 3.0)
project(myproject C)
enable_testing()
set(TEST_EXE_NAME tests)
add_executable(${TEST_EXE_NAME} test_main.c)
add_test(NAME "My tests" COMMAND ${TEST_EXE_NAME})
...and a very simple test program, test_main.c, that always passes:
int main() {
return 0;
}
I can run make && make test, and all is fine as long as TEST_EXE_NAME is set to tests. However, when I change the executable name to something else, e.g. mytests, I get the following error:
Could not find executable tests
Looked in the following places:
tests
tests
Release/tests
Release/tests
Debug/tests
...
What am I missing?
According to the CMake manual for add_test() the test name may not contain spaces:
The test name may not contain spaces, quotes, or other
characters special in CMake syntax.
In the problematic example, the test was named "My tests". Changing the test name to "My_tests" solves the problem.
Apparently, the part of the test name after the space character (i.e. "tests") was interpreted as the test executable name.
Related
I'm using gcov and I'd like to gather coverage results on a per test case basis.
QUESTION
When I execute the googletest executable, is it possible to pass in an argument on command line that says execute only the Nth test case?
I'd even be ok with passing in the name of the fixture+case; I can just use Python with regex to pull out all the tests.
Obviously I can accomplish the same thing by having one test case per .cpp file but that sounds ... stupid.
googletest allows to run a single test case or even the subset of the tests. You can specify a filter string in the GTEST_FILTER environment variable or in the --gtest_filter command line option and googletest will only run the tests whose full names (in the form of TestSuiteName.TestName) match the filter. More information about the format of the filter string can be found in Running a Subset of the Tests section. Also googletest supports --gtest_list_tests command line option to print the list of all test cases. It can be useful in your case.
I'm currently working on migrating our current build environment from MSBuild to CMake. I have a situation where I need to update the PATH variable in order for the units tests executable to run. This is not a issue for gtest_add_tests, as it uses the source to identify tests. But gtest_discover_tests, which executes the unit tests with the --gtest_list_tests flag, fails to identify any tests because a STATUS_DLL_NOT_FOUND error is encountered during the build.
For example:
add_executable(gTestExe ...)
target_include_directories(gTestExe ...)
target_compile_definitions(gTestExe ...)
target_link_libraries(gTestExe ...)
set (NEWPATH "/path/to/bin;$ENV{PATH}")
STRING(REPLACE ";" "\\;" NEWPATH "${NEWPATH}")
This works:
gtest_add_tests(TARGET gTestExe TEST_LIST allTests)
set_tests_properties(${all_tests} PROPERTIES ENVIRONMENT "PATH=${NEWPATH}")
But this does not:
#set_target_properties(gTestExe PROPERTIES ENVIRONMENT "PATH=${NEWPATH}")
#set_property(DIRECTORY PROPERTY ENVIRONMENT "PATH=${NEWPATH}")
gtest_discover_tests(gTestExe PROPERTIES ENVIRONMENT "PATH=${NEWPATH}")
Edit:
The tests themselves work when added using gtest_add_tests. The issue is the call to discover the tests, during the post build step that gtest_discover_tests registers, fails because the required libraries are not in the PATH.
I came across the same issue this morning and I found a (dirty ?) workaround. The reason why it won't work is a bit complicated, but the workaround is quite simple.
Why it won't work
gtest_discover_tests(gTestExe PROPERTIES ENVIRONMENT "PATH=${NEWPATH}")
Will not work is because the PATH contents are separated by semicolons and therefore are treated by CMake as a list value.
If you look a the GoogleTestAddTests.cmake file (located in C:\Program Files\CMake\share\cmake-3.17\Modules), it treats the PROPERTIES argument with a foreach.
The PROPERTIES value look like this for CMake at this point in the script : ENVIRONMENT;PATH=mypath;mypath2 and will treat mypath2 as a third argument instead of a value for the PATH environment variable.
CMake will then generate the following line :
set_tests_properties( mytest PROPERTIES ENVIRONMENT PATH=mypath mypath2)
Escaping the ; won't work because the list is automatically expended in add_custom_command() in GoogleTest.cmake (l. 420 in cmake 3.17.1) ignoring any form of escaping.
To prevent the cmake foreach to treat each value in the path as a list you can use a bracket argument like :
gtest_discover_tests(gTestExe PROPERTIES ENVIRONMENT "[==[PATH=${NEWPATH}]==]")
The cmake foreach will then treat your argument as one entity. Unfortunately CMake will also put a bracket in the generated code as it contains [ = and maybe spaces :
# This line
if(_arg MATCHES "[^-./:a-zA-Z0-9_]")
set(_args "${_args} [==[${_arg}]==]")
else()
set(_args "${_args} ${_arg}")
endif()
resulting in the following generated script :
set_tests_properties( mytest PROPERTIES ENVIRONMENT [==[ [==[PATH=mypath;mypath2] ]==])
And when executing the test cmake will attempt to read the value only removing the first bracket argument as they don't nest.
Possible workaround
So to do this we need CMake to not use bracket argument on our own bracket argument.
First make a local copy of GoogleTestAddTests.cmake file in your own repository (located in C:\Program Files\CMake\share\cmake-3.17\Modules).
At the beginning of your local copy of GoogleTestAddTests.cmake (l. 12) replace the function add_command by this one :
function(add_command NAME)
set(_args "")
foreach(_arg ${ARGN})
# Patch : allow us to pass a bracket arguments and escape the containing list.
if (_arg MATCHES "^\\[==\\[.*\\]==\\]$")
string(REPLACE ";" "\;" _arg "${_arg}")
set(_args "${_args} ${_arg}")
# end of patch
elseif(_arg MATCHES "[^-./:a-zA-Z0-9_]")
set(_args "${_args} [==[${_arg}]==]")
else()
set(_args "${_args} ${_arg}")
endif()
endforeach()
set(script "${script}${NAME}(${_args})\n" PARENT_SCOPE)
endfunction()
This will make cmake don't use bracket list on our bracket list and automatically escape the ; as set_tests_properties also treat the ; as a list.
Finally we need CMake to use our custom GoogleTestAddTests.cmake instead of the one in CMake.
After your call to include(GoogleTest) set the variable _GOOGLETEST_DISCOVER_TESTS_SCRIPT to the path to your local GoogleTestAddTests.cmake :
# Need google test
include(GoogleTest)
# Use our own version of GoogleTestAddTests.cmake
set(_GOOGLETEST_DISCOVER_TESTS_SCRIPT
${CMAKE_CURRENT_LIST_DIR}/GoogleTestAddTests.cmake
)
Note : In my example the GoogleTestAddTests.cmake is right next to the processing cmake file.
Then a simple call to
gtest_discover_tests(my_target
PROPERTIES ENVIRONMENT "[==[PATH=${my_path};$ENV{PATH}]==]"
)
should work.
I have a CTestList.cmake file containing this function definition and call:
function(add_test_r testname )
add_test(
${testname} python executeRegressionTestCase.py ${testname}
)
endfunction(add_test_r)
add_test_r(Test01)
I need to support testing through CMake directly, cmake ..; make test, and through CTest without running CMake first. When CMake builds the test target and I run make test, the function above executes fine. However, when I run the same file with CTest, ${testname} is empty. Are function variables not supported in CTestList when running with CTest or is there something I'm missing?
I don't have a definitive answer, but after some testing it looks like CMake does a preprocessing step for the input files. Specifically, it reads in CTestList.cmake, evaluates the variables and functions, and generates an "expanded" CTestTestfile.cmake. Running CTest with a steering script does not run the preprocessing step so the variables and functions are not expanded. A few people on the internet have suggested that the CTestTestfile.cmake used in the steering script should be generated by CMake in the first place. Unfortunately, thats not the use case I'm looking for, but it may help someone else with running into this problem.
I'm working on a project using CMake and just integrated some CppUnit tests. I would like to use CTest and thus I used add_test in my CMakeLists.txt files to have the tests executed when typing make test.
Yet I observe that, when typing make test, it says that all the tests passed even if I make a test with trivial errors. Erroneous tests report these errors when executed manually (e.g. ./my_test) but not when executed using make test.
Here is the content of my CMakeLists.txt in the test directory:
add_executable(TestDataSpace TestDataSpace.cpp)
target_link_libraries(TestDataSpace ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestVariableManager TestVariableManager.cpp)
target_link_libraries(TestVariableManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestLayoutManager TestLayoutManager.cpp)
target_link_libraries(TestLayoutManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_test(NAME "TestDataSpace" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestDataSpace)
add_test(NAME "TestVariableManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestVariableManager)
add_test(NAME "TestLayoutManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestLayoutManager)
CTest does find the executables, since putting a wrong path for the command makes CMake complain that it doesn't find them.
make test outputs the following:
Running tests... Test project
Start 1: TestDataSpace 1/3 Test #1: TestDataSpace .................... Passed 0.01 sec
Start 2: TestVariableManager 2/3 Test #2: TestVariableManager .............. Passed 0.02 sec
Start 3: TestLayoutManager 3/3 Test #3: TestLayoutManager ................ Passed 0.01 sec
100% tests passed, 0 tests failed out of 3
What am I missing?
I'm not familiar with CppUnit, but I suspect your executables are always returning 0, even if the test fails. CTest takes a return of 0 to indicate success.
If you change your return value when the test fails to a non-zero number, you should see the expected output from CTest.
Alternatively, you can modify CTest's behaviour by using set_tests_properties to set the values of PASS_REGULAR_EXPRESSION and/or FAIL_REGULAR_EXPRESSION. If either of these are set, the return value is ignored. So for example, you could do:
set_tests_properties(
TestDataSpace
TestVariableManager
TestLayoutManager
PROPERTIES PASS_REGULAR_EXPRESSION "TEST PASSED;Pass")
As an aside, you can avoid passing the full path to the test executables in your case since they are actual CMake targets defined in the same CMakeLists.txt:
add_test(NAME TestDataSpace COMMAND TestDataSpace)
add_test(NAME TestVariableManager COMMAND TestVariableManager)
add_test(NAME TestLayoutManager COMMAND TestLayoutManager)
I have a project that uses fruit for testing (fortran code).
This is my code.
calculator.f90
module calculator
implicit none
contains
subroutine add (a, b, output)
integer, intent(in) :: a, b
integer, intent(out):: output
output = a+b
end subroutine add
end module calculator
And my test calculator_test.f90
module calculator_test
use fruit
contains
subroutine test_5_2_2
use calculator, only: add
integer :: result
call add(2,2,result)
call assertEquals(4,result)
end subroutine test_5_2_2
subroutine test_5_2_3
use calculator, only: add
integer :: result
call add(2,3,result)
call assertEquals(5,result)
end subroutine test_5_2_3
end module
Now I'd like to use Cmake to build and run my tests (triggered by jenkins), so my question is: Do I need to change the tests or is it possible to just run the test I've written through cmake, and if so how?
I've searched a lot online but all testing with cmake seems to be done with c++ and then by using executeable testfiles files.
Thanks!
-Minde
You can run the tests you have written as is, you just need to tell CMake how to run them. This is what the COMMAND argument to ADD_TEST is for.
ENABLE_TESTING()
ADD_TEST(NAME "YourTest"
WORKING_DIRECTORY ${TEST_DIRECTORY}
COMMAND ${TEST_DIRECTORY}/test_dim)
Usually, you see some example like the one above, where the command is an executable (as you have seen in c++ examples). But it doesn't have to be. For example, I am running python tests through CMake, and I add the test like so:
ADD_TEST(NAME PythonTests
WORKING_DIRECTORY ${TEST_DIRECTORY}
COMMAND ${PYTHON_EXECUTABLE} setup.py test
So, to run your Fruit tests, you would call the command that creates the Fruit test runner for you (I believe it is a rake command... I will assume this is true below, but you should substitute for whatever you actually call on the command line to run your tests):
ADD_TEST(NAME FruitTests
WORKING_DIRECTORY ${TEST_DIRECTORY}
COMMAND rake test) # Or whatever the command is.
When you run make test on the command line, it should tell you if "FruitTests" failed or succeeded.
A word of caution CMake determines the success or failure of the test by the exit code of the program. By default Fortran programs do not have a exit code (or if they do, it is always 0). When I use Fruit and CMake to do my Fortran tests, I write the test runner myself and use the call exit(exit_code) builtin subroutine to make sure an exit code is returned to CMake. I am not sure if Fruit's automatic test runner creator does this; you will have to verify this yourself.