I'm working on a project using CMake and just integrated some CppUnit tests. I would like to use CTest and thus I used add_test in my CMakeLists.txt files to have the tests executed when typing make test.
Yet I observe that, when typing make test, it says that all the tests passed even if I make a test with trivial errors. Erroneous tests report these errors when executed manually (e.g. ./my_test) but not when executed using make test.
Here is the content of my CMakeLists.txt in the test directory:
add_executable(TestDataSpace TestDataSpace.cpp)
target_link_libraries(TestDataSpace ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestVariableManager TestVariableManager.cpp)
target_link_libraries(TestVariableManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestLayoutManager TestLayoutManager.cpp)
target_link_libraries(TestLayoutManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_test(NAME "TestDataSpace" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestDataSpace)
add_test(NAME "TestVariableManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestVariableManager)
add_test(NAME "TestLayoutManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestLayoutManager)
CTest does find the executables, since putting a wrong path for the command makes CMake complain that it doesn't find them.
make test outputs the following:
Running tests... Test project
Start 1: TestDataSpace 1/3 Test #1: TestDataSpace .................... Passed 0.01 sec
Start 2: TestVariableManager 2/3 Test #2: TestVariableManager .............. Passed 0.02 sec
Start 3: TestLayoutManager 3/3 Test #3: TestLayoutManager ................ Passed 0.01 sec
100% tests passed, 0 tests failed out of 3
What am I missing?
I'm not familiar with CppUnit, but I suspect your executables are always returning 0, even if the test fails. CTest takes a return of 0 to indicate success.
If you change your return value when the test fails to a non-zero number, you should see the expected output from CTest.
Alternatively, you can modify CTest's behaviour by using set_tests_properties to set the values of PASS_REGULAR_EXPRESSION and/or FAIL_REGULAR_EXPRESSION. If either of these are set, the return value is ignored. So for example, you could do:
set_tests_properties(
TestDataSpace
TestVariableManager
TestLayoutManager
PROPERTIES PASS_REGULAR_EXPRESSION "TEST PASSED;Pass")
As an aside, you can avoid passing the full path to the test executables in your case since they are actual CMake targets defined in the same CMakeLists.txt:
add_test(NAME TestDataSpace COMMAND TestDataSpace)
add_test(NAME TestVariableManager COMMAND TestVariableManager)
add_test(NAME TestLayoutManager COMMAND TestLayoutManager)
Related
I'm using CMake's 3.18 CTest with the Catch2 3.1.1 framework for testing purposes, and am currently having problems finding what the valid characters are in Catch2's macro TEST_CASE's name parameter.
For instance, I created a test case
TEST_CASE("Use mask '1:2;3:4'"){...}
and ran ctest in the command prompt, I get
1/1 Test 1: Use mask '1:2 ................ Failed 0.09 sec.
If I run ctest with the -V option, I can see it's trying to run test.exe "Use mask '1:2", which returns "No test cases matched.", and therefore fails.
So it looks like the semicolon in the name is messing things up.
Similarly, for test case
TEST_CASE("set prefix \\x"){...}
I get
1/1 Test 1: set prefix \x ................ Failed 0.09 sec.
Running ctest -V, it runs test_2.exe "set prefix \x", which again returns "No test cases matched.".
Removing the underscore in the first test case and double backslash in the second test case fixed the issue. I haven't found a solution online, Catch2 docs say the test case name is a free-form string, but that's about it, which doesn't help me.
Difference between go test's two flags -parallel and -test.parallel and which flag gets precedence?
-parallel n
Allow parallel execution of test functions that call t.Parallel.
The value of this flag is the maximum number of tests to run
simultaneously; by default, it is set to the value of GOMAXPROCS.
Note that -parallel only applies within a single test binary.
The 'go test' command may run tests for different packages
in parallel as well, according to the setting of the -p flag
(see 'go help build').
Above documentation says that the number of tests that are run in parallel are equal to GOMAXPROCS if nothing is provided, but the behavior is not like that for me. Because I am running tests on a machine that has only 4 cores. But for me 8 tests run in parallel, so the behavior is more like following:
-test.parallel int
maximum test parallelism (default 8)
So what is the difference between the two? When to use which flag.
More Information
I am running all tests on a single package which has 9 tests, all of them are run parallely and all those exist in single test function.
The -test. flags are generated by go test command. The go test command produces a pkg.test binary on the fly and runs it with modified arguments. All the recognized arguments passed to go test will be converted. So, in your case: -parallel n becomes -test.parallel n.
So this command:
go test -parallel 6
creates:
pkg.test -test.parallel 6
I have a CTestList.cmake file containing this function definition and call:
function(add_test_r testname )
add_test(
${testname} python executeRegressionTestCase.py ${testname}
)
endfunction(add_test_r)
add_test_r(Test01)
I need to support testing through CMake directly, cmake ..; make test, and through CTest without running CMake first. When CMake builds the test target and I run make test, the function above executes fine. However, when I run the same file with CTest, ${testname} is empty. Are function variables not supported in CTestList when running with CTest or is there something I'm missing?
I don't have a definitive answer, but after some testing it looks like CMake does a preprocessing step for the input files. Specifically, it reads in CTestList.cmake, evaluates the variables and functions, and generates an "expanded" CTestTestfile.cmake. Running CTest with a steering script does not run the preprocessing step so the variables and functions are not expanded. A few people on the internet have suggested that the CTestTestfile.cmake used in the steering script should be generated by CMake in the first place. Unfortunately, thats not the use case I'm looking for, but it may help someone else with running into this problem.
I am trying to get CMake/CTest to work. My problem is that CTest does not seem to pick up a test executable unless it is named tests.
In my minimal example project I have a single CMakeLists.txt file:
cmake_minimum_required(VERSION 3.0)
project(myproject C)
enable_testing()
set(TEST_EXE_NAME tests)
add_executable(${TEST_EXE_NAME} test_main.c)
add_test(NAME "My tests" COMMAND ${TEST_EXE_NAME})
...and a very simple test program, test_main.c, that always passes:
int main() {
return 0;
}
I can run make && make test, and all is fine as long as TEST_EXE_NAME is set to tests. However, when I change the executable name to something else, e.g. mytests, I get the following error:
Could not find executable tests
Looked in the following places:
tests
tests
Release/tests
Release/tests
Debug/tests
...
What am I missing?
According to the CMake manual for add_test() the test name may not contain spaces:
The test name may not contain spaces, quotes, or other
characters special in CMake syntax.
In the problematic example, the test was named "My tests". Changing the test name to "My_tests" solves the problem.
Apparently, the part of the test name after the space character (i.e. "tests") was interpreted as the test executable name.
I have a project that uses fruit for testing (fortran code).
This is my code.
calculator.f90
module calculator
implicit none
contains
subroutine add (a, b, output)
integer, intent(in) :: a, b
integer, intent(out):: output
output = a+b
end subroutine add
end module calculator
And my test calculator_test.f90
module calculator_test
use fruit
contains
subroutine test_5_2_2
use calculator, only: add
integer :: result
call add(2,2,result)
call assertEquals(4,result)
end subroutine test_5_2_2
subroutine test_5_2_3
use calculator, only: add
integer :: result
call add(2,3,result)
call assertEquals(5,result)
end subroutine test_5_2_3
end module
Now I'd like to use Cmake to build and run my tests (triggered by jenkins), so my question is: Do I need to change the tests or is it possible to just run the test I've written through cmake, and if so how?
I've searched a lot online but all testing with cmake seems to be done with c++ and then by using executeable testfiles files.
Thanks!
-Minde
You can run the tests you have written as is, you just need to tell CMake how to run them. This is what the COMMAND argument to ADD_TEST is for.
ENABLE_TESTING()
ADD_TEST(NAME "YourTest"
WORKING_DIRECTORY ${TEST_DIRECTORY}
COMMAND ${TEST_DIRECTORY}/test_dim)
Usually, you see some example like the one above, where the command is an executable (as you have seen in c++ examples). But it doesn't have to be. For example, I am running python tests through CMake, and I add the test like so:
ADD_TEST(NAME PythonTests
WORKING_DIRECTORY ${TEST_DIRECTORY}
COMMAND ${PYTHON_EXECUTABLE} setup.py test
So, to run your Fruit tests, you would call the command that creates the Fruit test runner for you (I believe it is a rake command... I will assume this is true below, but you should substitute for whatever you actually call on the command line to run your tests):
ADD_TEST(NAME FruitTests
WORKING_DIRECTORY ${TEST_DIRECTORY}
COMMAND rake test) # Or whatever the command is.
When you run make test on the command line, it should tell you if "FruitTests" failed or succeeded.
A word of caution CMake determines the success or failure of the test by the exit code of the program. By default Fortran programs do not have a exit code (or if they do, it is always 0). When I use Fruit and CMake to do my Fortran tests, I write the test runner myself and use the call exit(exit_code) builtin subroutine to make sure an exit code is returned to CMake. I am not sure if Fruit's automatic test runner creator does this; you will have to verify this yourself.