When using CMake + CTest + CDash can I have an optional test fixture? - cmake

Question

I am using CMake FIXTURES_SETUP/FIXTURES_REQUIRED to only run tests if an external resource is available. However, the external resource is optional (available on some test machines, but not others) so when the fixture that checks for the resource fails I do not want to consider the tests suite to have failed, I simply do not want to run any tests that require the fixture. Is there a way to mark a test fixture as 'allowed to fail'. I know there is WILL_FAIL but this inverts the sense of the test so that it would be marked as failure when it passes.

Answer 1

No. When you add a test that means you expect that test to pass. When a prerequisite of that test fails, CMake skips that test (does not actually run it) and counts it as failed because it did not succeed.
e.g.
# CMakeLists.txt
cmake_minimum_required(VERSION 3.3)
project(example)
enable_testing()
add_test(NAME failIfUnavail COMMAND false)
add_test(NAME dependentTest1 COMMAND true)
add_test(NAME dependentTest2 COMMAND true)
add_test(NAME cleaner COMMAND true)
set_tests_properties(failIfUnavail PROPERTIES FIXTURES_SETUP example_case)
set_tests_properties(dependentTest1 dependentTest2 PROPERTIES FIXTURES_REQUIRED example_case)
set_tests_properties(cleaner PROPERTIES FIXTURES_CLEANUP example_case)
$ cmake -H. -Bbuild
-- The C compiler identification is GNU 8.2.0
-- The CXX compiler identification is GNU 8.2.0
-- Check for working C compiler: /bin/gcc
-- Check for working C compiler: /bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /bin/g++
-- Check for working CXX compiler: /bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /build
$ cmake --build build/ --target test
Running tests...
Test project /home/pamini/cmake_test1/build
Start 1: failIfUnavail
1/4 Test #1: failIfUnavail ....................***Failed 0.00 sec
Start 2: dependentTest1
Failed test dependencies: failIfUnavail
2/4 Test #2: dependentTest1 ...................***Not Run 0.00 sec
Start 3: dependentTest2
Failed test dependencies: failIfUnavail
3/4 Test #3: dependentTest2 ...................***Not Run 0.00 sec
Start 4: cleaner
4/4 Test #4: cleaner .......................... Passed 0.00 sec
25% tests passed, 3 tests failed out of 4
Total Test time (real) = 0.02 sec
The following tests FAILED:
1 - failIfUnavail (Failed)
2 - dependentTest1 (Not Run)
3 - dependentTest2 (Not Run)
Errors while running CTest
gmake: *** [test] Error 8
What you can do is:
If the resource's availability is known at configuration time then check and only add the tests when you expect them to pass.
If the resource's availability is not known at configuration time then modify the tests themselves to account for this uncertainty. For example, unit tests avoid this issue by using mock objects that imitate the behavior of the resources they depend on.