I'm using CMake's 3.18 CTest with the Catch2 3.1.1 framework for testing purposes, and am currently having problems finding what the valid characters are in Catch2's macro TEST_CASE's name parameter.
For instance, I created a test case
TEST_CASE("Use mask '1:2;3:4'"){...}
and ran ctest in the command prompt, I get
1/1 Test 1: Use mask '1:2 ................ Failed 0.09 sec.
If I run ctest with the -V option, I can see it's trying to run test.exe "Use mask '1:2", which returns "No test cases matched.", and therefore fails.
So it looks like the semicolon in the name is messing things up.
Similarly, for test case
TEST_CASE("set prefix \\x"){...}
I get
1/1 Test 1: set prefix \x ................ Failed 0.09 sec.
Running ctest -V, it runs test_2.exe "set prefix \x", which again returns "No test cases matched.".
Removing the underscore in the first test case and double backslash in the second test case fixed the issue. I haven't found a solution online, Catch2 docs say the test case name is a free-form string, but that's about it, which doesn't help me.
Related
I'm using gcov and I'd like to gather coverage results on a per test case basis.
QUESTION
When I execute the googletest executable, is it possible to pass in an argument on command line that says execute only the Nth test case?
I'd even be ok with passing in the name of the fixture+case; I can just use Python with regex to pull out all the tests.
Obviously I can accomplish the same thing by having one test case per .cpp file but that sounds ... stupid.
googletest allows to run a single test case or even the subset of the tests. You can specify a filter string in the GTEST_FILTER environment variable or in the --gtest_filter command line option and googletest will only run the tests whose full names (in the form of TestSuiteName.TestName) match the filter. More information about the format of the filter string can be found in Running a Subset of the Tests section. Also googletest supports --gtest_list_tests command line option to print the list of all test cases. It can be useful in your case.
I'm tinkering with this project where Step 6 requires me to run a command like make db-prepare-artix7. This command corresponds to this section of the Makefile. I am confused by the #+SKIP_ENV=true in the recipe. What is #+SKIP_ENV here, and what does it do? Couldn't find anywhere referring to SKIP_ENV.
Thanks!
Explaining every part:
The # means the command will not be echoed by Make during recipe execution
The + means the command will be executed even during dry runs: make --dry-run ...
The SKIP_ENV=true is sh(ell) syntax for setting the environment variable SKIP_ENV to the string true for the duration of the command that follows
In your case the source ... command
The effect of SKIP_ENV depends on the command - dig deeper to find out
Difference between go test's two flags -parallel and -test.parallel and which flag gets precedence?
-parallel n
Allow parallel execution of test functions that call t.Parallel.
The value of this flag is the maximum number of tests to run
simultaneously; by default, it is set to the value of GOMAXPROCS.
Note that -parallel only applies within a single test binary.
The 'go test' command may run tests for different packages
in parallel as well, according to the setting of the -p flag
(see 'go help build').
Above documentation says that the number of tests that are run in parallel are equal to GOMAXPROCS if nothing is provided, but the behavior is not like that for me. Because I am running tests on a machine that has only 4 cores. But for me 8 tests run in parallel, so the behavior is more like following:
-test.parallel int
maximum test parallelism (default 8)
So what is the difference between the two? When to use which flag.
More Information
I am running all tests on a single package which has 9 tests, all of them are run parallely and all those exist in single test function.
The -test. flags are generated by go test command. The go test command produces a pkg.test binary on the fly and runs it with modified arguments. All the recognized arguments passed to go test will be converted. So, in your case: -parallel n becomes -test.parallel n.
So this command:
go test -parallel 6
creates:
pkg.test -test.parallel 6
I have been reading the CMake tutorial.
The section "Installing and Testing (Step 3)" has the following test script:
add_test (TutorialComp25 Tutorial 25)
set_tests_properties (TutorialComp25
PROPERTIES PASS_REGULAR_EXPRESSION "25 is 5")
According to the CMake documentation, add_test is straightforward, but I don't follow how set_tests_properties works; especially this "25 is 5".
It sounds like
if (INPUT_ARG is OUTPUT_RESULT)
test passed
else
test failed
Is that correct?
From the documentation of PASS_REGULAR_EXPRESSION:
The output must match this regular expression for the test to pass.
If set, the test output will be checked against the specified regular expressions and at least one of the regular expressions has to match, otherwise the test will fail.
The first test of the tutorial example (called TutorialRuns) doesn't have any properties set. This means CTest will treat the test as having passed if the Tutorial exe returns 0 on completion, having been passed the argument 25.
In the second test, the call
set_tests_properties (TutorialComp25 PROPERTIES PASS_REGULAR_EXPRESSION "25 is 5")
means that the console output from running the Tutorial exe with an arg of 25 must contain the string "25 is 5" to be considered a pass. The return value is ignored in this case. Since the test exe outputs:
The square root of 25 is 5
it passes.
Remember that it's a regex that's being applied. If the PASS_REGULAR_EXPRESSION was set to e.g. "^25 is 5", the test would fail, since this is looking for the phrase 25 is 5 at the start of the output. There's a brief description of CMake's regex syntax in the documentation for the string command.
I'm working on a project using CMake and just integrated some CppUnit tests. I would like to use CTest and thus I used add_test in my CMakeLists.txt files to have the tests executed when typing make test.
Yet I observe that, when typing make test, it says that all the tests passed even if I make a test with trivial errors. Erroneous tests report these errors when executed manually (e.g. ./my_test) but not when executed using make test.
Here is the content of my CMakeLists.txt in the test directory:
add_executable(TestDataSpace TestDataSpace.cpp)
target_link_libraries(TestDataSpace ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestVariableManager TestVariableManager.cpp)
target_link_libraries(TestVariableManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestLayoutManager TestLayoutManager.cpp)
target_link_libraries(TestLayoutManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_test(NAME "TestDataSpace" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestDataSpace)
add_test(NAME "TestVariableManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestVariableManager)
add_test(NAME "TestLayoutManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestLayoutManager)
CTest does find the executables, since putting a wrong path for the command makes CMake complain that it doesn't find them.
make test outputs the following:
Running tests... Test project
Start 1: TestDataSpace 1/3 Test #1: TestDataSpace .................... Passed 0.01 sec
Start 2: TestVariableManager 2/3 Test #2: TestVariableManager .............. Passed 0.02 sec
Start 3: TestLayoutManager 3/3 Test #3: TestLayoutManager ................ Passed 0.01 sec
100% tests passed, 0 tests failed out of 3
What am I missing?
I'm not familiar with CppUnit, but I suspect your executables are always returning 0, even if the test fails. CTest takes a return of 0 to indicate success.
If you change your return value when the test fails to a non-zero number, you should see the expected output from CTest.
Alternatively, you can modify CTest's behaviour by using set_tests_properties to set the values of PASS_REGULAR_EXPRESSION and/or FAIL_REGULAR_EXPRESSION. If either of these are set, the return value is ignored. So for example, you could do:
set_tests_properties(
TestDataSpace
TestVariableManager
TestLayoutManager
PROPERTIES PASS_REGULAR_EXPRESSION "TEST PASSED;Pass")
As an aside, you can avoid passing the full path to the test executables in your case since they are actual CMake targets defined in the same CMakeLists.txt:
add_test(NAME TestDataSpace COMMAND TestDataSpace)
add_test(NAME TestVariableManager COMMAND TestVariableManager)
add_test(NAME TestLayoutManager COMMAND TestLayoutManager)