I have been reading the CMake tutorial.
The section "Installing and Testing (Step 3)" has the following test script:
add_test (TutorialComp25 Tutorial 25)
set_tests_properties (TutorialComp25
PROPERTIES PASS_REGULAR_EXPRESSION "25 is 5")
According to the CMake documentation, add_test is straightforward, but I don't follow how set_tests_properties works; especially this "25 is 5".
It sounds like
if (INPUT_ARG is OUTPUT_RESULT)
test passed
else
test failed
Is that correct?
From the documentation of PASS_REGULAR_EXPRESSION:
The output must match this regular expression for the test to pass.
If set, the test output will be checked against the specified regular expressions and at least one of the regular expressions has to match, otherwise the test will fail.
The first test of the tutorial example (called TutorialRuns) doesn't have any properties set. This means CTest will treat the test as having passed if the Tutorial exe returns 0 on completion, having been passed the argument 25.
In the second test, the call
set_tests_properties (TutorialComp25 PROPERTIES PASS_REGULAR_EXPRESSION "25 is 5")
means that the console output from running the Tutorial exe with an arg of 25 must contain the string "25 is 5" to be considered a pass. The return value is ignored in this case. Since the test exe outputs:
The square root of 25 is 5
it passes.
Remember that it's a regex that's being applied. If the PASS_REGULAR_EXPRESSION was set to e.g. "^25 is 5", the test would fail, since this is looking for the phrase 25 is 5 at the start of the output. There's a brief description of CMake's regex syntax in the documentation for the string command.
Related
I'm using CMake's 3.18 CTest with the Catch2 3.1.1 framework for testing purposes, and am currently having problems finding what the valid characters are in Catch2's macro TEST_CASE's name parameter.
For instance, I created a test case
TEST_CASE("Use mask '1:2;3:4'"){...}
and ran ctest in the command prompt, I get
1/1 Test 1: Use mask '1:2 ................ Failed 0.09 sec.
If I run ctest with the -V option, I can see it's trying to run test.exe "Use mask '1:2", which returns "No test cases matched.", and therefore fails.
So it looks like the semicolon in the name is messing things up.
Similarly, for test case
TEST_CASE("set prefix \\x"){...}
I get
1/1 Test 1: set prefix \x ................ Failed 0.09 sec.
Running ctest -V, it runs test_2.exe "set prefix \x", which again returns "No test cases matched.".
Removing the underscore in the first test case and double backslash in the second test case fixed the issue. I haven't found a solution online, Catch2 docs say the test case name is a free-form string, but that's about it, which doesn't help me.
I'm using gcov and I'd like to gather coverage results on a per test case basis.
QUESTION
When I execute the googletest executable, is it possible to pass in an argument on command line that says execute only the Nth test case?
I'd even be ok with passing in the name of the fixture+case; I can just use Python with regex to pull out all the tests.
Obviously I can accomplish the same thing by having one test case per .cpp file but that sounds ... stupid.
googletest allows to run a single test case or even the subset of the tests. You can specify a filter string in the GTEST_FILTER environment variable or in the --gtest_filter command line option and googletest will only run the tests whose full names (in the form of TestSuiteName.TestName) match the filter. More information about the format of the filter string can be found in Running a Subset of the Tests section. Also googletest supports --gtest_list_tests command line option to print the list of all test cases. It can be useful in your case.
Due to the following warning:
CMake Error at test/CMakeLists.txt:29 (get_target_property):
The LOCATION property may not be read from target "my_exe". Use the
target name directly with add_custom_command, or use the generator
expression $<TARGET_FILE>, as appropriate.
which is the result from lines like this:
get_target_property(my_exe_path my_exe LOCATION)
Like recommended in the docs, I tried to use a generator expression like this:
add_executable(my_exe_path main.cpp)
message("path to executable: $<TARGET_FILE:my_exe_path>")
But TARGET_FILE is not being evaluated
path to executable: $<TARGET_FILE:my_exe>
I'm using CMake 3.4 and added cmake_minimum_required(VERSION 3.4) to my CMakeLists.txt so what am I doing wrong?
Here is a quick and easy way to print the value of a generator expression:
add_custom_target(print
${CMAKE_COMMAND} -E echo $<1:hello> $<0:world>
)
In this example, if you run cmake . and then make print, you will see "hello" (without the quotation marks) in the output.
However, if you just use message($<1:hello> $<0:world>), you will see "$<1:hello> $<0:world>" as output (again, without the quotation marks).
While generator expression is stored at configuration stage (when corresponded CMake command is executed),
evaluation of generator expressions is performed at build stage.
This is why message() command prints generator expression in non-dereferenced form: value denoted by the generator expression is not known at this stage.
Moreover, CMake never dereferences generator expressions by itself. Instead, it generates appropriate string in the build file, which is then interpreted by build utility (make, Visual Studio, etc.).
Note, that not every CMake command accepts generator expressions. Each possible usage of generator expressions is explicitely described in documentation for specific command. Moreover, different CMake command flows or different options have different policy about using of generator expressions.
For example, command flow
add_test(NAME <name> COMMAND <executable>)
accepts generator expressions for COMMAND option,
but command flow
add_test(<name> <executable>)
doesn't!
Another example of policies difference:
install(DIRECTORY <dir> DESTINATION <dest>)
In this command flow generator expressions are allowed for DESTINATION, but not for DIRECTORY option.
Again, read documentation carefully.
I would like check whether I am in Mac OS X or not, and have the following code
cmake_minimum_required (VERSION 3.0)
project (test)
set (FOO 1)
if (${FOO} AND ${APPLE})
message ("MAC OS X")
endif ()
It failed on non-OSX system with error message
CMake Error at CMakeLists.txt:4 (if):
if given arguments:
"1" "AND"
Unknown arguments specified
If I replace ${APPLE} with APPLE, the error went away. But I am a little puzzled by this. When should we refer to a variable with ${VAR} and when should we not to?
Thanks in advance.
Not 100% relevant but when googling for how to check for OSx in CMake this is the top post. For others who land here asking the same question this worked for me.
if (${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
set(MACOSX TRUE)
endif()
To put it shortly: Everything inside the if parentheses is evaluated as an expression, this is the semantic of the if keyword. So if you put APPLE there, it gets evaluated as a variable name and yields the correct result.
Now if you put ${APPLE} there, ${} will evaluate its contents before if evaluates the expression. Therefore, it's the same as if you'd written
if (1 AND )
(in the case that the variable APPLE isn't set, which is the case on non-OSX systems). This is invalid syntax and yields the error you get. You should write:
if (FOO AND APPLE)
Quoting from the CMake Documentation:
The if command was written very early in CMake’s history, predating the ${} variable evaluation syntax, and for convenience evaluates variables named by its arguments as shown in the above signatures. Note that normal variable evaluation with ${} applies before the if command even receives the arguments. Therefore code like:
set(var1 OFF)
set(var2 "var1")
if(${var2})
appears to the if command as:
if(var1)
and is evaluated according to the if() case documented above. The result is OFF which is false. However, if we remove the ${} from the example then the command sees:
if(var2)
which is true because var2 is defined to “var1” which is not a false constant.
This also works to test for Windows, Mac and *Nix
if(WIN32)
message("https://cmake.org/cmake/help/latest/variable/WIN32.html?highlight=win32")
elseif(APPLE)
message("https://cmake.org/cmake/help/latest/variable/APPLE.html?highlight=apple")
elseif(UNIX)
message("https://cmake.org/cmake/help/latest/variable/UNIX.html?highlight=unix")
endif()
APPLE should be tested before UNIX because UNIX is set to true for Mac OS.
Testing the CMAKE_SYSTEM_NAME variable, as proposed by #Matthew Hogan allows you to test for something more specific.
I'm working on a project using CMake and just integrated some CppUnit tests. I would like to use CTest and thus I used add_test in my CMakeLists.txt files to have the tests executed when typing make test.
Yet I observe that, when typing make test, it says that all the tests passed even if I make a test with trivial errors. Erroneous tests report these errors when executed manually (e.g. ./my_test) but not when executed using make test.
Here is the content of my CMakeLists.txt in the test directory:
add_executable(TestDataSpace TestDataSpace.cpp)
target_link_libraries(TestDataSpace ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestVariableManager TestVariableManager.cpp)
target_link_libraries(TestVariableManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestLayoutManager TestLayoutManager.cpp)
target_link_libraries(TestLayoutManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_test(NAME "TestDataSpace" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestDataSpace)
add_test(NAME "TestVariableManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestVariableManager)
add_test(NAME "TestLayoutManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestLayoutManager)
CTest does find the executables, since putting a wrong path for the command makes CMake complain that it doesn't find them.
make test outputs the following:
Running tests... Test project
Start 1: TestDataSpace 1/3 Test #1: TestDataSpace .................... Passed 0.01 sec
Start 2: TestVariableManager 2/3 Test #2: TestVariableManager .............. Passed 0.02 sec
Start 3: TestLayoutManager 3/3 Test #3: TestLayoutManager ................ Passed 0.01 sec
100% tests passed, 0 tests failed out of 3
What am I missing?
I'm not familiar with CppUnit, but I suspect your executables are always returning 0, even if the test fails. CTest takes a return of 0 to indicate success.
If you change your return value when the test fails to a non-zero number, you should see the expected output from CTest.
Alternatively, you can modify CTest's behaviour by using set_tests_properties to set the values of PASS_REGULAR_EXPRESSION and/or FAIL_REGULAR_EXPRESSION. If either of these are set, the return value is ignored. So for example, you could do:
set_tests_properties(
TestDataSpace
TestVariableManager
TestLayoutManager
PROPERTIES PASS_REGULAR_EXPRESSION "TEST PASSED;Pass")
As an aside, you can avoid passing the full path to the test executables in your case since they are actual CMake targets defined in the same CMakeLists.txt:
add_test(NAME TestDataSpace COMMAND TestDataSpace)
add_test(NAME TestVariableManager COMMAND TestVariableManager)
add_test(NAME TestLayoutManager COMMAND TestLayoutManager)