Suppose in CMakeLists.txt I have
add_executable(mytarget main.cpp)
enable_testing()
add_test(mytarget_test0 mytarget -option0)
Is there any easy way how can I run mytarget in GDB with all command line options from some particular CTest test? (Other than searching for test in CMakeLists and then copy-pasting add_test parameters to command line manually?)
Real life scenario: I run all tests using ctest, one fails, I want to open it in debugger quickly.
In other build systems there are command line parameters to use gdb, for example in Meson meson test --gdb testname , in bazel bazel --run_under=gdbserver. I did not found anything similar for CTest
It is possible to get test command with arguments:
ctest -R $regex_matching_test -V -N
As output you will get something like:
Test project ../cmake-build-debug-gcc7
Constructing a list of tests
Done constructing a list of tests
1: Test command: ../cmake-build-debug-gcc7/my_tool "-v" "test0"
Test #1: my_tool_test0
Total Tests: 1
Then using regexp it is possible to grab command line args for gdb
I use the following procedure:
make clean
make # Build target and all unit tests
make test # Run all unit tests: make test
You will get an result list and some output, depending of the unit test frame work. Fix the issue and re-run the failed test within gdb until it succeeds. Restart the whole procedure.
In case you are using an IDE (e.g. QtCreator) it is easy to debug through the failed test case.
Related
I'm trying to build some tests with googleTest which should test some MPI parallel code. Ideally I want our CI server to execute them through ctest.
My naive approach was to simply call ctest with MPI:
mpirun -n 3 ctest
However, this results in even trivial tests failing as long as at least two are executed.
Example of a trivial tests:
TEST(TestSuite, DummyTest1) {
EXPECT_TRUE(true);
}
TEST(TestSuite, DummyTest2) {
EXPECT_TRUE(true);
}
Am I supposed to launch ctest in some other way? Or do I have to approach this completely differently?
Additional info:
The tests are added via gtest_discover_tests().
Launching the test executable directly with MPI (mpirun -n 3 testExe) yields successful tests but I would prefer using ctest.
Version info:
googletest: 1.11.0
MPI: OpenMPI 4.0.3 AND MPICH 3.3.2
According to this CMake Forum post, which links to a Merge Requests and an issue in the CMake GitLab repository, gtest_discover_tests() currently simply doesn't support MPI.
A possible workaround is to abuse the CROSSCOMPILING_EMULATOR property to inject the wrapper into the test command. However, this changes the whole target and not only the tests so your mileage may vary.
set_property(TARGET TheExe PROPERTY CROSSCOMPILING_EMULATOR '${MPIEXEC_EXECUTABLE} ${MPIEXEC_NUMPROC_FLAG} 3')
gtest_discover_tests(TheExe)
See the forum post for a full minimal example.
In CMake, one can use add_subdirectory(foo EXCLUDE_FROM_ALL) to exclude the targets defined under foo from being built by default unless they are needed by some other target, like this:
add_subdirectory(foo EXCLUDE_FROM_ALL)
add_custom_target(wanted_foo_targets ALL DEPENDS
bar_tool # only build bar_tool from foo
)
Suppose foo defines two targets, bar_tool and foo_tool, and tests for each:
add_test(NAME test_foo_tool COMMAND foo_tool)
add_test(NAME test_bar_tool COMMAND bar_tool)
When you build the overall project, foo_tool is not built, because it is not a dependency of anything, but bar_tool is built because it is a dependency of wanted_foo_targets.
But CTest still registers the test_foo_tool tool, which now fails because foo_tool doesn't exist.
Can I remove/hide/skip tests that refer to targets that weren't built? I know I could make the test command something like [ -f foo_tool ] && foo_tool but this will erroneously state that the test passed.
Bonus points if I can do this from the parent directory, without modifying how the tests are set up inside foo. In my case, foo is a project provided by a third party, and I don't want to have to maintain a patch of their code as they write new tests.
This can be done by using the SKIP_RETURN_CODE test property in combination with a small test driver script (which you have already considered using).
In the source code directory add a test driver script named test_driver.sh with the following contents:
#!/bin/sh
if [ -x "$1" ] ; then exec "$1" ; else exit 127 ; fi
The script checks for existence of a test executable passed as the first argument and exits with error code 127 (aka "command not found") if it does not exist.
In your CMakeLists.txt add the tests in the following way:
add_test(NAME test_foo_tool COMMAND "${CMAKE_SOURCE_DIR}/test_driver.sh" $<TARGET_FILE:foo_tool>)
set_tests_properties(test_foo_tool PROPERTIES SKIP_RETURN_CODE 127)
When you run the tests with the executable foo_tool missing, ctest will report the test_foo_tool as a skipped test but not as a failed test.
$ ctest
...
Start 1: test_foo_tool
1/2 Test #1: test_foo_tool ....................***Skipped 0.01 sec
Start 2: test_bar_tool
2/2 Test #2: test_bar_tool .................... Passed 0.01 sec
100% tests passed, 0 tests failed out of 2
Total Test time (real) = 0.03 sec
The following tests did not run:
1 - test_foo_tool (Skipped)
Tests can be disabled by setting the DISABLED property of the test in question.
You can also get a list of tests defined in a sub-directory by checking the TESTS property.
So you should be able to use get_property to get a lists of all the tests for that specific directory and then disable all of them. If only a sub-set of the tests needs to be disabled then filter the list of all tests, using list(FILTER) to just those that need to be disabled. This is only convenient if there is a naming convention that is followed.
Unfortunately set_property can only modify the property of a test within the same directory not a sub-directory. This means that the CMakeLists.txt file would need be updated.
Alternatively if modifying the CMakeLists.txt is too much of an issue just don't run those tests. That could be done by using the ctest -E regex option to exclude those tests.
The boost test library supports various logging options, for example: --log_level=all or log_level=test_suite.
And the ctest test driver program has the option --output-on-failure which nicely outputs the messages from boost test.
Is it possible to enable the above output for passed tests too?
It will give a little more than what you're after, but you can add the --verbose flag to ctest and it will show the output of all tests whether they pass or fail.
Although its an old quetion, but could help to answer it.
yes its possible to putput both failed and passed test with -O <file>, --output-log <file>
-O <file>, --output-log <file>
Output to log file.
This option tells CTest to write all its output to a log file.
https://cmake.org/cmake/help/latest/manual/ctest.1.html
I need to run python units test cases as part of bamboo build step and the build needs to fail if unit tests fail.
For this, I have a Script step in bamboo build and i am trying to run the following in it:
python -m unittest discover /test
Here, /test folder has all the unit tests.
The output of the above script it
Ran (0) tests
So the problem is that bamboo isn't able to discover these tests. Bamboo agent is linux.
Wondering if anyone has done such a thing before and has any suggestions.
The following worked. Used -p (pattern) attribute to discover/run the unit tests in bamboo (unix build agent)
python -m unittest discover -s test -p "T*.py"
Note: 1. all my test cases start with "T" e.g. Test_check.py
2. "test" is the package where all my test cases are.
If you haven't figured it out, likely because in windows filenames aren't case sensitive but in Linux they are...
And you're test file named Test_xxxx.py isn't the same as test_xxxx.py which is the pattern that discovery is trying to use...
How do I explicitly say with my go test command to run only tests for the main package and not others in my source directory.
At the moment it's working with $go test -v. But... I am using goconvey as well and it seems to be running recursively. According to this page https://github.com/smartystreets/goconvey/wiki/Profiles I have a file where I can pass arguments into the go test command. I know you can go test -v ./... for recursive or go test -c packagename/... but how do I just do it for the main?
Profiles is one to accomplish this, but you can also specify a 'depth' for the runner:
$ goconvey -depth=0
A value of 0 limits the runner to the working directory.
Run goconvey -help for details.