I'm working on a C++ project with CMake + clang. I would now like to integrate source-based coverage with my unit tests. When compiling with the right flags, raw coverage data is placed into files according to a pattern given by the LLVM_PROFILE_FILE environment variable. Since I'm using catch2 as a test framework I'm thus calling:
catch_discover_tests(
my_test
PROPERTIES ENVIRONMENT LLVM_PROFILE_FILE=coverage/my_test_%p.profraw
)
When running the test target, this will place a .profraw per test process in the coverage directory. I have also added a custom coverage target that merges these files into a .profdata file:
add_custom_command(
OUTPUT coverage/my_test.profdata
COMMAND llvm-profdata merge -sparse coverage/*.profraw -o coverage/my_test.profdata
)
add_custom_target(coverage DEPENDS coverage/my_test.profdata)
This works well enough. However, if I now run the test target again multiple times and forget to clear the coverage directory in between, running the coverage target will merge data from multiple test runs. That's not what I want so I would like to make sure that the coverage directory is always deleted before the tests run. But I'm not sure how to do this, I've tried:
add_custom_target(
clean_coverage_dir
COMMAND ${CMAKE_COMMAND} -E rm -rf coverage
)
add_dependencies(test clean_coverage_dir)
After catch_discover_tests but this results in:
Cannot add target-level dependencies to non-existent target "test".
What can I do? Should I maybe use a different approach altogether?
If anyone is interested, I've found a sort of obvious solution:
add_custom_target(
test_coverage
COMMAND ${CMAKE_COMMAND} -E rm -rf coverage &&
${CMAKE_CTEST_COMMAND} --force-new-ctest-process &&
llvm-profdata merge -sparse coverage/*.profraw -o coverage/my_test.profdata
WORKING_DIRECTORY ${PROJECT_BINARY_DIR}
)
I have a cmake project which produces several executables. I want to package each executable in seperate Docker containers, so inside the Dockefile, I only built the target that I need:
RUN mkdir build \
&& cd build \
&& cmake /app/project -DCMAKE_BUILD_TYPE=Release
&& make -j 2 myExecutable \
&& make install/fast
This works as expected, but I run into an issue with the conan cmake integration. The installation is done when cmake is called, not during the actual build - this means that no matter which target I want to actually build, all the conan installation calls present in my cmake files are called - so way more packages are installed than necessary.
# for every target
# include conan dependencies (each target has its own conanfile.txt)
conan_cmake_run(CONANFILE conanfile.txt
BASIC_SETUP CMAKE_TARGETS
BUILD_TYPE "${CMAKE_BUILD_TYPE}"
BUILD outdated
${update_conan}
)
conan_target_link_libraries(${PROJECT_NAME})
Is there a way to make the cmake calls dependend on which target I actually want to build?
Unfortunately not, the macro conan_cmake_run has no distinction about which target is involved or even it was executed before. You could use CMake options to run or not conan_cmake_run.
Also, you could comment/vote your request thorough the issue https://github.com/conan-io/cmake-conan/issues/105
Regards!
I am using cmake 3.0.2 (on debian 8). I am trying to add some tests using a custom check target like this:
ADD_CUSTOM_TARGET(check
COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1 ${CMAKE_CTEST_COMMAND}
DEPENDS test_1 test_2)
I am using the built-in CMAKE_CTEST_COMMAND which is supposedly available in cmake However, the variable is undefined and the command does not work. My questions are the following:
1: Am I doing it wrong? It seems to work with cmake 3.3.2, when was this variable added?
2: How can I get this to work in cmake 3.0.2, should I just replace CMAKE_CTEST_COMMAND with "ctest"?
I have two sets of tests (functional and unit tests) and I want to be able to specify which set to run through cmake.
One set of tests are my unittests that I want to run by doing "make test".
Another set of tests are my functional tests that I would like to run by doing "make functionaltests".
Currently both are part of ctest, in that I run both suites through add_test. My CMakeLists.txt file is like this:
FOREACH(functional_test ${functional_tests})
ADD_TEST(NAME functional_test COMMAND f_test.sh functional_test)
ENDFOREACH(functional_test)
FOREACH(unit_test ${unit_tests})
ADD_TEST(NAME unit_test COMMAND u_test.sh unit_test)
ENDFOREACH(unit_test)
I want to leverage ctest for both suites because it gives me a nice, readable format for the test suite (which tests passed and which failed).
I would prefer to not have to create a custom executable, create a target called functionaltests for it, and try to mimic how ctest prints out test results.
When you run ctest you can give it a regular expression to choose what tests to run. So you can name the tests in your CMakeLists file according to a pattern which supports this:
set(functional_tests "test1" "test2" "test3")
set(unit_tests "test1" "test2" "test3")
FOREACH(test ${functional_tests})
ADD_TEST(NAME functional_${test} COMMAND f_test.sh ${test})
ENDFOREACH()
FOREACH(test ${unit_tests})
ADD_TEST(NAME unit_${test} COMMAND u_test.sh ${test})
ENDFOREACH()
Now you can run the functional tests:
ctest -R functional_
Or the unit tests:
ctest -R unit_
If you want to create targets so that you can execute these through make, you can do:
add_custom_target(unit_tests COMMAND ${CMAKE_CTEST_COMMAND} -R unit_)
add_custom_target(functional_tests COMMAND ${CMAKE_CTEST_COMMAND} -R functional_)
Then you can run:
make unit_tests
make functional_tests
You may want to add dependencies to the custom commands so that they cause whatever executable you're testing to rebuild if necessary.
I'm using CMake to build my project. I have added a unit test binary which is using the Boost unit testing framework. This one binary contains all of the unit tests. I've added that binary to be run by CTest:
ADD_EXECUTABLE( tftest test-main.cpp )
ENABLE_TESTING()
ADD_TEST( UnitTests tftest)
But the build output in Visual Studio only shows the result of running CTest:
Start 1: UnitTests
1/1 Test #1: UnitTests ................***Failed 0.05 sec
0% tests passed, 1 tests failed out of 1
This is not very helpful, because I can't see which test failed. If I run ctest manually from the command line with --verbose I get the output from a Boost unit test which tells what actually failed:
1: Test command: tftest.exe
1: Test timeout computed to be: 9.99988e+006
1: Running 4 test cases...
1: test-main.cpp(20): error in "sanity_check3": check 1 == 2 failed
1:
1: *** 1 failure detected in test suite "Master Test Suite"
1/1 Test #1: UnitTests ................***Failed 0.00 sec
So, what do I need to change in the CMakeLists.txt to have CTest run with --verbose at all times? Is there a better way to use Boost unit tests with CMake/CTest?
You can use the ctest --output-on-failure option, or set the environment variable CTEST_OUTPUT_ON_FAILURE, which will show you any output from the test program whenever the test fails. One way to do this when using Makefiles and the command line would be as follows:
env CTEST_OUTPUT_ON_FAILURE=1 make check
This Stack Overflow question and answer shows how to set environment variables in Visual Studio.
You could call ctest directly, after cmaking and making your project.
ctest --verbose
There is a very simple solution (which for some reason is difficult to find via Google Search):
ctest --output-on-failure
If you use CMake with Visual Studio's open folder function you can add the
"ctestCommandArgs": "--output-on-failure"
setting to your build configuration.
You can check the Testing/Temporary subfolder. It is automatically created after running make test. This folder contains two files: LastTest.log and LastTestsFailed.log. LastTest.log contains desired output for run tests. LastTestFailed.log contains names of failed tests. So you can check them manually after executing make test.
The second way is to get ctest to show you the content of log files after running tests:
place in build dir (from which you run make test) file CTestCustom.ctest (you can do it with configure file command, for example) with following contents
CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log")
Instead of cat you may use whatever Windows cmd command that does similar things.
run make test again and get profit!
additional info about customizing ctest you can find here. Just step to "Customizing cmake" section.
Good luck!
I had to add "check" target by myself. "make tests" does nothing by some reason. So what I did (as was suggest somewhere on stackoverflow) - I added this target manually. To get verbose output I just wrote it like:
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
make check CTEST_OUTPUT_ON_FAILURE=TRUE
This makes test output more verbose:
make test ARGS="-V"
My approach is a combination of the answers from ony, from zbyszek, and from tarc. I use the ${CMAKE_COMMAND} variable (which is set to the absolute path to the invoked cmake executable) with the -E env CTEST_OUTPUT_ON_FAILURE=1 argument to invoke the actual ctest command using ${CMAKE_CTEST_COMMAND} -C $<CONFIG>. To help clarify what is going on, I start with three cmake -E echo commands to show the current working directory and the ctest command to be invoked. Here is how I call add_custom_target.
add_custom_target(check
${CMAKE_COMMAND} -E echo CWD=${CMAKE_BINARY_DIR}
COMMAND ${CMAKE_COMMAND} -E echo CMD=${CMAKE_CTEST_COMMAND} -C $<CONFIG>
COMMAND ${CMAKE_COMMAND} -E echo ----------------------------------
COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS ALL_BUILD
)
This plays nice with the MSVC IDE where any test errors are shown as clickable compilation errors. See cmake -E env for documentation of the cmake -E portable command line tool mode. I also add a dependency on ALL_BUILD so that all projects will be built before invoking the check target. (On Linux builds, one may need to replace ALL_BUILD with ALL; I have not tested this on Linux yet.)
For people using Visual Studio, here another variation (hack) on the theme:
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build . --target RUN_TESTS
ctest -VV or ctest --extra-verbose
From documentation:
Enable more verbose output from tests.
Test output is normally suppressed and only summary information is
displayed. This option will show even more test output.
There's now a CMake variable that allows you to modify the behaviour of make test. CMAKE_CTEST_ARGUMENTS lets you set a list of arguments to pass to ctest when run via make test.
So adding this to your CMake file:
set(CMAKE_CTEST_ARGUMENTS "--verbose")
Means CTest will always run verbose. Or for just the output of the failed tests, use:
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
Edit:
As suggested by RobLoach, since it's a list of arguments, you'll want to append to the list instead.
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
to show the result with XML file you have to execute the test with the following command
~$ ctest -T Test
and we found the result in the Testing/1234123432/test.xml
and other files are generated too in Testing Folder