Continue after dependent target fails in custom CMake target - testing

I have a regression test suite consisting of multiple custom targets created with add_custom_target().
Moreover there is a "convenience" target regressions to run all regressions. It simply contains all single regression targets as dependency.
This is reflected in the following MCVE:
cmake_minimum_required(VERSION 3.17)
project (Regressions)
add_custom_target(reg_1 COMMAND ${CMAKE_COMMAND} -E echo 'First regression')
add_custom_target(reg_2 COMMAND ${CMAKE_COMMAND} -E echo 'Second regression')
# ...
add_custom_target(regressions DEPENDS reg_1 reg_2)
Now I can run cmake --build . --target regressions from the build directory and reg_1 and reg_2 are run as part of regressions.
My problem is that if one of the regressions fail, the remaining are not executed. But of course I want to always run all regressions and only have a summary of the failed ones. How can I achieve this behavior, i.e. always execute all subtargets, no matter whether some of them fail?
I assume that the natural way to do this is to use add_test() (after all regressions runs are tests), but I failed because the custom targets are no executables and AFAIK you cannot use custom CMake targets with add_test().
Please feel free to recommend an alternative to my current approach. If I could handle everything using ctest that would be preferred anyway.

Thanks to #KamilCuk's answer, I realized my problem of not being able to add custom targets as tests is not really a problem.
I can invoke CMake with add_test(), and the cmake command can run custom targets.
Adding:
enable_testing()
add_test(NAME regression1 COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target reg_1)
add_test(NAME regression2 COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target reg_2)
to the code above solves the problem. Additionally, instead of depending on the single custom targets, regression can simply invoke ctest -R "regression*" to invoke all (and only) regression targets, in case other tests exist in the CMake project, like this:
add_custom_target(regressions COMMAND ${CMAKE_CTEST_COMMAND} -R "regression*")

Related

What cmake command will copy a directory of files to a directory in a post build step

I have a set of resource files that have nothing to do with the build steps of GCC or some other compiler's output. I need them copied from a folder in my project to the cmake build output folder. The goal is for the executable, when run from the build output folder, can see the resources.
How do people typically copy and install resources in cmake builds? Additionally, I want them copied regardless of changes in the build and I want it executed every time I run some cmake command, like build. See below for what I tried to solve this issue.
For example:
I have a bunch of shader files that I want copied. All shaders/* files should be copied into a directory in the build output called "shaders", because that's where the executable for the program lives.
file(GLOB out shaders/*)
foreach (o ${out})
message("${o} was copied to shaders")
file(COPY ${o} DESTINATION shaders)
endforeach ()
This only works sometimes, like when I reload the CMake project, e.g.:
/opt/clion-2021.2.3/bin/cmake/linux/bin/cmake \
-DCMAKE_BUILD_TYPE=Debug -DCMAKE_DEPENDS_USE_COMPILER=FALSE \
-G "CodeBlocks - Unix Makefiles" \
/home/hack/glad
Also, it doesn't execute "POST_BUILD", so the lastest version of the shaders/a.vert file doesn't get copied to the shaders/ directory in the output.
I tried using this, too, but it gave me some headaches:
add_custom_command(TARGET my-executable POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy shaders/* shaders)
I think there's something incorrect with that above, because it wasn't run every POST_BUILD if the build's code didn't change. I don't care if the build's code doesn't change because the files in shaders/* could have changed and should be copied regardless of cmake determining if there was a change in my-executable.
This gist on github was very helpful, here but the gist that applies to my question is included below.
add_custom_target(bar
COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/shaders
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/shaders ${CMAKE_BINARY_DIR}/shaders
COMMENT "copying ${CMAKE_SOURCE_DIR}/shaders to ${CMAKE_BINARY_DIR}/shaders"
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
)
The above code creates a bar cmake target that can be run separately with Make bar; then, if you add another target that requires those resources (the shaders above are resources for this other executable) then you can tie the dependency together using add_dependencies like this:
add_executable(a
main.c
opengl.c)
add_dependencies(a bar)
Now, every time the a target is run, the bar target is run as well, which has the effect of creating the directory and copying the files.
This was good enough for me, but to finish up, you can use this to create the files in a post build step after the other dependency is finished running:
add_custom_command(TARGET bar
# Run after all other rules within the target have been executed
POST_BUILD
COMMAND echo "executing a POST_BUILD command"
COMMENT "This command will be executed after building bar"
VERBATIM
)
Note that ${CMAKE_COMMAND} in the above examples of add_custom_command is a variable that points to the cmake executable, and you can run cmake -E to see the very helpful list of commands that come with cmake.
YIKES the post build step is only running after bar's target is built, not a's target. Hopefully, somebody can help me answer this better. I would still like to know how to copy files after a target is built, unless that's completely unnecessary and nobody should ever want to do that.

CMake: clean coverage directory before test target

I'm working on a C++ project with CMake + clang. I would now like to integrate source-based coverage with my unit tests. When compiling with the right flags, raw coverage data is placed into files according to a pattern given by the LLVM_PROFILE_FILE environment variable. Since I'm using catch2 as a test framework I'm thus calling:
catch_discover_tests(
my_test
PROPERTIES ENVIRONMENT LLVM_PROFILE_FILE=coverage/my_test_%p.profraw
)
When running the test target, this will place a .profraw per test process in the coverage directory. I have also added a custom coverage target that merges these files into a .profdata file:
add_custom_command(
OUTPUT coverage/my_test.profdata
COMMAND llvm-profdata merge -sparse coverage/*.profraw -o coverage/my_test.profdata
)
add_custom_target(coverage DEPENDS coverage/my_test.profdata)
This works well enough. However, if I now run the test target again multiple times and forget to clear the coverage directory in between, running the coverage target will merge data from multiple test runs. That's not what I want so I would like to make sure that the coverage directory is always deleted before the tests run. But I'm not sure how to do this, I've tried:
add_custom_target(
clean_coverage_dir
COMMAND ${CMAKE_COMMAND} -E rm -rf coverage
)
add_dependencies(test clean_coverage_dir)
After catch_discover_tests but this results in:
Cannot add target-level dependencies to non-existent target "test".
What can I do? Should I maybe use a different approach altogether?
If anyone is interested, I've found a sort of obvious solution:
add_custom_target(
test_coverage
COMMAND ${CMAKE_COMMAND} -E rm -rf coverage &&
${CMAKE_CTEST_COMMAND} --force-new-ctest-process &&
llvm-profdata merge -sparse coverage/*.profraw -o coverage/my_test.profdata
WORKING_DIRECTORY ${PROJECT_BINARY_DIR}
)

Should I use ctest dashboard if I don't use CDash?

I'm redefining my project's cmake configuration based on what I have learned from Daniel Pfeifer's “Effective CMake" talks, but I still cannot get my head around ctest.
Currently I have the following setup for my tests:
enable_testing()
find_program(MEMCHECK_COMMAND valgrind)
# executable
add_executable(myTest ${SRC_LIST})
target_link_libraries(myTest ${LIB_LIST})
# test
add_test(NAME myTest_test
COMMAND ${CMAKE_CURRENT_BINARY_DIR}/myTest)
add_custom_target(myTest_test
COMMAND ctest -R myTest_test --output-on-failure
DEPENDS myTest)
# memcheck
add_test(NAME myTest_memcheck
COMMAND ${MEMCHECK_COMMAND} ${CMAKE_CURRENT_BINARY_DIR}/myTest)
add_custom_target(myTest_memcheck
COMMAND ctest -R myTest_memcheck --output-on-failure
DEPENDS ${TEST_NAME})
and I could run make myTest_test and make myTest_memcheck and know that I didn't break anything.
It pains me that I have to do this for every test executable (and project is big so I have dozens of them) and even extracted as a function it just feels wrong.
I have read ctest docs (https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Testing-With-CTest) and dashboards seems almost perfect to me - if I could just run ctest -R myTest -T test, ctest -R myTest -T memcheck and ctest -R myTest -T coverage I would be very happy :) (and even more happy if those commands will trigger build if needed).
The problem is that include(CTest) creates targets that I do not need (Continuous/Nightly dashboards with Start/Update/Configure/Submit steps). It will be used solely as experimental builds and executed by devs during coding. For CI I have Jenkins with it's own magic. I would like to end up with:
- Dashboard/Build
- Dashboard/Test
- Dashboard/MemCheck
- Dashboard/Coverage
What should I do? Should I create manually my own dashboard (if it is possible) or drop this idea and stay with my custom targets?

CMake add_custom_command ('POST_BUILD') 'DEPENDS' option is ignored

I have a library and a test projects on CMake, and I'm using this directory structure with two (project) CMakeLists.txt:
/
|- CMakeLists.txt
|- include/libName
|- src/...
|
|- test/
|- CMakeLists.txt
|- src/...
The outer project list defines the library, like:
add_library(libName ${SRC} ${INCLUDE})
And adds 'test' as subdirectory:
add_subdirectory(test)
The test project list defines the executable and a test, like:
add_executable(NameTest ${SRC})
target_link_libraries(NameTest libName)
add_test(NAME NameTest COMMAND NameTest)
The problem
I'm trying to build and execute the test program when the library is built. If any test fails, I want the build of the library fail too.
This is what I have (inside the outer lists file):
add_custom_command(
TARGET libName
POST_BUILD
COMMAND CTEST_OUTPUT_ON_FAILURE=1 ctest
DEPENDS NameTest # <- This is driving me crazy!
)
This command ignores completely if the target 'NameTest' is built, if there is a file with that name, or if not. I can't notice any difference if the whole 'DEPENDS' option is removed.
I even modified like:
add_custom_command(
TARGET libName
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E echo "Bip! Bip! Bip!"
DEPENDS this_is_not_an_existent_file_nor_target
)
And the command is triggered anyway. I'm not very sure about if this is the option I need, so:
Why is this not working?
How can I achieve my real purpose?
Thank you.
Edit: ctest will execute every test (add_test), but the NameTest executable (yet listed) must be built before calling it! Now would be built after the library, but before the 'POST_BUILD' custom command. It fails, of course.
I want CMake realize NameTest is necessary for running that custom command.
Edit: I find useful the Angew's answer, so I accepted his answer and refined it a little bit:
add_custom_command(
TARGET libName
POST_BUILD
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target NameTest --config $<CONFIG>
COMMAND ${CMAKE_CTEST_COMMAND} -C $<CONFIG> --output-on-failure
)
Thank you!
1. Why is this not working?
Because you're mixing options from two distinct signatures of add_custom_command. DEPENDS comes from the form which is used to generate a file. TARGET and POST_BUILD are from the form which adds pre/post build commands to existing targets.
See the documentation of add_custom_command for more details on the two uses.
2. How can I achieve my real purpose?
I believe the following should do what you want to:
add_custom_command(
TARGET libName
POST_BUILD
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target NameTest --config $<CONFIG>
COMMAND CTEST_OUTPUT_ON_FAILURE=1 ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target test --config $<CONFIG>
)

Using CMake, how do I get verbose output from CTest?

I'm using CMake to build my project. I have added a unit test binary which is using the Boost unit testing framework. This one binary contains all of the unit tests. I've added that binary to be run by CTest:
ADD_EXECUTABLE( tftest test-main.cpp )
ENABLE_TESTING()
ADD_TEST( UnitTests tftest)
But the build output in Visual Studio only shows the result of running CTest:
Start 1: UnitTests
1/1 Test #1: UnitTests ................***Failed 0.05 sec
0% tests passed, 1 tests failed out of 1
This is not very helpful, because I can't see which test failed. If I run ctest manually from the command line with --verbose I get the output from a Boost unit test which tells what actually failed:
1: Test command: tftest.exe
1: Test timeout computed to be: 9.99988e+006
1: Running 4 test cases...
1: test-main.cpp(20): error in "sanity_check3": check 1 == 2 failed
1:
1: *** 1 failure detected in test suite "Master Test Suite"
1/1 Test #1: UnitTests ................***Failed 0.00 sec
So, what do I need to change in the CMakeLists.txt to have CTest run with --verbose at all times? Is there a better way to use Boost unit tests with CMake/CTest?
You can use the ctest --output-on-failure option, or set the environment variable CTEST_OUTPUT_ON_FAILURE, which will show you any output from the test program whenever the test fails. One way to do this when using Makefiles and the command line would be as follows:
env CTEST_OUTPUT_ON_FAILURE=1 make check
This Stack Overflow question and answer shows how to set environment variables in Visual Studio.
You could call ctest directly, after cmaking and making your project.
ctest --verbose
There is a very simple solution (which for some reason is difficult to find via Google Search):
ctest --output-on-failure
If you use CMake with Visual Studio's open folder function you can add the
"ctestCommandArgs": "--output-on-failure"
setting to your build configuration.
You can check the Testing/Temporary subfolder. It is automatically created after running make test. This folder contains two files: LastTest.log and LastTestsFailed.log. LastTest.log contains desired output for run tests. LastTestFailed.log contains names of failed tests. So you can check them manually after executing make test.
The second way is to get ctest to show you the content of log files after running tests:
place in build dir (from which you run make test) file CTestCustom.ctest (you can do it with configure file command, for example) with following contents
CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log")
Instead of cat you may use whatever Windows cmd command that does similar things.
run make test again and get profit!
additional info about customizing ctest you can find here. Just step to "Customizing cmake" section.
Good luck!
I had to add "check" target by myself. "make tests" does nothing by some reason. So what I did (as was suggest somewhere on stackoverflow) - I added this target manually. To get verbose output I just wrote it like:
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
make check CTEST_OUTPUT_ON_FAILURE=TRUE
This makes test output more verbose:
make test ARGS="-V"
My approach is a combination of the answers from ony, from zbyszek, and from tarc. I use the ${CMAKE_COMMAND} variable (which is set to the absolute path to the invoked cmake executable) with the -E env CTEST_OUTPUT_ON_FAILURE=1 argument to invoke the actual ctest command using ${CMAKE_CTEST_COMMAND} -C $<CONFIG>. To help clarify what is going on, I start with three cmake -E echo commands to show the current working directory and the ctest command to be invoked. Here is how I call add_custom_target.
add_custom_target(check
${CMAKE_COMMAND} -E echo CWD=${CMAKE_BINARY_DIR}
COMMAND ${CMAKE_COMMAND} -E echo CMD=${CMAKE_CTEST_COMMAND} -C $<CONFIG>
COMMAND ${CMAKE_COMMAND} -E echo ----------------------------------
COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS ALL_BUILD
)
This plays nice with the MSVC IDE where any test errors are shown as clickable compilation errors. See cmake -E env for documentation of the cmake -E portable command line tool mode. I also add a dependency on ALL_BUILD so that all projects will be built before invoking the check target. (On Linux builds, one may need to replace ALL_BUILD with ALL; I have not tested this on Linux yet.)
For people using Visual Studio, here another variation (hack) on the theme:
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build . --target RUN_TESTS
ctest -VV or ctest --extra-verbose
From documentation:
Enable more verbose output from tests.
Test output is normally suppressed and only summary information is
displayed. This option will show even more test output.
There's now a CMake variable that allows you to modify the behaviour of make test. CMAKE_CTEST_ARGUMENTS lets you set a list of arguments to pass to ctest when run via make test.
So adding this to your CMake file:
set(CMAKE_CTEST_ARGUMENTS "--verbose")
Means CTest will always run verbose. Or for just the output of the failed tests, use:
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
Edit:
As suggested by RobLoach, since it's a list of arguments, you'll want to append to the list instead.
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
to show the result with XML file you have to execute the test with the following command
~$ ctest -T Test
and we found the result in the Testing/1234123432/test.xml
and other files are generated too in Testing Folder