I have a custom target that runs a test when "built", which is provided to me by a custom function belonging to an external dependency (i.e., it is opaque to me and I have no control over its definition).
How do I invoke building this target as a CTest test case? Is this considered an anti-pattern?
If you're able to configure a cmake project that adds a custom target, you're able to see the cmake sources where this is added. Otherwise those sources wouldn't be available. This would allow you to turn add_custom_target commands equivalent add_test commands.
If you're not supposed to look into the cmake logic providing the test cases there may be the option of defining a custom cmake function that can be called to extract the COMMAND arguments ect. for use with either add_custom_target or add_test, so if you're collaborating with the people creating the "test" logic, you may be able to rewrite their cmake logic.
If you have no other option though, you could still create a test that builds a specific target.
function(add_target_build_test TEST_NAME TARGET_NAME)
get_property(IS_MULTI_CONFIG_GENERATOR GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG)
set(CONFIG_PARAM)
if(IS_MULTI_CONFIG_GENERATOR)
set(CONFIG_PARAM --config $<CONFIG>)
endif()
add_test(NAME ${TEST_NAME} COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} ${CONFIG_PARAM} --target ${TARGET_NAME})
endfunction()
add_target_build_test(my_test extern_target)
I'd avoid doing this though, if possible.
Related
I'm working on an embedded project. As a part of this project, I have unit-tests that use gcc and Gtest. The question what is the best to approach to incorporate these unit-tests. My current implementation is that I have a different build type called unittest. I have a clause of CMAKE_BUILD_TYPE and decide which sources to use which targets to create. I see this is not a good design and this screws up multiconfiguration gnerators. What could be the elegant solution for this?
Thanks in advance for answering.
Create separate executables for testing and use ctest:
add_test to add the combination of executable+command line parameters as a test ctest runs and enable_testing() in the toplevel CMakeLists.txt.
This allows you to simply run ctest in the build dir and you can pass a configuration to test using -C command line option.
add_executable(MyTest test.cpp test_helper.cpp test_helper.h)
target_include_directories(MyTest PRIVATE .)
target_link_libraries(MyTest PRIVATE theLibToTest)
add_test(NAME NameOfTest COMMAND MyTest --gtest_repeat=1000)
enable_testing()
Running ctest from the build directory runs a test named NameOfTest. For multi configuration generators you simply specify the configuration to test with the -C command line option
ctest -C Release
Of course you can use add_test multiple times to add different test executables or the same test executable with different command line options.
Furthermore I recommend figuring out a way of storing results in a file, since this makes via the test parameters, since ctest's output probably won't do the trick. I'm not familiar enough with gtest to give advice on this.
Btw: ctest treats exit code 0 as success and any other exit code as failure, but I guess gtest produces executables that satisfy this property.
If you don't necessarily want to build the tests at the same time as the rest, you could exclude them from all, possibly adding a custom target that depends on all of the unit tests and is also excluded from all to allow building all of them at once.
You could also use a cache variable to toggle testing on and off:
# enabled via -D TEST_MY_PROJECT:BOOL=1 parameter for cmake
set(TEST_MY_PROJECT 0 CACHE BOOL "enable tests for my project")
if (TEST_MY_PROJECT)
# testing setup goes here
endif()
Part of my source code is generated by a tool which is also built under our main project with a add_subdirectory. We execute this tool with a execute_process command. Clearly, if the tool is not built before we reach the execute_process statement it will fail.
I use a GLOB (file(GLOB...)) to find the source files generated. I do this because it is not possible to know beforehand how many files are generated, neither their names.
How do I force cmake to wait for the subproject to be compiled before the execute process? I would need something like a DEPENDS property for the execute_process but this option is not available.
# This subproject will source generator the tool
add_subdirectory(generator)
# I need something like: wait_for(generator)
execute_process(COMMAND generator ${CMAKE_SOURCE_DIR}/src)
file(GLOB GeneratedSources ${CMAKE_SOURCE_DIR}/src/*.cpp)
add_executable(mainprject.exe ${ProcessorSourceFiles}
Command execute_process executes its COMMAND immediately, at configuration stage. So it cannot be arranged after the executable is created with add_executable command: that executable will be built only at build stage.
You need to build subproject at configuration stage too. E.g. with
execute_process(COMMAND ${CMAKE_COMMAND}
-S ${CMAKE_SOURCE_DIR}/generator
-B ${CMAKE_BINARY_DIR}/generator
-G ${CMAKE_GENERATOR}
)
execute_process(COMMAND ${CMAKE_COMMAND}
--build ${CMAKE_BINARY_DIR}/generator
)
The first command invokes cmake for configure the 'generator' project, located under ${CMAKE_SOURCE_DIR}/generator directory. With -G option we use for subproject the same CMake generator, as one used for the main project.
The second command builds that project, so it produces generator executable.
After generator executable is created, you may use it for your project:
execute_process(COMMAND ${CMAKE_BINARY_DIR}/generator/<...>/generator ${CMAKE_SOURCE_DIR}/src)
Here you need to pass absolute path to the generator executable as the first parameter to COMMAND: CMake no longer have generator executable target, so it won't substitute its path automatically.
You will need to model this with target dependencies. The tool "generator" should be a cmake target. In that case use add_custom_target instead of execute_process somthing like this:
add_custom_target(generate_sources ALL COMMAND generator ${CMAKE_SOURCE_DIR}/src))
Then add a target dependency to "generator" using add_dependencies:
add_dependencies(generate_sources generator)
This will make sure your target "generate_sources", which runs the tool will only run during build after the target "generator" has been compiled.
The following is false, see the comments for more info:
Use add_dependencies to add a dependency from "mainproject.exe" to "generate_sources". Now this I have never tested, so take with a grain of salt: With CMake more recent than version 3.12, according to the entry on file, you should then be able to change your file command to:
file(GLOB GeneratedSources CONFIGURE_DEPENDS ${CMAKE_SOURCE_DIR}/src/*.cpp)
Which I interpret as this will re-glob the files during build if the directory changes.
I have written a Clang-based tool and I want to run it on existing CMake executable target. I want this to be a separate Makefile target, so I can run it without builiding exe target.
There is a solution to run it during exe target build (described in cmake clang-tidy (or other script) as custom target)
set(CLANG_TIDY_EXE ${MY_CLANG_BASED_TOOL} )
set(DO_CLANG_TIDY "${CLANG_TIDY_EXE}" " --my-additional-options")
set_target_properties(
my_exe_target PROPERTIES
CXX_CLANG_TIDY "${DO_CLANG_TIDY}"
)
CMake runs my tool during my_exe_target build. In build log I see:
...
cmake -E __run_co_compile --tidy=my_tool --source=main.cpp -- ..
But is it possible to create a separate target?
Maybe you could use add_custom_command, e.g. (adjust according to your vars and other needs):
add_custom_target(tidyup
COMMAND ${DO_CLANG_TIDY} [...] ${SOURCES}
DEPENDS [...]
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR})
edit (to address OP question):
A good starting point is to search for __run_co_compile and try to recreate the command from the Makefile rule (if your generator is make). There's no "automatic" propagation of the attributes, because a custom target or command can be anything. You could use the corresponding cmake variables (e.g. CMAKE_CXX_FLAGS, etc) or target properties (e.g. COMPILE_DEFINITIONS) to emulate that.
I have some test added by command add_test:
find_program(PYTEST "pytest")
add_test(NAME test_something COMMAND ${PYTEST})
But before this test I need to copy some test files (including python test scripts to be run). For this purpose there is a custom target generate_init_queries. Since add_test doesn't create a target I can't use add_dependencies to link my custom target generate_init_queries and this test. I supposed that there should exist a test target in CMake and added the command:
add_dependencies(test generate_init_queries)
But it resulted in error annot add target-level dependencies to non-existent target "test". How can I copy files before running the test in make test?
This question is highly related to CMake & CTest : make test doesn't build tests which has gotten a lot more attention. The best answer nowadays (the question is from 2009) is IMO this one. I.e. using the fixtures feature in cmake available from version 3.7 onwards.
See the FIXTURES_REQUIRED docs for a good overview and usage example. I can also recommend the test-properties documentation page for a broader overview on what properties can be used to influence ctest test execution.
So modified for your use case, a possible solution would roughly look like this:
add_test(gen_init_queries
"${CMAKE_COMMAND}"
--build "${CMAKE_BINARY_DIR}"
--config "$<CONFIG>"
--target generate_init_queries
)
set_tests_properties(gen_init_queries PROPERTIES FIXTURES_SETUP f_init_queries)
set_tests_properties(test PROPERTIES FIXTURES_REQUIRED f_init_queries)
Did it through add_custom_target with ALL keyword:
find_program(PYTEST "pytest")
add_test(test_something ${PYTEST} test_something.py)
add_custom_target(test_something_py_copy ALL COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_CURRENT_SOURCE_DIR}/test_something.py ${TEST_DATA_DIR}/test_something.py DEPENDS generate_init_queries)
To register tests under CMake, we need
enable_testing()
or
include(CTest)
and then for each single test (name fooTest, executable foo)
add_executable(foo <foo_sources>)
add_test(fooTest foo)
Tests can then be run with the command ctest.
Additionally, we can run tests with the command make check, provided we add once
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND})
and for each test we extend the above by a keyword EXCLUDE_FROM_ALL and a command add_dependencies:
add_executable(foo EXCLUDE_FROM_ALL <foo_sources>)
add_test(fooTest foo)
add_dependencies(check foo)
Ideally, this would make make check an alias of ctest. It does not so for at least two reasons:
(1) make check is flawed because it does not pass options to ctest [2]. In particular, ctest -j4 will run 4 tests in parallel, whereas make -j4 check will work in one thread on target check, and the other three threads will remain idle.
(2) ctest is flawed [3,4] because all tests are build under the all target, i.e. along with the main application. This may be desired behavior in some situations, but in other situations it ought to be possible to postpone the build until the tests are to be run.
Does this correctly summarize the current state of affairs?
Is there any way around (to eat the cake and have it)?
[1] https://cmake.org/Wiki/CMakeEmulateMakeCheck
[2] http://comments.gmane.org/gmane.comp.programming.tools.cmake.user/47300
[3] CMake & CTest : make test doesn't build tests
[4] http://public.kitware.com/Bug/view.php?id=8774
First, let me remark that ctest and make test are only simple command line tools, for simple testing tasks. If you want a tool for serious testing, use CDash, Buildbot, Jenkins or whatever.
Concerning the flaws of CTest: It is intentional, that the call for CTest does not build the tests. It is a bad idea in several scenarios:
Compiling tests can take more resources then running the tests itself. This might be true with respect to memory consumption, read/writes to the hard disk or compilation time. So compiling and linking in parallel might be bad, but executing the tests in parallel might be beneficial.
How to handle compilation or linking failure? Report it as failing? Report is as not compiling? Continuing with compiling the other tests or aborting immediately?
Autotools did it the way you want it and people got used to it. But why should it be a unit? Why not having two commands? What's the benefit of mixing two tasks and making it more difficult for project with special needs?
I came to the conclusion, to create a target build-tests or similar, and follow the decision made by the CMake developers to decouple building test and executing tests. Then I can decide whether I want parallel builds, how to treat compilation failures (e.g., passing -k to make) and so on.
The only downside is, that this target is only present in the top level directory and cannot be used in sub-directories.
To get such a target built-in by CMake would be a good feature request. Ranting on SO does no good.
CTest is not flawed at all, but the way you use CMake and CTest seems "flawed". The invocation of the command line interface (CLI) tool ctest is in general not related to the invokation of a CMake build targets (with the exception of the target test).
In my opinion the custom check target solution described in the CMake Wiki should not be used, since it changes the default behavior of CMake and is not configurable.
Instead the following approach using the built-in option BUILD_TESTING should be used:
include(CTest)
if(BUILD_TESTING)
find_package(GTest MODULE REQUIRED)
add_executable(example_test example_test.cpp)
target_link_libraries(
example_test
PRIVATE
GTest::GTest
GTest::Main
)
add_test(NAME example_test COMMAND example_test)
endif()
include(CTest) defines in the option BUILD_TESTING, which allows to control whether to build all tests of the project or not.
Quote from the official documentation:
CMake will generate tests only if the enable_testing() command has been invoked. The CTest module invokes the command automatically when the BUILD_TESTING option is ON.
The above can be used on the CLI as follows:
Create tests (default):
cmake -Hexample-testing -B_builds/example-testing/release -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release
cmake --build _builds/example-testing/release --config Release
In this case the commands cd _builds/example-testing/release and ctest / cmake --build . --target test build and run the test(s).
Do not create tests, setting -DBUILD_TESTING=OFF:
cmake -Hexample-testing -B_builds/example-testing/release-no-tests -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF
cmake --build _builds/example-testing/release-no-tests --config Release
In this case the commands cd _builds/example-testing/release-no-tests and ctest run no test(s), since no test(s) have been built.
The command cmake --build . --target test fails since it has not been created during configure phase of CMake.
We are only scratching the surface here. Refer to ctest --help, e.g. there are a lot of --build-<...> options that allow finer control regarding testing/building, though I have not any experience with that.
I highly recommend reading the following:
CTest - CMake Documentation
CTest Commands - CMake Documentation
ctest(1) - CMake Documentation
add_test - CMake Documentation
If you really want to enable building of tests, but via a separate target that is not invoked by default and run the test not via CTest but directly you can do the following:
include(CTest)
if(BUILD_TESTING)
find_package(GTest MODULE REQUIRED)
option(
BUILD_TESTING_EXCLUDE_FROM_ALL
"Do not build the testing tree together with the default build target."
OFF
)
if(BUILD_TESTING_EXCLUDE_FROM_ALL)
set(add_executable_args_for_test EXCLUDE_FROM_ALL)
endif()
# The "build_test" target is used to build all test executables.
add_custom_target(
build_test
# Workaround for printing the COMMENT, it does not work without a NOOP
# COMMAND.
COMMAND ${CMAKE_COMMAND} -E echo
COMMENT "Building tests..."
VERBATIM
)
add_executable(example_test ${add_executable_args_for_test} example_test.cpp)
target_link_libraries(
example_test
PRIVATE
GTest::GTest
GTest::Main
)
add_test(NAME example_test COMMAND example_test)
add_dependencies(build_test example_test)
# The "check" target is used to build AND run all test executables.
add_custom_target(
check
# Either invoke the test(s) indirectly via "CTest" (commented) or directly.
# COMMAND ${CMAKE_CTEST_COMMAND}
COMMAND example_test
COMMENT "Building and running test..."
VERBATIM
)
# Alternative to the COMMAND in the add_custom_target. Leads to the same
# behavior as calling "CTest" directly.
# add_custom_command(
# TARGET check
# COMMAND ${CMAKE_COMMAND} ARGS --build ${CMAKE_BINARY_DIR} --target test
# VERBATIM
# )
add_dependencies(check build_test)
endif()
Note that the above code does not invoke CTest or the target test in order to run the test, but the test directly.
Please read the comments and the commented code for alternative approaches using CTest that are similar to the approach described in the question.
It's easy to enhance the above code to support more than one test executable.
IMHO, Kitware should remove the entire CMake Wiki, since the Wiki contains almost only out-of-date information for CMake versions < 3.0. Most information in it cannot be considered as Modern CMake.