I have some test added by command add_test:
find_program(PYTEST "pytest")
add_test(NAME test_something COMMAND ${PYTEST})
But before this test I need to copy some test files (including python test scripts to be run). For this purpose there is a custom target generate_init_queries. Since add_test doesn't create a target I can't use add_dependencies to link my custom target generate_init_queries and this test. I supposed that there should exist a test target in CMake and added the command:
add_dependencies(test generate_init_queries)
But it resulted in error annot add target-level dependencies to non-existent target "test". How can I copy files before running the test in make test?
This question is highly related to CMake & CTest : make test doesn't build tests which has gotten a lot more attention. The best answer nowadays (the question is from 2009) is IMO this one. I.e. using the fixtures feature in cmake available from version 3.7 onwards.
See the FIXTURES_REQUIRED docs for a good overview and usage example. I can also recommend the test-properties documentation page for a broader overview on what properties can be used to influence ctest test execution.
So modified for your use case, a possible solution would roughly look like this:
add_test(gen_init_queries
"${CMAKE_COMMAND}"
--build "${CMAKE_BINARY_DIR}"
--config "$<CONFIG>"
--target generate_init_queries
)
set_tests_properties(gen_init_queries PROPERTIES FIXTURES_SETUP f_init_queries)
set_tests_properties(test PROPERTIES FIXTURES_REQUIRED f_init_queries)
Did it through add_custom_target with ALL keyword:
find_program(PYTEST "pytest")
add_test(test_something ${PYTEST} test_something.py)
add_custom_target(test_something_py_copy ALL COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_CURRENT_SOURCE_DIR}/test_something.py ${TEST_DATA_DIR}/test_something.py DEPENDS generate_init_queries)
Related
I'm working on an embedded project. As a part of this project, I have unit-tests that use gcc and Gtest. The question what is the best to approach to incorporate these unit-tests. My current implementation is that I have a different build type called unittest. I have a clause of CMAKE_BUILD_TYPE and decide which sources to use which targets to create. I see this is not a good design and this screws up multiconfiguration gnerators. What could be the elegant solution for this?
Thanks in advance for answering.
Create separate executables for testing and use ctest:
add_test to add the combination of executable+command line parameters as a test ctest runs and enable_testing() in the toplevel CMakeLists.txt.
This allows you to simply run ctest in the build dir and you can pass a configuration to test using -C command line option.
add_executable(MyTest test.cpp test_helper.cpp test_helper.h)
target_include_directories(MyTest PRIVATE .)
target_link_libraries(MyTest PRIVATE theLibToTest)
add_test(NAME NameOfTest COMMAND MyTest --gtest_repeat=1000)
enable_testing()
Running ctest from the build directory runs a test named NameOfTest. For multi configuration generators you simply specify the configuration to test with the -C command line option
ctest -C Release
Of course you can use add_test multiple times to add different test executables or the same test executable with different command line options.
Furthermore I recommend figuring out a way of storing results in a file, since this makes via the test parameters, since ctest's output probably won't do the trick. I'm not familiar enough with gtest to give advice on this.
Btw: ctest treats exit code 0 as success and any other exit code as failure, but I guess gtest produces executables that satisfy this property.
If you don't necessarily want to build the tests at the same time as the rest, you could exclude them from all, possibly adding a custom target that depends on all of the unit tests and is also excluded from all to allow building all of them at once.
You could also use a cache variable to toggle testing on and off:
# enabled via -D TEST_MY_PROJECT:BOOL=1 parameter for cmake
set(TEST_MY_PROJECT 0 CACHE BOOL "enable tests for my project")
if (TEST_MY_PROJECT)
# testing setup goes here
endif()
I would like to define a CMake variable BUILD_TIME_VAR in CMakeLists.txt:
computed with a python script during build phase
I can then access its content with ${VAR}
In other words, the equivalent during build phase of:
execute_process(COMMAND bash -c "python $SCRIPT $FILE" OUTPUT_VARIABLE GEN_TIME_VAR)
The variable is then used to generate a file which is a dependency to make binaries.
The goal is to make the code more easy to read since otherwise, the computation occur several times.
Rather than calling n times, the python script, to compute BUILD_TIME_VAR, I would like to use the script once, to factor the code in this way:
if(expression_1)
add_custom_command(OUTPUT foo
COMMAND cmd_1(${BUILD_TIME_VAR}))
...
elseif(expression_2)
# elseif section.
add_custom_command(OUTPUT foo
COMMAND cmd_2(${BUILD_TIME_VAR}))
...
else(expression_n)
# else section.
add_custom_command(OUTPUT foo
COMMAND cmd_n(${BUILD_TIME_VAR}))
...
endif(expression)
add_custom_target(${BINARY} ALL
DEPENDS foo)
Thanks you for your help.
If I understand your question correctly, you're effectively trying to create a "build-time variable." That is something that the build tool (actually all the build tools supported by CMake) would have to support. I know of no such functionality in build tools (make, ninja, VS, ...), and hence of no support for such a thing in CMake either.
You could emulate this by writing the results to a file and reading that file in all subsequent build steps using it.
I believe you've misinterpreted the role of CMake. In short, CMake generates build files for the back-end system of your choice (Make, ninja, etc.). This is concisely reviewed here, where the configure/generate step is briefly documented along with how it precedes the actual build step.
It's not clear what you're trying to achieve exactly, but maybe defining custom commands and using their outputs to link them to custom targets for further dependency chaining (see DEPENDS here) might be helpful, e.g.
cmake_minimum_required(VERSION 3.11)
project(foobar)
set(FOO_FILES "foo.txt")
add_custom_command(OUTPUT ${FOO_FILES}
COMMAND ${CMAKE_COMMAND} -E touch ${FOO_FILES}
COMMAND echo hello >> ${FOO_FILES}
WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
COMMENT "Creating ${FOO_FILES}"
VERBATIM)
add_custom_target(foo DEPENDS ${FOO_FILES})
set(BAR_FILES "bar_dummy.txt") # this is only use to link the custom command to the corresponding custom target
add_custom_command(OUTPUT ${BAR_FILES}
COMMAND ${CMAKE_COMMAND} -E touch ${BAR_FILES}
COMMAND cat ${FOO_FILES}
WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
COMMENT "Displaying ${FOO_FILES}"
VERBATIM)
add_custom_target(bar DEPENDS ${BAR_FILES})
add_dependencies(bar foo)
However, you might need to consider the probability of going against the tool, and thus, it might be clearer to have a required step prior to the configuration of your project.
I want to build my documentation (doxygen) alongside my project with CMake by default. I let CMake generate Makefiles. I know of these approaches so far:
1)
add_custom_target(doc ALL
COMMAND ${DOXYGEN_COMMAND})
2)
add_custom_command(
TARGET my-executable
POST_BUILD
COMMAND ${DOXYGEN_COMMAND})
The issue I have with the first approach is that the target is build both during build and install (i.e. when running make and when running make install). This is a no-go.
The issue with the second approach is that it is not actually correct: the documentation doesn't have to be built after the executable (in fact, it can be done in parallel even). Moreover I cannot build only the executable now.
Is there a way to create this custom command / target such that it is run only during make (building) and not during make install (installing) but independently from other targets such as executables?
Basically I'd need a "default target". Also I really want to avoid having to execute CMake recursively or make directly.
Just create a file, which existence means that documentation file have been built:
set(documentation_file ${CMAKE__BINARY_DIR}/docs_ready)
# Documentation target
add_custom_target(docs ALL DEPENDS ${documentation_file})
# Documentation command
add_custom_command(OUTPUT ${documentation_file}
COMMAND ${DOXYGEN_COMMAND}
# And mark that documentation is created
COMMAND ${CMAKE_COMMAND} -E touch ${documentation_file}
)
The solution given by #Tsyvarev works perfectly except for one issue:
Running make docs manually doesn't cause a re-build of the documentation because the "output file" ${documentation_file} is always considered up to date (there are no dependencies specified that could be newer).
Three workarounds I found:
Run make -B docs. The -B flags causes GNU Make to consider all targets out of date, and thus causes the documentation to be build.
Add a "manual docs" target which depends on first a docs-clean target which removes ${documentation_file} and then on docs itself (to build the documentation and the file), and run make docs-manual:
add_custom_target(docs-clean
COMMAND ${CMAKE_COMMAND} -E remove ${documentation_file})
add_custom_target(docs-manual)
add_dependencies(docs-manual docs-clean docs)
Note that I fear that this is just a dirty hack: AFAIK the order in which the dependencies (docs-clean and docs) are build isn't specified, and as they don't have a dependency between them, they could even be built in parallel.
Using add_custom_command with a TARGET docs-manual and PRE_BUILD (which runs the command before building dependencies) instead of the docs-clean target would've been perfect, but this only works for Visual Studio 7 or later.
Add a custom target that depends on docs but removes the generated ${documentation_file}:
add_custom_target(docs-manual
COMMAND ${CMAKE_COMMAND} -E remove ${documentation_file})
add_dependencies(docs-manual docs)
This works except for when running it the first time after a make docs, because then the generated file will still be there when docs is built.
edit: Use add_dependencies instead of DEPENDS of add_custom_target because the later is only intended for files, not for dependencies on targets.
I'm trying CTest in CMake in order to automatically run some of my tests using make test target. The problem is CMake does not "understand" that the test I'm willing to run has to be built since it is part of the project.
So I'm looking for a way to explicitly specify this dependency.
It is arguably a bug in CMake (previously tracked here) that this doesn't work out of the box. A workaround is to do the following:
add_test(TestName ExeName)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND}
DEPENDS ExeName)
Then you can run make check and it will compile and run the test. If you have several tests, then you would have to use DEPENDS exe1 exe2 exe3 ... in the above line.
There is actually a way to use make test. You need to define the build of the test executable as one of the tests and then add dependencies between the tests. That is:
ADD_TEST(ctest_build_test_code
"${CMAKE_COMMAND}" --build ${CMAKE_BINARY_DIR} --target test_code)
ADD_TEST(ctest_run_test_code test_code)
SET_TESTS_PROPERTIES(ctest_run_test_code
PROPERTIES DEPENDS ctest_build_test_code)
I use a variant of richq's answer. In the top-level CMakeLists.txt, I add a custom target, build_and_test, for building and running all tests:
find_package(GTest)
if (GTEST_FOUND)
enable_testing()
add_custom_target(build_and_test ${CMAKE_CTEST_COMMAND} -V)
add_subdirectory(test)
endif()
In the various sub-project CMakeLists.txt files under test/, I add each test executable as a dependency of build_and_test:
include_directories(${CMAKE_SOURCE_DIR}/src/proj1)
include_directories(${GTEST_INCLUDE_DIRS})
add_executable(proj1_test proj1_test.cpp)
target_link_libraries(proj1_test ${GTEST_BOTH_LIBRARIES} pthread)
add_test(proj1_test proj1_test)
add_dependencies(build_and_test proj1_test)
With this approach, I just need to make build_and_test instead of make test (or make all test), and it has the benefit of only building test code (and its dependencies). It's a shame I can't use the target name test. In my case, it's not so bad because I have a top-level script that does out-of-tree debug and release (and cross-compiled) builds by calling cmake and then make, and it translates test into build_and_test.
Obviously, the GTest stuff isn't required. I just happen to use/like Google Test, and wanted to share a complete example of using it with CMake/CTest. IMHO, this approach also has the benefit of allowing me to use ctest -V, which shows the Google Test output while the tests run:
1: Running main() from gtest_main.cc
1: [==========] Running 1 test from 1 test case.
1: [----------] Global test environment set-up.
1: [----------] 1 test from proj1
1: [ RUN ] proj1.dummy
1: [ OK ] proj1.dummy (0 ms)
1: [----------] 1 test from proj1 (1 ms total)
1:
1: [----------] Global test environment tear-down
1: [==========] 1 test from 1 test case ran. (1 ms total)
1: [ PASSED ] 1 test.
1/2 Test #1: proj1_test ....................... Passed 0.03 sec
If you are using CMake >= 3.7, then the recommended approach is to use fixtures:
add_executable(test test.cpp)
add_test(test_build
"${CMAKE_COMMAND}"
--build "${CMAKE_BINARY_DIR}"
--config "$<CONFIG>"
--target test
)
set_tests_properties(test_build PROPERTIES FIXTURES_SETUP test_fixture)
add_test(test test)
set_tests_properties(test PROPERTIES FIXTURES_REQUIRED test_fixture)
This does the following:
Adds a test executable target built from test.cpp
Adds a test_build "test" that runs Cmake to build target test
Marks the test_build test to be a setup task of fixture test_fixture
Add a test test that just runs the test executable
Marks the test test to need fixture test_fixture.
So, every time test test is to be run, it first runs test test_build, which builds the necessary executable.
If you are trying to emulate make check, you may find this wiki entry usefull :
http://www.cmake.org/Wiki/CMakeEmulateMakeCheck
I have just checked that is does what it says with success (CMake 2.8.10).
Save yourself the headache:
make all test
Works out of the box for me and will build dependencies before running the test. Given how simple this is, it almost makes the native make test functionality convenient because it gives you the option of running the last compiling tests even if your code is broken.
For CMake 3.10 or later, another option is to use the TEST_INCLUDE_FILES directory property to set up a script that triggers a build before a test is run. In your outermost CMakeLists.txt add the following code:
set_property(DIRECTORY APPEND
PROPERTY TEST_INCLUDE_FILES "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake"
"execute_process(COMMAND \"${CMAKE_COMMAND}\""
" --build \"${CMAKE_BINARY_DIR}\""
" --config \"\$ENV{CMAKE_CONFIG_TYPE}\")")
The actual test configuration is passed through to the build via the environment variable CMAKE_CONFIG_TYPE. Optionally you can add a --target option to only build targets required by the test.
This is what I hammered out and have been using:
set(${PROJECT_NAME}_TESTS a b c)
enable_testing()
add_custom_target(all_tests)
foreach(test ${${PROJECT_NAME}_TESTS})
add_executable(${test} EXCLUDE_FROM_ALL ${test}.cc)
add_test(NAME ${test} COMMAND $<TARGET_FILE:${test}>)
add_dependencies(all_tests ${test})
endforeach(test)
build_command(CTEST_CUSTOM_PRE_TEST TARGET all_tests)
string(CONFIGURE \"#CTEST_CUSTOM_PRE_TEST#\" CTEST_CUSTOM_PRE_TEST_QUOTED ESCAPE_QUOTES)
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake" "set(CTEST_CUSTOM_PRE_TEST ${CTEST_CUSTOM_PRE_TEST_QUOTED})" "\n")
YMMV
Derrick's answer, simplified and commented:
# It is impossible to make target "test" depend on "all":
# https://gitlab.kitware.com/cmake/cmake/-/issues/8774
# Set a magic variable in a magic file that tells ctest
# to invoke the generator once before running the tests:
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake"
"set(CTEST_CUSTOM_PRE_TEST ${CMAKE_MAKE_PROGRAM})\n"
)
It is not perfectly correct, as it does not solve the concurrency problem of running ninja all test, in case anyone does that. On the contrary, because now, you have two ninja processes.
(Ftr, I also shared this solution here.)
All above answers are perfect. But actually CMake use CTest as its testing tools, so the standard method (I think it is) to do the mission is:
enable_testing ()
add_test (TestName TestCommand)
add_test (TestName2 AnotherTestCommand)
Then run cmake and make to build the targets. After that, you can either run make test, or just run
ctest
you will get the result. This is tested under CMake 2.8 .
Check details at: http://cmake.org/Wiki/CMake/Testing_With_CTest#Simple_Testing
All the answers are good, but they imply a breach of tradition to run a test by command make test. I've done this trick:
add_test(NAME <mytest>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMAND sh -c "make <mytarget>; $<TARGET_FILE:<mytarget>>")
This means that the test consists of building (optionally) and running of executable target.
To register tests under CMake, we need
enable_testing()
or
include(CTest)
and then for each single test (name fooTest, executable foo)
add_executable(foo <foo_sources>)
add_test(fooTest foo)
Tests can then be run with the command ctest.
Additionally, we can run tests with the command make check, provided we add once
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND})
and for each test we extend the above by a keyword EXCLUDE_FROM_ALL and a command add_dependencies:
add_executable(foo EXCLUDE_FROM_ALL <foo_sources>)
add_test(fooTest foo)
add_dependencies(check foo)
Ideally, this would make make check an alias of ctest. It does not so for at least two reasons:
(1) make check is flawed because it does not pass options to ctest [2]. In particular, ctest -j4 will run 4 tests in parallel, whereas make -j4 check will work in one thread on target check, and the other three threads will remain idle.
(2) ctest is flawed [3,4] because all tests are build under the all target, i.e. along with the main application. This may be desired behavior in some situations, but in other situations it ought to be possible to postpone the build until the tests are to be run.
Does this correctly summarize the current state of affairs?
Is there any way around (to eat the cake and have it)?
[1] https://cmake.org/Wiki/CMakeEmulateMakeCheck
[2] http://comments.gmane.org/gmane.comp.programming.tools.cmake.user/47300
[3] CMake & CTest : make test doesn't build tests
[4] http://public.kitware.com/Bug/view.php?id=8774
First, let me remark that ctest and make test are only simple command line tools, for simple testing tasks. If you want a tool for serious testing, use CDash, Buildbot, Jenkins or whatever.
Concerning the flaws of CTest: It is intentional, that the call for CTest does not build the tests. It is a bad idea in several scenarios:
Compiling tests can take more resources then running the tests itself. This might be true with respect to memory consumption, read/writes to the hard disk or compilation time. So compiling and linking in parallel might be bad, but executing the tests in parallel might be beneficial.
How to handle compilation or linking failure? Report it as failing? Report is as not compiling? Continuing with compiling the other tests or aborting immediately?
Autotools did it the way you want it and people got used to it. But why should it be a unit? Why not having two commands? What's the benefit of mixing two tasks and making it more difficult for project with special needs?
I came to the conclusion, to create a target build-tests or similar, and follow the decision made by the CMake developers to decouple building test and executing tests. Then I can decide whether I want parallel builds, how to treat compilation failures (e.g., passing -k to make) and so on.
The only downside is, that this target is only present in the top level directory and cannot be used in sub-directories.
To get such a target built-in by CMake would be a good feature request. Ranting on SO does no good.
CTest is not flawed at all, but the way you use CMake and CTest seems "flawed". The invocation of the command line interface (CLI) tool ctest is in general not related to the invokation of a CMake build targets (with the exception of the target test).
In my opinion the custom check target solution described in the CMake Wiki should not be used, since it changes the default behavior of CMake and is not configurable.
Instead the following approach using the built-in option BUILD_TESTING should be used:
include(CTest)
if(BUILD_TESTING)
find_package(GTest MODULE REQUIRED)
add_executable(example_test example_test.cpp)
target_link_libraries(
example_test
PRIVATE
GTest::GTest
GTest::Main
)
add_test(NAME example_test COMMAND example_test)
endif()
include(CTest) defines in the option BUILD_TESTING, which allows to control whether to build all tests of the project or not.
Quote from the official documentation:
CMake will generate tests only if the enable_testing() command has been invoked. The CTest module invokes the command automatically when the BUILD_TESTING option is ON.
The above can be used on the CLI as follows:
Create tests (default):
cmake -Hexample-testing -B_builds/example-testing/release -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release
cmake --build _builds/example-testing/release --config Release
In this case the commands cd _builds/example-testing/release and ctest / cmake --build . --target test build and run the test(s).
Do not create tests, setting -DBUILD_TESTING=OFF:
cmake -Hexample-testing -B_builds/example-testing/release-no-tests -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF
cmake --build _builds/example-testing/release-no-tests --config Release
In this case the commands cd _builds/example-testing/release-no-tests and ctest run no test(s), since no test(s) have been built.
The command cmake --build . --target test fails since it has not been created during configure phase of CMake.
We are only scratching the surface here. Refer to ctest --help, e.g. there are a lot of --build-<...> options that allow finer control regarding testing/building, though I have not any experience with that.
I highly recommend reading the following:
CTest - CMake Documentation
CTest Commands - CMake Documentation
ctest(1) - CMake Documentation
add_test - CMake Documentation
If you really want to enable building of tests, but via a separate target that is not invoked by default and run the test not via CTest but directly you can do the following:
include(CTest)
if(BUILD_TESTING)
find_package(GTest MODULE REQUIRED)
option(
BUILD_TESTING_EXCLUDE_FROM_ALL
"Do not build the testing tree together with the default build target."
OFF
)
if(BUILD_TESTING_EXCLUDE_FROM_ALL)
set(add_executable_args_for_test EXCLUDE_FROM_ALL)
endif()
# The "build_test" target is used to build all test executables.
add_custom_target(
build_test
# Workaround for printing the COMMENT, it does not work without a NOOP
# COMMAND.
COMMAND ${CMAKE_COMMAND} -E echo
COMMENT "Building tests..."
VERBATIM
)
add_executable(example_test ${add_executable_args_for_test} example_test.cpp)
target_link_libraries(
example_test
PRIVATE
GTest::GTest
GTest::Main
)
add_test(NAME example_test COMMAND example_test)
add_dependencies(build_test example_test)
# The "check" target is used to build AND run all test executables.
add_custom_target(
check
# Either invoke the test(s) indirectly via "CTest" (commented) or directly.
# COMMAND ${CMAKE_CTEST_COMMAND}
COMMAND example_test
COMMENT "Building and running test..."
VERBATIM
)
# Alternative to the COMMAND in the add_custom_target. Leads to the same
# behavior as calling "CTest" directly.
# add_custom_command(
# TARGET check
# COMMAND ${CMAKE_COMMAND} ARGS --build ${CMAKE_BINARY_DIR} --target test
# VERBATIM
# )
add_dependencies(check build_test)
endif()
Note that the above code does not invoke CTest or the target test in order to run the test, but the test directly.
Please read the comments and the commented code for alternative approaches using CTest that are similar to the approach described in the question.
It's easy to enhance the above code to support more than one test executable.
IMHO, Kitware should remove the entire CMake Wiki, since the Wiki contains almost only out-of-date information for CMake versions < 3.0. Most information in it cannot be considered as Modern CMake.