I'm using a CMakeLists.txt file to generate (among others) some tests. Therefore I have something like
ENABLE_TESTING()
ADD_TEST(NAME my_name COMMAND my_command)
I'd like to add somme commands after the test is run. I found ADD_CUSTOM_COMMAND which seems to do exactly what I need but unfortunately it doesn't work.
Here is what I have tried
ADD_CUSTOM_COMMAND(
TARGET my_name
POST_BUILD
COMMAND my_other_command
)
It seems that I'm not using the right TARGET.
Could you help me by telling me what I'm supposed to do?
Many thanks in advance,
My goal was to execute a diff command after the test execution finished. This is very similar to the problem you are facing. I used this cmake script technique. Leaving it as an answer in case it helps someone else.
In the original CMakeLists.txt file I added a test
add_test(NAME testCommand
COMMAND ${CMAKE_COMMAND}
-DCMD1=$<TARGET_FILE:target>
-DTEST_DATA_DIR=${CMAKE_SOURCE_DIR}
-P ${CMAKE_SOURCE_DIR}/runtests.cmake)
Create a file called runtests.cmake in the directory which has the CMakeLists.txt file. This new file will hold the commands to execute the test and run commands after
include(FindUnixCommands)
macro(EXEC_CHECK CMD)
execute_process(COMMAND ${CMD} ${TEST_DATA_DIR}/test_input.txt RESULT_VARIABLE CMD_RESULT)
if(CMD_RESULT)
message(FATAL_ERROR "Error running ${CMD}")
else()
if (BASH)
execute_process(COMMAND ${BASH} -c "diff -b ${TEST_DATA_DIR}/test_input.txt ${TEST_DATA_DIR}/test_output.txt" RESULT_VARIABLE RES)
if(RES)
message(FATAL_ERROR "Diff is not clean")
endif()
else(BASH)
message(FATAL_ERROR "BASH not found : no diff script run")
endif(BASH)
endif()
endmacro()
exec_check(${CMD1})
From the build directory I can now issue make test and it picks up the cmake script properly.
Related
Is there a way to compare files using cmake?
I've checked all parameters from https://cmake.org/cmake/help/latest/command/file.html
cmake executable has a tool mode, when it performs some useful actions instead of project's configuration. And compare_files is one of the commands for that mode.
For get features of the CMake command line tool mode in the CMakeLists.txt, use execute_process command:
execute_process( COMMAND ${CMAKE_COMMAND} -E compare_files <file1> <file2>
RESULT_VARIABLE compare_result
)
if( compare_result EQUAL 0)
message("The files are identical.")
elseif( compare_result EQUAL 1)
message("The files are different.")
else()
message("Error while comparing the files.")
endif()
I have a library and a test projects on CMake, and I'm using this directory structure with two (project) CMakeLists.txt:
/
|- CMakeLists.txt
|- include/libName
|- src/...
|
|- test/
|- CMakeLists.txt
|- src/...
The outer project list defines the library, like:
add_library(libName ${SRC} ${INCLUDE})
And adds 'test' as subdirectory:
add_subdirectory(test)
The test project list defines the executable and a test, like:
add_executable(NameTest ${SRC})
target_link_libraries(NameTest libName)
add_test(NAME NameTest COMMAND NameTest)
The problem
I'm trying to build and execute the test program when the library is built. If any test fails, I want the build of the library fail too.
This is what I have (inside the outer lists file):
add_custom_command(
TARGET libName
POST_BUILD
COMMAND CTEST_OUTPUT_ON_FAILURE=1 ctest
DEPENDS NameTest # <- This is driving me crazy!
)
This command ignores completely if the target 'NameTest' is built, if there is a file with that name, or if not. I can't notice any difference if the whole 'DEPENDS' option is removed.
I even modified like:
add_custom_command(
TARGET libName
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E echo "Bip! Bip! Bip!"
DEPENDS this_is_not_an_existent_file_nor_target
)
And the command is triggered anyway. I'm not very sure about if this is the option I need, so:
Why is this not working?
How can I achieve my real purpose?
Thank you.
Edit: ctest will execute every test (add_test), but the NameTest executable (yet listed) must be built before calling it! Now would be built after the library, but before the 'POST_BUILD' custom command. It fails, of course.
I want CMake realize NameTest is necessary for running that custom command.
Edit: I find useful the Angew's answer, so I accepted his answer and refined it a little bit:
add_custom_command(
TARGET libName
POST_BUILD
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target NameTest --config $<CONFIG>
COMMAND ${CMAKE_CTEST_COMMAND} -C $<CONFIG> --output-on-failure
)
Thank you!
1. Why is this not working?
Because you're mixing options from two distinct signatures of add_custom_command. DEPENDS comes from the form which is used to generate a file. TARGET and POST_BUILD are from the form which adds pre/post build commands to existing targets.
See the documentation of add_custom_command for more details on the two uses.
2. How can I achieve my real purpose?
I believe the following should do what you want to:
add_custom_command(
TARGET libName
POST_BUILD
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target NameTest --config $<CONFIG>
COMMAND CTEST_OUTPUT_ON_FAILURE=1 ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target test --config $<CONFIG>
)
I have a cmake project that runs through a number of tests, i.e.
add_test(test_title executable arg1 arg2)
On runing these tests a number of files are produced.
Once the test has run I would like to delete one of these files produced, i.e.
delete(${arg1}.txt)
or
delete(${arg2}.pdf)
If you could provide an example it would be very much appreciated.
As advised in the letter, уou can wrap actual testing command with cmake script. In your case it can be:
add_test(NAME test_title
COMMAND ${CMAKE_COMMAND}
-Darg1=${arg1}
-Darg2=${arg2}
-P ${CMAKE_CURRENT_SOURCE_DIR}/runtest.cmake
)
And the wrapper runtest.cmake can be:
execute_process(COMMAND executable ${arg1} ${arg2}
TIMEOUT 1000 # it should be less than in add_test
RESULT_VARIABLE status
)
file(REMOVE ${arg1}.txt)
file(REMOVE ${arg2}.txt)
if(status)
MESSAGE(FATAL_ERROR "Test executing status: ${status}")
endif()
I am not sure whether this option was available at the time of asking the question, but I used a hack along the lines of:
define a test: test_title
define another test: test_title_remove_file
depends on test_title (so test_title will be run first)
requires files you want to delete to exist (explicit is better than implicit)
calls cmake -E remove <file> doc
This way you'll have one fake extra test but at least it will tell you if there are any errors.
My implementation:
add_test(NAME test_title_remove_file
COMMAND ${CMAKE_COMMAND} -E remove
${arg1}
${CMAKE_CURRENT_BINARY_DIR}/${arg2}
)
set_tests_properties(test_title_remove_xmls PROPERTIES
DEPENDS test_title ## note it is your test_title
REQUIRED_FILES
${arg1};
${CMAKE_CURRENT_BINARY_DIR}/${arg2};
)
I have a console application called "foo", which takes a reference text file as input (in.txt) and generates text at standard output (I want to keep this behaviour).
In make (not cmake), I use a test target, which calls foo and redirects the output to a file (out.txt) as follows. Then, I use diff to compare the file out.txt with the expected refernece (ref.txt)
test:
./foo -a test/in.txt > test/out.txt
diff test/out.txt test/ref.txt
This works fine using make. Now my question is; how can I use cmake to create a similar Makefile?
From within a subdrectory called build, I tried
project(foo)
...
add_test(NAME test1 COMMAND ./foo ../test/in.txt > ../test/out.txt)
enable_testing()
Using cmake version 3.5, I get a Makefile without errors, but when I call make test, the test itself fails. It seems the cmake command add_test supports command line arguments, but not the redirection. I tried quotes and escaping witout success. Since I could not pass this part, I didn't try to use diff. I just imagine that I could pack foo and diff in one line using & as you can do with bash. That would be the second step.
Turning my comment into an answer
As #Tsyvarev has stated, CTest commands are not run in a shell's context. But you could just add the shell needed yourself and use e.g. sh as the command to be called with add_test().
I've run some tests with your example code and the following did work successfully:
add_test(NAME test1 COMMAND sh -c "$<TARGET_FILE:foo> ../test/in.txt > ../test/out.txt")
This solution is not platform independent (it depends on sh to be available in the search paths).
So if you want to be more flexible you could do something like:
include(FindUnixCommands)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/in.txt" _in)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/out.txt" _out)
if (BASH)
add_test(
NAME test1
COMMAND ${BASH} -c "$<TARGET_FILE:foo> ${_in} > ${_out}"
)
else()
if (WIN32)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E chdir $<TARGET_FILE_DIR:foo> $ENV{ComSpec} /c "$<TARGET_FILE_NAME:foo> ${_in} > ${_out}"
)
else()
message(FATAL_ERROR "Unknown shell command for ${CMAKE_HOST_SYSTEM_NAME}")
endif()
endif()
Additionally there is the possibility to execute a more platform independent diff with ${CMAKE_COMMAND} -E compare_files <file1> <file2>. So you could simplify your complete makefile based example in CMake with:
add_custom_command(
TARGET foo
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E echo "Running $<TARGET_FILE_NAME:foo> ..."
COMMAND foo in.txt > out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E compare_files in.txt out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
References
Integrate bash test scripts in cmake
CMake: piping commands to executable
cmake: make tests successfully passing part of the build process
They say you cannot:
There is no redirection of output using add_test arguments.
Unlike to commands in add_custom_command, which are executed as a part of makefile receipts (that is, in the context of some shell), tests are executed directly by CTest, without any shell involved. So, shell mechanisms don't work for tests.
You may create wrapper script, which calls program, given as parameter, and performs redirection, futher diff and so on. Then use this script (with appropriate arguments) as a COMMAND for add_test.
I have a console application called "foo", which takes a reference text file as input (in.txt) and generates text at standard output (I want to keep this behaviour).
In make (not cmake), I use a test target, which calls foo and redirects the output to a file (out.txt) as follows. Then, I use diff to compare the file out.txt with the expected refernece (ref.txt)
test:
./foo -a test/in.txt > test/out.txt
diff test/out.txt test/ref.txt
This works fine using make. Now my question is; how can I use cmake to create a similar Makefile?
From within a subdrectory called build, I tried
project(foo)
...
add_test(NAME test1 COMMAND ./foo ../test/in.txt > ../test/out.txt)
enable_testing()
Using cmake version 3.5, I get a Makefile without errors, but when I call make test, the test itself fails. It seems the cmake command add_test supports command line arguments, but not the redirection. I tried quotes and escaping witout success. Since I could not pass this part, I didn't try to use diff. I just imagine that I could pack foo and diff in one line using & as you can do with bash. That would be the second step.
Turning my comment into an answer
As #Tsyvarev has stated, CTest commands are not run in a shell's context. But you could just add the shell needed yourself and use e.g. sh as the command to be called with add_test().
I've run some tests with your example code and the following did work successfully:
add_test(NAME test1 COMMAND sh -c "$<TARGET_FILE:foo> ../test/in.txt > ../test/out.txt")
This solution is not platform independent (it depends on sh to be available in the search paths).
So if you want to be more flexible you could do something like:
include(FindUnixCommands)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/in.txt" _in)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/out.txt" _out)
if (BASH)
add_test(
NAME test1
COMMAND ${BASH} -c "$<TARGET_FILE:foo> ${_in} > ${_out}"
)
else()
if (WIN32)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E chdir $<TARGET_FILE_DIR:foo> $ENV{ComSpec} /c "$<TARGET_FILE_NAME:foo> ${_in} > ${_out}"
)
else()
message(FATAL_ERROR "Unknown shell command for ${CMAKE_HOST_SYSTEM_NAME}")
endif()
endif()
Additionally there is the possibility to execute a more platform independent diff with ${CMAKE_COMMAND} -E compare_files <file1> <file2>. So you could simplify your complete makefile based example in CMake with:
add_custom_command(
TARGET foo
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E echo "Running $<TARGET_FILE_NAME:foo> ..."
COMMAND foo in.txt > out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E compare_files in.txt out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
References
Integrate bash test scripts in cmake
CMake: piping commands to executable
cmake: make tests successfully passing part of the build process
They say you cannot:
There is no redirection of output using add_test arguments.
Unlike to commands in add_custom_command, which are executed as a part of makefile receipts (that is, in the context of some shell), tests are executed directly by CTest, without any shell involved. So, shell mechanisms don't work for tests.
You may create wrapper script, which calls program, given as parameter, and performs redirection, futher diff and so on. Then use this script (with appropriate arguments) as a COMMAND for add_test.