I would like PASS_REGULAR_EXPRESSION to also check stderr. Is it possible? I tried the following:
add_test(NAME test COMMAND executable)
set_tests_properties(test PROPERTIES
WILL_FAIL true
PASS_REGULAR_EXPRESSION "I was printed to stderr"
)
but it doesn't work since PASS_REGULAR_EXPRESSION checks only stdout. I tried to redirect stderr to stdout with:
add_test(NAME test COMMAND executable 2>&1)
But it doesn't seem to work either.
Related
I use CMake. A custom build step saves error output when it fails.
FIND_PROGRAM (BASH bash HINTS /bin)
SET (BASH_CMD "my-script input.file >output.file 2>error.file")
ADD_CUSTOM_COMMAND (
OUTPUT output.file
COMMAND ${CMAKE_COMMAND} -E env ${BASH} -c "${BASH_CMD}"
...)
This works. If my-script fails for the given input.file then the stderr is saved in error.file, however when I run make and the target fails to build, the normal output does not make the location of error.file obvious. (The actual path of this file is generated in a tangly way.)
I don't want the noisy stderr to show in the terminal during make. I would like to do something like
MESSAGE ("input.file failed, see error.file")
(ideally coloured red or something) to be executed when the command for output.file failed.
Can I express this behaviour in CMakeLists.txt recipes?
Not sure about the highlighting, but you could create a cmake script file executing the command via execute_process, check it's error code and print a custom message in case there's an issue. The following example runs on windows, not on linux, but this should be sufficient for demonstration.
Some command that fails: script.bat
echo "some long message" 1>&2
exit 1
CMake script: execute_script_bat.cmake
execute_process(COMMAND script.bat RESULT_VARIBALE _EXIT_CODE ERROR_FILE error.log)
if (NOT _EXIT_CODE EQUAL 0)
message(FATAL_ERROR "command failed; output see ${CMAKE_SOURCE_DIR}/error.log")
endif()
CMakeLists.txt
add_custom_command(
OUTPUT output.file
COMMAND "${CMAKE_COMMAND}" -P "${CMAKE_CURRENT_SOURCE_DIR}/execute_script_bat.cmake")
Additional info can be passed by adding -D "SOME_VARIABLE=some value" arguments after "${CMAKE_COMMAND}"
So you can use the VERBOSE option to get CMake to print all the compiler command lines to the console as it builds, but that doesn't seem to have any effect on other commands that aren't compiler commands, e.g. this execute_process command:
execute_process(
COMMAND ${_TEST_EXECUTOR} "${_TEST_EXECUTABLE}" --gtest_list_tests
WORKING_DIRECTORY "${_TEST_WORKING_DIR}"
TIMEOUT ${_TEST_DISCOVERY_TIMEOUT}
OUTPUT_VARIABLE output
RESULT_VARIABLE result
)
It's actually part of the google test module in CMake. I had an issue where I needed to track the exact command line that was being executed. I was able to but only by manually hacking the files. If I had just been able to spew all the commands CMake was executing it would have been much much quicker.
Is there some way to do that in CMake?
To get the commands in execute_process printed, you can set where you want them printed individually, in each execute_process command using COMMAND_ECHO:
execute_process(
COMMAND ${_TEST_EXECUTOR} "${_TEST_EXECUTABLE}" --gtest_list_tests
WORKING_DIRECTORY "${_TEST_WORKING_DIR}"
TIMEOUT ${_TEST_DISCOVERY_TIMEOUT}
OUTPUT_VARIABLE output
RESULT_VARIABLE result
COMMAND_ECHO STDOUT
)
or universally for all execute_process commands throughout your CMake project by setting this variable in your top-level CMake file:
set(CMAKE_EXECUTE_PROCESS_COMMAND_ECHO STDOUT)
These features are available in CMake versions 3.15 and above.
For what it's worth, you can also get the full printout of every command CMake runs (with expanded CMake variables) using the cmake command line option:
cmake --trace-expand ..
but this may be much more verbosity than you're looking for.
As part of our build process we automatically run unit tests through valgrind during the actual build (ie: it's not a separate target such as make test)
We create a sentinel file when the tests pass, so that subsequent build won't rerun the tests if not necessary.
We also save the command line and test output to a file.
Here I have built the valgrind command line:
set(VALGRIND_BIN "valgrind")
set(VALGRIND_OPTS "--leak-check=full --track-origins=yes")
set(VALGRIND_CMD "${VALGRIND_BIN} ${VALGRIND_OPTS}")
separate_arguments(VALGRIND_CMD)
These are the "passed" sentinal file, and the test output file.
set(OUTPUT_FILE ${CMAKE_CURRENT_BINARY_DIR}/${ARG_NAME}.output)
set(PASSED_FILE ${CMAKE_CURRENT_BINARY_DIR}/${ARG_NAME}.passed)
Here I add a custom_command which works in the following way:
It echos the command line and saves it to the output file
It runs the test through valgrind, saving all output to the output file
If the test doesn't pass it will cat the output file and the command fails
If the test passes it will touch the passed sentinel file.
Here is the cmake source:
add_custom_command(
OUTPUT
${PASSED_FILE}
COMMAND
echo "\"${VALGRIND_BIN} ${VALGRIND_OPTS} $<TARGET_FILE:${TEST_NAME}>\"" > ${OUTPUT_FILE}
COMMAND
${VALGRIND_CMD} $<TARGET_FILE:${TEST_NAME}> >> ${OUTPUT_FILE} 2>&1 || (cat ${OUTPUT_FILE} && false)
COMMAND
${CMAKE_COMMAND} -E touch ${PASSED_FILE}
COMMENT
"Running ${ARG_NAME} tests"
DEPENDS
${TEST_NAME}
USES_TERMINAL
)
Unfortunately cmake is escaping all the whitespace in my echo of the test command line, so that the first line in the output file looks like this:
valgrind\ --leak-check=full\ --track-origins=yes\ /home/steve/src/test\
I have proven to myself the escapes aren't in the variables, as if I output a message they aren't in there.
message(STATUS "\"${VALGRIND_BIN} ${VALGRIND_OPTS} $<TARGET_FILE:${TEST_NAME}>\"")
The resulting output:
-- "valgrind --leak-check=full --track-origins=yes $<TARGET_FILE:test>"
Question:
How can I unescape the whitespace when echoing to a file?
That is, how can I have the line not be this:
valgrind\ --leak-check=full\ --track-origins=yes\ /home/steve/src/test\
but instead be this:
valgrind --leak-check=full --track-origins=yes /home/steve/src/test
You can put everything into a list, which will be expanded and the spaces will not be escaped.
Because CMake will be escaping spaces if it believes the string to be a single argument. Giving it as a list will take every element as a separate argument:
list(APPEND VALGRIND_CMD "$<TARGET_FILE:${TEST_NAME}>")
add_custom_command(
OUTPUT
${PASSED_FILE}
COMMAND
${CMAKE_COMMAND} -E echo \"${VALGRIND_CMD}\" > ${OUTPUT_FILE}
COMMAND
${VALGRIND_CMD} >> ${OUTPUT_FILE} 2>&1 || (cat ${OUTPUT_FILE} && false)
COMMAND
${CMAKE_COMMAND} -E touch ${PASSED_FILE}
COMMENT
"Running ${ARG_NAME} tests"
USES_TERMINAL
)
References
cmake: How to include literal double-quote in custom command?
cmake: when to quote variables?
As pointed by #MuertoExcobito, option VERBATIM cares about properly escaping parameters, no needs in additional double quotes escaped manually:
COMMAND
echo "${VALGRIND_BIN} ${VALGRIND_OPTS} $<TARGET_FILE:${TEST_NAME}>" > ${OUTPUT_FILE}
VERBATIM
(Outer double quotes are needed for CMake do not separate echo parameters).
I have a console application called "foo", which takes a reference text file as input (in.txt) and generates text at standard output (I want to keep this behaviour).
In make (not cmake), I use a test target, which calls foo and redirects the output to a file (out.txt) as follows. Then, I use diff to compare the file out.txt with the expected refernece (ref.txt)
test:
./foo -a test/in.txt > test/out.txt
diff test/out.txt test/ref.txt
This works fine using make. Now my question is; how can I use cmake to create a similar Makefile?
From within a subdrectory called build, I tried
project(foo)
...
add_test(NAME test1 COMMAND ./foo ../test/in.txt > ../test/out.txt)
enable_testing()
Using cmake version 3.5, I get a Makefile without errors, but when I call make test, the test itself fails. It seems the cmake command add_test supports command line arguments, but not the redirection. I tried quotes and escaping witout success. Since I could not pass this part, I didn't try to use diff. I just imagine that I could pack foo and diff in one line using & as you can do with bash. That would be the second step.
Turning my comment into an answer
As #Tsyvarev has stated, CTest commands are not run in a shell's context. But you could just add the shell needed yourself and use e.g. sh as the command to be called with add_test().
I've run some tests with your example code and the following did work successfully:
add_test(NAME test1 COMMAND sh -c "$<TARGET_FILE:foo> ../test/in.txt > ../test/out.txt")
This solution is not platform independent (it depends on sh to be available in the search paths).
So if you want to be more flexible you could do something like:
include(FindUnixCommands)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/in.txt" _in)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/out.txt" _out)
if (BASH)
add_test(
NAME test1
COMMAND ${BASH} -c "$<TARGET_FILE:foo> ${_in} > ${_out}"
)
else()
if (WIN32)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E chdir $<TARGET_FILE_DIR:foo> $ENV{ComSpec} /c "$<TARGET_FILE_NAME:foo> ${_in} > ${_out}"
)
else()
message(FATAL_ERROR "Unknown shell command for ${CMAKE_HOST_SYSTEM_NAME}")
endif()
endif()
Additionally there is the possibility to execute a more platform independent diff with ${CMAKE_COMMAND} -E compare_files <file1> <file2>. So you could simplify your complete makefile based example in CMake with:
add_custom_command(
TARGET foo
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E echo "Running $<TARGET_FILE_NAME:foo> ..."
COMMAND foo in.txt > out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E compare_files in.txt out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
References
Integrate bash test scripts in cmake
CMake: piping commands to executable
cmake: make tests successfully passing part of the build process
They say you cannot:
There is no redirection of output using add_test arguments.
Unlike to commands in add_custom_command, which are executed as a part of makefile receipts (that is, in the context of some shell), tests are executed directly by CTest, without any shell involved. So, shell mechanisms don't work for tests.
You may create wrapper script, which calls program, given as parameter, and performs redirection, futher diff and so on. Then use this script (with appropriate arguments) as a COMMAND for add_test.
I have a console application called "foo", which takes a reference text file as input (in.txt) and generates text at standard output (I want to keep this behaviour).
In make (not cmake), I use a test target, which calls foo and redirects the output to a file (out.txt) as follows. Then, I use diff to compare the file out.txt with the expected refernece (ref.txt)
test:
./foo -a test/in.txt > test/out.txt
diff test/out.txt test/ref.txt
This works fine using make. Now my question is; how can I use cmake to create a similar Makefile?
From within a subdrectory called build, I tried
project(foo)
...
add_test(NAME test1 COMMAND ./foo ../test/in.txt > ../test/out.txt)
enable_testing()
Using cmake version 3.5, I get a Makefile without errors, but when I call make test, the test itself fails. It seems the cmake command add_test supports command line arguments, but not the redirection. I tried quotes and escaping witout success. Since I could not pass this part, I didn't try to use diff. I just imagine that I could pack foo and diff in one line using & as you can do with bash. That would be the second step.
Turning my comment into an answer
As #Tsyvarev has stated, CTest commands are not run in a shell's context. But you could just add the shell needed yourself and use e.g. sh as the command to be called with add_test().
I've run some tests with your example code and the following did work successfully:
add_test(NAME test1 COMMAND sh -c "$<TARGET_FILE:foo> ../test/in.txt > ../test/out.txt")
This solution is not platform independent (it depends on sh to be available in the search paths).
So if you want to be more flexible you could do something like:
include(FindUnixCommands)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/in.txt" _in)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/out.txt" _out)
if (BASH)
add_test(
NAME test1
COMMAND ${BASH} -c "$<TARGET_FILE:foo> ${_in} > ${_out}"
)
else()
if (WIN32)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E chdir $<TARGET_FILE_DIR:foo> $ENV{ComSpec} /c "$<TARGET_FILE_NAME:foo> ${_in} > ${_out}"
)
else()
message(FATAL_ERROR "Unknown shell command for ${CMAKE_HOST_SYSTEM_NAME}")
endif()
endif()
Additionally there is the possibility to execute a more platform independent diff with ${CMAKE_COMMAND} -E compare_files <file1> <file2>. So you could simplify your complete makefile based example in CMake with:
add_custom_command(
TARGET foo
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E echo "Running $<TARGET_FILE_NAME:foo> ..."
COMMAND foo in.txt > out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
add_test(
NAME test1
COMMAND ${CMAKE_COMMAND} -E compare_files in.txt out.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)
References
Integrate bash test scripts in cmake
CMake: piping commands to executable
cmake: make tests successfully passing part of the build process
They say you cannot:
There is no redirection of output using add_test arguments.
Unlike to commands in add_custom_command, which are executed as a part of makefile receipts (that is, in the context of some shell), tests are executed directly by CTest, without any shell involved. So, shell mechanisms don't work for tests.
You may create wrapper script, which calls program, given as parameter, and performs redirection, futher diff and so on. Then use this script (with appropriate arguments) as a COMMAND for add_test.