Is there a way to color the ctest output.
For example is it possible to have something similar to the last line being green in case of 100% success
100% tests passed, 0 test failed out of x
or red otherwise?
Related
I added a simple test to ctest with the following lines in a .cmake file:
add_test( NAME ktxsc-test-many-in-one-out
COMMAND ktxsc -o foo a.ktx2 b.ktx2 c.ktx2
)
set_tests_properties(
ktxsc-test-many-in-one-out
PROPERTIES
WILL_FAIL TRUE
FAIL_REGULAR_EXPRESSION "^Can't use -o when there are multiple infiles."
)
The test passes and the TestLog shows
----------------------------------------------------------
Test Pass Reason:
Error regular expression found in output. Regex=[^Can't use -o when there are multiple infiles.]
"ktxsc-test-many-in-one-out" end time: Jun 30 16:34 JST
"ktxsc-test-many-in-one-out" time elapsed: 00:00:00
----------------------------------------------------------
If I change FAIL_REGULAR_EXPRESSION to
FAIL_REGULAR_EXPRESSION "some rubbish"
the test still passes even though the app is printing the same message as before. This time the test log shows
----------------------------------------------------------
Test Passed.
"ktxsc-test-many-in-one-out" end time: Jun 30 16:53 JST
"ktxsc-test-many-in-one-out" time elapsed: 00:00:00
----------------------------------------------------------
which is what I normally see when no *_REGULAR_EXPRESSION is set.
Why is this happening? How can I get ctest to fail the test when the FAIL_REGULAR_EXPRESSION doesn't match?
Thanks to #Tsyvarev for this answer.
ctest is doing
match FAIL_REGULAR_EXPRESSION || exit code != 0
to determine if a test has failed. If the match succeeds the exit code is not checked. If the match fails the exit code is checked and the failed match is ignored.
The documentation is unclear. I've read it many times and still didn't figure out this behavior until steered in the right direction by #Tsyvarev. I also find this a poor behavioral choice. All my tools output both a non-zero error code and a message when there is an error condition. I need to test both. I previously asked this question about that. It requires duplicating tests.
If I compile and run
! main.f90
print*, 1/0
end program
with ifort then I get a division by zero error with an exit status of 2 (echo $?), as expected. However, If I compile using ifort -coarray=shared then I still get the error but now the exit status is 0. The problem is that CTest is unable to catch the error. This is my CMakeLists.txt:
cmake_minimum_required(VERSION 2.8)
project(testing Fortran)
enable_testing()
add_executable(main EXCLUDE_FROM_ALL main.f90)
add_test(main main)
add_custom_target(check COMMAND ctest DEPENDS main)
If I run make check then the output is
100% tests passed, 0 tests failed out of 1
even though the test actually failed. If I remove -coarray=shared or use gfortran then I get the correct output
0% tests passed, 1 tests failed out of 1
How can I make CTest report that the test failed? Am I doing something wrong or is this a compiler bug?
I'm testing with ifort 14.0.1 with the example code
program test
implicit none
print *, 1/0
end program
I can't quite replicate your initial return value of 3. In my testing the result of echo $? is 71, which correlates to the message thrown from the runtime error (note: gfortran won't even compile the example):
forrtl: severe (71): integer divide by zero
I am however replicating your return code of 0 from the coarray version. What is happening here is a bit more complicated, as ifort implements coarrays through MPI calls and for -coarray=shared basically wraps your program in a wrapper so you do not have to call mpirun, mpiexec or have to run intel's mpd to handle MPI communications. It is clear that the coarray images (MPI ranks) are all returning with error code 3:
application called MPI_Abort(comm=0x84000000, 3) - process 4
and
exit status of rank 1: return code 3
is emitted by each MPI rank, but the executable itself always returns with exit code 0. Whether this is the intended behavior or a bug is not clear to me, but it seems likely that the wrapper code to launch the MPI processes doesn't look at the MPI return codes from each rank. As pointed out in the comments, how would we expect different return values from different mpi ranks to be handled? It doesn't seem you are doing anything wrong.
Contrast this with a normal MPI example:
program mpitest
use mpi
implicit none
integer :: rank, msize, merror, mstatus(MPI_STATUS_SIZE)
call MPI_INIT(merror)
call MPI_COMM_SIZE(MPI_COMM_WORLD, msize, merror)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, merror)
print *, 1/0
call MPI_FINALIZE(merror)
end program
compiled with
mpiifort -o testmpi testmpi.f90
and run as (with mpd running):
mpirun -np 4 ./testmpi
This produced a severe runtime error 71 for integer divide by zero for each rank as before, but the error code is propagated back to the shell:
$ echo $?
71
I am using Gitlab-CI for my build tests. I have a very simple test which compares the output of the test install/build with the known output. I put the test in a makefile.
The Makefile entry looks like this:
test:clean
make install DESTDIR=$(TEST_DIR)
$(TEST_DIR)/path/to/executable > $(TEST_DIR)/tmp.out
diff test/test.result $(TEST_DIR)/tmp.out
When the diff passes, an exit code of 0 is returned, a exit code of 1 is returned if the diff shows a difference in the files.
What I've tried:
Running make test from any shell runs the tests and exits, regardless of diff result
Running make test from the shell as gitlab_ci_runner runs the tests and exists regardless of diff result
When ran from Gitlab-CI, and the diff exit status is 0, the build returns success
The problem:
When ran in the Gitlab-CI and the diff exit status is non-0, the build hangs.
The output on the build screen is the output of the diff, and the last line is the expected error: make: *** [test] Error 1
After that, the cycle symbol keeps on, the runner does not exit with a build fail.
Any ideas? I thought that it might be something with Makefiles, but the Gitlab-CI will exit with a fail status if the Make exits with Error 1 for any other test. I can only see it happening on the output of the diff.
Thanks!
Also posted this to the GitLab mailinglist https://groups.google.com/d/msgid/gitlabhq/77e82813-b98e-4abe-9755-f39e07043384%40googlegroups.com?utm_medium=email&utm_source=footer
When I create a plan in Bamboo and add a task for running CxxTest test code(running function TS_ASSERT(1==1) or st). When I try to run for checking failure (TS_ASSERT(1==2)), this test case is fail and Bamboo output a log as:
12-Mar-2014 15:12:07 Failed 1 and Skipped 0 of 2 tests
12-Mar-2014 15:12:07 Success rate: 50%
12-Mar-2014 15:12:07 Failing task since return code was 1 while expected 0
So, does anyone here know why bamboo can understand the test result, and what is the return code here(return code was 1 while expected 0)?
From my observation, one windows, Bamboo consider value of the %ERRORLEVEL% as the result of the task. so return code was 1 while expected 0 means your task is returning 1 and Bamboo is expecting 0. Yes, it's expecting 0 since it considers any value other than 0 as failure.
I have a task of breaking down webkit layout test results by passed/failed/skiped/crashed tests
I'm trying to comprehend what kind of "categories" to expect on the output.
I understood the NEW tests and the FAILED tests are shown in the results.txt but what about the rest?
These are the current categories I discovered so far
stderr
incorrect layout
succeeded
timed out
what else should I consider?
You can see the failure types in test_failures.py. They are
Test crashed
Test timed out
Expected output (whether image, audio, or text) is missing
Expected text output did not match actual output
Expected image hash (checksum) output did not match actual output
Expected image data (actual pixels) output did not match actual output
Expected image and text output did not match actual output
Expected audio output did not match actual output