I added a simple test to ctest with the following lines in a .cmake file:
add_test( NAME ktxsc-test-many-in-one-out
COMMAND ktxsc -o foo a.ktx2 b.ktx2 c.ktx2
)
set_tests_properties(
ktxsc-test-many-in-one-out
PROPERTIES
WILL_FAIL TRUE
FAIL_REGULAR_EXPRESSION "^Can't use -o when there are multiple infiles."
)
The test passes and the TestLog shows
----------------------------------------------------------
Test Pass Reason:
Error regular expression found in output. Regex=[^Can't use -o when there are multiple infiles.]
"ktxsc-test-many-in-one-out" end time: Jun 30 16:34 JST
"ktxsc-test-many-in-one-out" time elapsed: 00:00:00
----------------------------------------------------------
If I change FAIL_REGULAR_EXPRESSION to
FAIL_REGULAR_EXPRESSION "some rubbish"
the test still passes even though the app is printing the same message as before. This time the test log shows
----------------------------------------------------------
Test Passed.
"ktxsc-test-many-in-one-out" end time: Jun 30 16:53 JST
"ktxsc-test-many-in-one-out" time elapsed: 00:00:00
----------------------------------------------------------
which is what I normally see when no *_REGULAR_EXPRESSION is set.
Why is this happening? How can I get ctest to fail the test when the FAIL_REGULAR_EXPRESSION doesn't match?
Thanks to #Tsyvarev for this answer.
ctest is doing
match FAIL_REGULAR_EXPRESSION || exit code != 0
to determine if a test has failed. If the match succeeds the exit code is not checked. If the match fails the exit code is checked and the failed match is ignored.
The documentation is unclear. I've read it many times and still didn't figure out this behavior until steered in the right direction by #Tsyvarev. I also find this a poor behavioral choice. All my tools output both a non-zero error code and a message when there is an error condition. I need to test both. I previously asked this question about that. It requires duplicating tests.
Related
Hello I'm trying to do a simple groovy script in Soapui
I try to get a testcase property and increment it then save it.
when I run then script it increments it two times I don't know why. I tried different syntaxes but nothing seems to work up to now.
Here is a screenshot that shows my problem
here I run the test 2 times, first the variable was 3, normally when I run the test the second time the before value should be at 4 and the after at 5, and not 5 and 6.
I believe that you do not want to have the increment logic in Script Assertion.
Instead increment counter in the Setup Script of test case.
If you need the counter value in the script assertion, just read it alone.
Hope this helps.
By the way, I do not see any issue with script you have shown.
Check if there any where else if this variable is being manipulated.
def cnt = context.testCase.getPropertyValue('COUNT') as Integer
if (cnt< 10){
log.info "before : $cnt"
cnt += 1
log.info "after : $cnt"
context.testCase.setPropertyValue('COUNT', cnt.toString())
}
Can't comment yet. I'm seeing the same issue in 5.3.0 - here's my script that grabs a string value from properties, converts it to an Integer, increments it and sends back as a string.
loopsInt = messageExchange.modelItem.testStep.testCase.getPropertyValue("loops").toInteger();
log.info loopsInt;
loopsInt++;
log.info loopsInt;
messageExchange.modelItem.testStep.testCase.setPropertyValue("loops", String.valueOf(loopsInt));
I log the value before I increment, and immediately after, and as you can see, the value is being incremented twice. Here I run the script 3 times:
Thu Mar 16 12:04:54 NZDT 2017:INFO:52
Thu Mar 16 12:04:54 NZDT 2017:INFO:53
Thu Mar 16 12:04:56 NZDT 2017:INFO:54
Thu Mar 16 12:04:56 NZDT 2017:INFO:55
Thu Mar 16 12:04:59 NZDT 2017:INFO:56
Thu Mar 16 12:04:59 NZDT 2017:INFO:57
I get the same result whether I use loopsInt++ or loopsInt = loopsInt + 1. The "loops" property is not being used anywhere else. Weird.
When you execute the assertion script with the green arrow in Script Assertion Window, it is executed twice.
I have used the following script:
def loopsInt = messageExchange.modelItem.testStep.testCase.getPropertyValue("myNum").toInteger();
log.info loopsInt
loopsInt++
messageExchange.modelItem.testStep.testCase.setPropertyValue("myNum", String.valueOf(loopsInt))
See the following picture. One window logs even numbers and second the odd numbers.
Please note that execution in Script Assertion Window shall be utilized just for debugging of the script. When you execute the test case (test step), the script is executed only once, as expected.
Anyway, I think there are better places to set test case properties (setUp script, Groovy Script test step and others). I recommend to use assertion scripts for checking the message exchange.
Karel
Found a very strange reason for this.
If the testStep having this assertion is been run successfully [i.e., if SOAP test step, then it becomes green], and after this if you open the assertion and run it separately, then it's incremented twice. Once in your editor, once within the test step itself.
Say the testSetp had failed [is red in color], then you try running the assertion separately, it works absolutely fine.
When I run ANY test I get the same message. Here is an example test:
package require tcltest
namespace import -force ::tcltest::*
test foo-1.1 {save 1 in variable name foo} {} {
set foo 1
} {1}
I get the following output:
WARNING: unknown option -run: should be one of -asidefromdir, -constraints, -debug, -errfile, -file, -limitconstraints, -load, -loadfile, -match, -notfile, -outfile, -preservecore, -relateddir, -singleproc, -skip, -testdir, -tmpdir, or -verbose
I've tried multiple tests and nothing seems to work. Does anyone know how to get this working?
Update #1:
The above error was my fault, it was due to it being run in my script. However if I run the following at a command line I got no output:
[root#server1 ~]$ tcl
tcl>package require tcltest
2.3.3
tcl>namespace import -force ::tcltest::*
tcl>test foo-1.1 {save 1 in variable name foo} {expr 1+1} {2}
tcl>echo [test foo-1.1 {save 1 in variable name foo} {expr 1+1} {2}]
tcl>
How do I get it to output pass or fail?
You don't get any output from the test command itself (as long as the test passes, as in the example: if it fails, the command prints a "contents of test case" / "actual result" / "expected result" summary; see also the remark on configuration below). The test statistics are saved internally: you can use the cleanupTests command to print the Total/Passed/Skipped/Failed numbers (that command also resets the counters and does some cleanup).
(When you run runAllTests, it runs test files in child processes, intercepting the output from each file's cleanupTests and adding them up to a grand total.)
The internal statistics collected during testing is available in AFACT undocumented namespace variables like ::tcltest::numTests. If you want to work with the statistics yourself, you can access them before calling cleanupTests, e.g.
parray ::tcltest::numTests
array set myTestData [array get ::tcltest::numTests]
set passed $::tcltest::numTests(Passed)
Look at the source for tcltest in your library to see what variables are available.
The amount of output from the test command is configurable, and you can get output even when the test passes if you add p / pass to the -verbose option. This option can also let you have less output on failure, etc.
You can also create a command called ::tcltest::ReportToMaster which, if it exists, will be called by cleanupTests with the pertinent data as arguments. Doing so seems to suppress both output of statistics and at least most resetting and cleanup. (I didn't go very far in investigating that method.) Be aware that messing about with this is more likely to create trouble than solve problems, but if you are writing your own testing software based on tcltest you might still want to look at it.
Oh, and please use the newer syntax for the test command. It's more verbose, but you'll thank yourself later on if you get started with it.
Obligatory-but-fairly-useless (in this case) documentation link: tcltest
If I compile and run
! main.f90
print*, 1/0
end program
with ifort then I get a division by zero error with an exit status of 2 (echo $?), as expected. However, If I compile using ifort -coarray=shared then I still get the error but now the exit status is 0. The problem is that CTest is unable to catch the error. This is my CMakeLists.txt:
cmake_minimum_required(VERSION 2.8)
project(testing Fortran)
enable_testing()
add_executable(main EXCLUDE_FROM_ALL main.f90)
add_test(main main)
add_custom_target(check COMMAND ctest DEPENDS main)
If I run make check then the output is
100% tests passed, 0 tests failed out of 1
even though the test actually failed. If I remove -coarray=shared or use gfortran then I get the correct output
0% tests passed, 1 tests failed out of 1
How can I make CTest report that the test failed? Am I doing something wrong or is this a compiler bug?
I'm testing with ifort 14.0.1 with the example code
program test
implicit none
print *, 1/0
end program
I can't quite replicate your initial return value of 3. In my testing the result of echo $? is 71, which correlates to the message thrown from the runtime error (note: gfortran won't even compile the example):
forrtl: severe (71): integer divide by zero
I am however replicating your return code of 0 from the coarray version. What is happening here is a bit more complicated, as ifort implements coarrays through MPI calls and for -coarray=shared basically wraps your program in a wrapper so you do not have to call mpirun, mpiexec or have to run intel's mpd to handle MPI communications. It is clear that the coarray images (MPI ranks) are all returning with error code 3:
application called MPI_Abort(comm=0x84000000, 3) - process 4
and
exit status of rank 1: return code 3
is emitted by each MPI rank, but the executable itself always returns with exit code 0. Whether this is the intended behavior or a bug is not clear to me, but it seems likely that the wrapper code to launch the MPI processes doesn't look at the MPI return codes from each rank. As pointed out in the comments, how would we expect different return values from different mpi ranks to be handled? It doesn't seem you are doing anything wrong.
Contrast this with a normal MPI example:
program mpitest
use mpi
implicit none
integer :: rank, msize, merror, mstatus(MPI_STATUS_SIZE)
call MPI_INIT(merror)
call MPI_COMM_SIZE(MPI_COMM_WORLD, msize, merror)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, merror)
print *, 1/0
call MPI_FINALIZE(merror)
end program
compiled with
mpiifort -o testmpi testmpi.f90
and run as (with mpd running):
mpirun -np 4 ./testmpi
This produced a severe runtime error 71 for integer divide by zero for each rank as before, but the error code is propagated back to the shell:
$ echo $?
71
I am using Gitlab-CI for my build tests. I have a very simple test which compares the output of the test install/build with the known output. I put the test in a makefile.
The Makefile entry looks like this:
test:clean
make install DESTDIR=$(TEST_DIR)
$(TEST_DIR)/path/to/executable > $(TEST_DIR)/tmp.out
diff test/test.result $(TEST_DIR)/tmp.out
When the diff passes, an exit code of 0 is returned, a exit code of 1 is returned if the diff shows a difference in the files.
What I've tried:
Running make test from any shell runs the tests and exits, regardless of diff result
Running make test from the shell as gitlab_ci_runner runs the tests and exists regardless of diff result
When ran from Gitlab-CI, and the diff exit status is 0, the build returns success
The problem:
When ran in the Gitlab-CI and the diff exit status is non-0, the build hangs.
The output on the build screen is the output of the diff, and the last line is the expected error: make: *** [test] Error 1
After that, the cycle symbol keeps on, the runner does not exit with a build fail.
Any ideas? I thought that it might be something with Makefiles, but the Gitlab-CI will exit with a fail status if the Make exits with Error 1 for any other test. I can only see it happening on the output of the diff.
Thanks!
Also posted this to the GitLab mailinglist https://groups.google.com/d/msgid/gitlabhq/77e82813-b98e-4abe-9755-f39e07043384%40googlegroups.com?utm_medium=email&utm_source=footer
What is the meaning of return value 127 from $? in UNIX.
Value 127 is returned by /bin/sh when the given command is not found within your PATH system variable and it is not a built-in shell command. In other words, the system doesn't understand your command, because it doesn't know where to find the binary you're trying to call.
Generally it means:
127 - command not found
but it can also mean that the command is found,
but a library that is required by the command is NOT found.
127 - command not found
example: $caat
The error message will
bash:
caat: command not found
now you check using echo $?
A shell convention is that a successful executable should exit with the value 0. Anything else can be interpreted as a failure of some sort, on part of bash or the executable you that just ran. See also $PIPESTATUS and the EXIT STATUS section of the bash man page:
For the shell’s purposes, a command which exits with a zero exit status has succeeded. An exit status
of zero indicates success. A non-zero exit status indicates failure. When a command terminates on a
fatal signal N, bash uses the value of 128+N as the exit status.
If a command is not found, the child process created to execute it returns a status of 127. If a com-
mand is found but is not executable, the return status is 126.
If a command fails because of an error during expansion or redirection, the exit status is greater than
zero.
Shell builtin commands return a status of 0 (true) if successful, and non-zero (false) if an error
occurs while they execute. All builtins return an exit status of 2 to indicate incorrect usage.
Bash itself returns the exit status of the last command executed, unless a syntax error occurs, in
which case it exits with a non-zero value. See also the exit builtin command below.
It has no special meaning, other than that the last process to exit did so with an exit status of 127.
However, it is also used by bash (assuming you're using bash as a shell) to tell you that the command you tried to execute couldn't be executed (i.e. it couldn't be found). It's unfortunately not immediately deducible though, if the process exited with status 127, or if it couldn't found.
EDIT:
Not immediately deducible, except for the output on the console, but this is stack overflow, so I assume you're doing this in a script.
If you're trying to run a program using a scripting language, you may need to include the full path of the scripting language and the file to execute. For example:
exec('/usr/local/bin/node /usr/local/lib/node_modules/uglifycss/uglifycss in.css > out.css');
This error is also at times deceiving. It says file is not found even though the files is indeed present. It could be because of invalid unreadable special characters present in the files that could be caused by the editor you are using. This link might help you in such cases.
-bash: ./my_script: /bin/bash^M: bad interpreter: No such file or directory
The best way to find out if it is this issue is to simple place an echo statement in the entire file and verify if the same error is thrown.
If the IBM mainframe JCL has some extra characters or numbers at the end of the name of unix script being called then it can throw such error.
In addition to the given answers, note that running a script file with incorrect end-of-line characters could also result in 127 exit code if you use /bin/sh as your shell.
As an example, if you run a shell script with CRLF end-of-line characters in a UNIX-based system and in the /bin/sh shell, it is possible to encounter some errors like the following I've got after running my script named my_test.sh :
$ ./my_test.sh
sh: 2: ./my_test.sh: not found
$ echo $?
127
As a note, using /bin/bash, I got 126 exit code, which is in accordance with gnu.org documentation about the bash :
If a command is not found, the child process created to execute it returns a status of 127. If a command is found but is not executable, the return status is 126.
Finally, here is the result of running my script in /bin/bash :
arman#Debian-1100:~$ ./my_test.sh
-bash: ./my_test.sh: /bin/bash^M: bad interpreter: No such file or directory
arman#Debian-1100:~$ echo $?
126
go to C:\Program Files\Git\etc
open gitconfig with notepad
change
[core]
autocrlf = true
To
[core]
autocrlf = false