I've got some tests setup to run via Taurus and kicking them off in a Jenkins stage like so:
...previous stages...
stage('Load Tests'){
dir('./tests'){
bat "bat _testFile.yml"
}
}
...stages to execute if Load Tests stage succeeds
I want to bail out of the whole build if any one of the iterations in any of my tests fails. But this setup, as well as wrapping in a try/catch don't work.
My fault. The fail criteria were not properly implemented. Now that they are, the above 'successfully' fails the jenkins build when the criteria are not met.
Related
I am working on a node.js application and would like to know if there is a way I can run all the unit tests from all the sub modules even if there are some test failures to know how many tests are failing in total to start putting the fixes for them. We use mocha for our tests on the back-end and jest for the ui.
Thanks.
The default behavior for mocha is to run all the tests. If it is exiting after the first test failure, that would suggest that you are using the "bail" option typically enabled on the command line with either --bail or -b.
Relevant docs: https://mochajs.org/#-bail-b
It can also be caused by passing the option { bail: true } to mocha.setup(). Look in your test runner and in your package.json.
Lastly, the least likely of these possibilities is that it could also be caused by using this.bail() somewhere in the Mocha test runner.
Is there a way to execute TestNG.xml in dry run mode so that I can figure out what methods gets qualified for the test run. I am using Eclipse and intend to run the tests via testng.xml. How to configure Run Configurations for this.
Newbie to Selenium-TestNG and Eclipse
I tried to provide -Dtestng.mode.dryrun=true in Run Configurations -> Arguments tab under both Program argument and VM arguments
The run configurations had no effect on the execution. The tests were executed in normal fashion. I expected the configurations would just list test methods in the console
You are going to see all tests with no failures. That what you expect when you run testNg with the argument.
To check the dryrun argument works, make your test to fail. Then run your test with "-Dtestng.mode.dryrun=true". Add the argument in "VM arguments"
Also, check your version of testNG is 6.14 or higher
I am trying to set up an automated tests using PHPUnit and Selenium with headless firefox. When Travis CI tries to run my tests, Selenium server fails to start, but my test is considered OK, because PHPUnit marks it as skipped:
The Selenium Server is not active on host localhost at port 4444.
OK, but incomplete, skipped, or risky tests!
Tests: 1, Assertions: 0, Skipped: 1.
The command "make test" exited with 0.
In my opinion, test should be considered as failed when it couldn't be even started for internal error. This is really stupid as my tests could fail this way and if didn't read the full report, I could believe everything is in a fact running okay, because Travis CI considers return value 0 to be successful test.
Is there an option to make PHPUnit return non-zero result when there are skipped tests? Or even make it directly report test as failed on Selenium error?
Update:
See the accepted answer ( --fail-on-skipped ), added in version 9.4.3 ( about two years after the question was open )
Answer for previous versions:
Consider configuring the following parameters in your phpunit.xml file.
stopOnError="true"
stopOnFailure="true"
stopOnIncomplete="true"
stopOnSkipped="true"
stopOnRisky="true"
Reference
If you want to use the commandline args equivalents are:
--stop-on-error Stop execution upon first error.
--stop-on-failure Stop execution upon first error or failure.
--stop-on-warning Stop execution upon first warning.
--stop-on-risky Stop execution upon first risky test.
--stop-on-skipped Stop execution upon first skipped test.
--stop-on-incomplete Stop execution upon first incomplete test.
For your case, you want to stop on skipped.
SIDENOTE For a test to be considered FAILED there must exist a failing assertion. Since skipped tests are not actually executed, you cannot consider them as failed, you should rely on the execution summary and make sure that no risky or skipped tests took place.
If you want your risky and warned tests to be considered as a FAILING TEST, you may use the following args:
--fail-on-warning Treat tests with warnings as failures.
--fail-on-risky Treat risky tests as failures.
Reference
If for any reason you would like to make PHPUnit return an exit code other than 0 (success), consider taking a look to How to make PHPunit return nonzero exit status on warnings
Since version 9.4.3, PHPUnit has a --fail-on-skipped option:
$ vendor/bin/phpunit --help
(...)
--fail-on-incomplete Treat incomplete tests as failures
--fail-on-risky Treat risky tests as failures
--fail-on-skipped Treat skipped tests as failures
--fail-on-warning Treat tests with warnings as failures
(...)
I have Groovy Script in my Test Case in SOAP UI.
There are one big script and a lot of test steps which are run by this script. I know about functionality of TestRunner but it run all steps in test case, but i need to run only my groovy script. How?
UPDATE
I disable all Test steps and left active only groovy script. Then I Launch TestRunner on Test Case, it return exception :
java.lang.Exception: TestCase [Case1] failed without assertions
at com.eviware.soapui.tools.SoapUITestCaseRunner.throwFailureException(SoapUITestCaseRunner.java:535)
at com.eviware.soapui.tools.SoapUITestCaseRunner.runRunner(SoapUITestCaseRunner.java:437)
at com.eviware.soapui.tools.AbstractSoapUIRunner.run(AbstractSoapUIRunner.java:162)
at com.eviware.soapui.tools.AbstractSoapUIRunner.runFromCommandLine(AbstractSoapUIRunner.java:93)
at com.eviware.soapui.tools.SoapUITestCaseRunner.main(SoapUITestCaseRunner.java:119)
Just disable the rest of the steps and run the test case. The disabled test steps are not run automatically by soapui, you can call those conditionally :)
I am creating a TeamCity build configuration with a few steps. One of those steps is running tests using MSTest. Currently, my tests are failing (by design so I can test the build process), but the build steps after the step for running the tests happen, even though I can see the test process exits with code 1 (ie, something other than code 0) in the build log. It does mark the build as failed, but I'd prefer it if the steps in the build stopped once failing tests were detected. Is that possible and I'm just missing something in the configuration?
When you create build step there are select Execute step for execution policy. Where you should select Only if all previous steps were successful.
Here is it:
But in your case there are bug in the MsTest and NUnit build steps. Here is workaround. And here is related issue.