I am trying to set up an automated tests using PHPUnit and Selenium with headless firefox. When Travis CI tries to run my tests, Selenium server fails to start, but my test is considered OK, because PHPUnit marks it as skipped:
The Selenium Server is not active on host localhost at port 4444.
OK, but incomplete, skipped, or risky tests!
Tests: 1, Assertions: 0, Skipped: 1.
The command "make test" exited with 0.
In my opinion, test should be considered as failed when it couldn't be even started for internal error. This is really stupid as my tests could fail this way and if didn't read the full report, I could believe everything is in a fact running okay, because Travis CI considers return value 0 to be successful test.
Is there an option to make PHPUnit return non-zero result when there are skipped tests? Or even make it directly report test as failed on Selenium error?
Update:
See the accepted answer ( --fail-on-skipped ), added in version 9.4.3 ( about two years after the question was open )
Answer for previous versions:
Consider configuring the following parameters in your phpunit.xml file.
stopOnError="true"
stopOnFailure="true"
stopOnIncomplete="true"
stopOnSkipped="true"
stopOnRisky="true"
Reference
If you want to use the commandline args equivalents are:
--stop-on-error Stop execution upon first error.
--stop-on-failure Stop execution upon first error or failure.
--stop-on-warning Stop execution upon first warning.
--stop-on-risky Stop execution upon first risky test.
--stop-on-skipped Stop execution upon first skipped test.
--stop-on-incomplete Stop execution upon first incomplete test.
For your case, you want to stop on skipped.
SIDENOTE For a test to be considered FAILED there must exist a failing assertion. Since skipped tests are not actually executed, you cannot consider them as failed, you should rely on the execution summary and make sure that no risky or skipped tests took place.
If you want your risky and warned tests to be considered as a FAILING TEST, you may use the following args:
--fail-on-warning Treat tests with warnings as failures.
--fail-on-risky Treat risky tests as failures.
Reference
If for any reason you would like to make PHPUnit return an exit code other than 0 (success), consider taking a look to How to make PHPunit return nonzero exit status on warnings
Since version 9.4.3, PHPUnit has a --fail-on-skipped option:
$ vendor/bin/phpunit --help
(...)
--fail-on-incomplete Treat incomplete tests as failures
--fail-on-risky Treat risky tests as failures
--fail-on-skipped Treat skipped tests as failures
--fail-on-warning Treat tests with warnings as failures
(...)
Related
As mentioned in the title, the cypress client proceeds on to other tests without marking a test as failed where it has in fact found a mismatch in the expectation. This behavior can be seen in the attached image:
This is intermittent but can go completely unnoticed when the tests run in a CI environment.
How am I supposed to debug & fix this issue?
Is there a way to tell the system to restart the test in case a specific rare system error comes up?
Basically sometimes we get strange errors related to elements being "obscured" or "stale", but which do not mean the site is not working etc. It has to do with the site's latency I believe like CSS not loading quickly enough etc.
For example is there a directive to tell the system that if an error like
[Facebook\WebDriver\Exception\ElementClickInterceptedException] Element
\<li id="nav_step0" class="nav-steps selected"> is not clickable at point (330,237)
because another element \<div id="ajaxloading_mask" class="mask"> obscures it
To simply relaunch the test again?
No, there is no way to relaunch failed test on specific error.
You can rerun all failed tests:
codecep run || codecept run -g failed
This command executes all tests, if any tests failed, it reruns only failed tests.
I am working on a node.js application and would like to know if there is a way I can run all the unit tests from all the sub modules even if there are some test failures to know how many tests are failing in total to start putting the fixes for them. We use mocha for our tests on the back-end and jest for the ui.
Thanks.
The default behavior for mocha is to run all the tests. If it is exiting after the first test failure, that would suggest that you are using the "bail" option typically enabled on the command line with either --bail or -b.
Relevant docs: https://mochajs.org/#-bail-b
It can also be caused by passing the option { bail: true } to mocha.setup(). Look in your test runner and in your package.json.
Lastly, the least likely of these possibilities is that it could also be caused by using this.bail() somewhere in the Mocha test runner.
I've got some tests setup to run via Taurus and kicking them off in a Jenkins stage like so:
...previous stages...
stage('Load Tests'){
dir('./tests'){
bat "bat _testFile.yml"
}
}
...stages to execute if Load Tests stage succeeds
I want to bail out of the whole build if any one of the iterations in any of my tests fails. But this setup, as well as wrapping in a try/catch don't work.
My fault. The fail criteria were not properly implemented. Now that they are, the above 'successfully' fails the jenkins build when the criteria are not met.
Allure always set BROKEN status for tests that failed before assertion.
But often tests fail because of existing bug before assertion (like login not working).
Is there a possibility to mark tests that have issue assigned and have failed - as FAILED, not BROKEN in Allure?
The best way is to fix it in Allure adaptor and replace default one with your version.
I have done the same with java adaptor, we were need to add additional execeptions as FAILED and some description to all tests.