I am working on a node.js application and would like to know if there is a way I can run all the unit tests from all the sub modules even if there are some test failures to know how many tests are failing in total to start putting the fixes for them. We use mocha for our tests on the back-end and jest for the ui.
Thanks.
The default behavior for mocha is to run all the tests. If it is exiting after the first test failure, that would suggest that you are using the "bail" option typically enabled on the command line with either --bail or -b.
Relevant docs: https://mochajs.org/#-bail-b
It can also be caused by passing the option { bail: true } to mocha.setup(). Look in your test runner and in your package.json.
Lastly, the least likely of these possibilities is that it could also be caused by using this.bail() somewhere in the Mocha test runner.
Related
Is there a way to execute TestNG.xml in dry run mode so that I can figure out what methods gets qualified for the test run. I am using Eclipse and intend to run the tests via testng.xml. How to configure Run Configurations for this.
Newbie to Selenium-TestNG and Eclipse
I tried to provide -Dtestng.mode.dryrun=true in Run Configurations -> Arguments tab under both Program argument and VM arguments
The run configurations had no effect on the execution. The tests were executed in normal fashion. I expected the configurations would just list test methods in the console
You are going to see all tests with no failures. That what you expect when you run testNg with the argument.
To check the dryrun argument works, make your test to fail. Then run your test with "-Dtestng.mode.dryrun=true". Add the argument in "VM arguments"
Also, check your version of testNG is 6.14 or higher
I am trying to set up an automated tests using PHPUnit and Selenium with headless firefox. When Travis CI tries to run my tests, Selenium server fails to start, but my test is considered OK, because PHPUnit marks it as skipped:
The Selenium Server is not active on host localhost at port 4444.
OK, but incomplete, skipped, or risky tests!
Tests: 1, Assertions: 0, Skipped: 1.
The command "make test" exited with 0.
In my opinion, test should be considered as failed when it couldn't be even started for internal error. This is really stupid as my tests could fail this way and if didn't read the full report, I could believe everything is in a fact running okay, because Travis CI considers return value 0 to be successful test.
Is there an option to make PHPUnit return non-zero result when there are skipped tests? Or even make it directly report test as failed on Selenium error?
Update:
See the accepted answer ( --fail-on-skipped ), added in version 9.4.3 ( about two years after the question was open )
Answer for previous versions:
Consider configuring the following parameters in your phpunit.xml file.
stopOnError="true"
stopOnFailure="true"
stopOnIncomplete="true"
stopOnSkipped="true"
stopOnRisky="true"
Reference
If you want to use the commandline args equivalents are:
--stop-on-error Stop execution upon first error.
--stop-on-failure Stop execution upon first error or failure.
--stop-on-warning Stop execution upon first warning.
--stop-on-risky Stop execution upon first risky test.
--stop-on-skipped Stop execution upon first skipped test.
--stop-on-incomplete Stop execution upon first incomplete test.
For your case, you want to stop on skipped.
SIDENOTE For a test to be considered FAILED there must exist a failing assertion. Since skipped tests are not actually executed, you cannot consider them as failed, you should rely on the execution summary and make sure that no risky or skipped tests took place.
If you want your risky and warned tests to be considered as a FAILING TEST, you may use the following args:
--fail-on-warning Treat tests with warnings as failures.
--fail-on-risky Treat risky tests as failures.
Reference
If for any reason you would like to make PHPUnit return an exit code other than 0 (success), consider taking a look to How to make PHPunit return nonzero exit status on warnings
Since version 9.4.3, PHPUnit has a --fail-on-skipped option:
$ vendor/bin/phpunit --help
(...)
--fail-on-incomplete Treat incomplete tests as failures
--fail-on-risky Treat risky tests as failures
--fail-on-skipped Treat skipped tests as failures
--fail-on-warning Treat tests with warnings as failures
(...)
seems like something quite basic but it dosen't seem to be available in jest automatically.
when i run my jest tests, during the test run i get the logging info from the code itself while the tests are run.
and only after the tests are finished i receive a summary of the spec results(which tests passed and which didn't).
what i'm trying to achieve is to have extra lines in my console, that fill while the tests are running, that tell which test we have just finished, and what was the result.. this will make log debugging so much simpler..
thanks!
How do I configure my unit tests to run automatically with Elm-Live?
I currently run elm-live as follows:
elm-live Home.elm --open --output=home.js
In addition to having automated compilations per modification of my web app, I would also like to ensure that I did not introduce breaking changes as well by having unit tests execute automatically after compiling.
Any suggestions?
You can use concurrently to run both processes in the same terminal instance.
The downside is that the stdout will probably not preserve the colors, so reading errors will be a little tricky.
concurrently 'elm-live Home.elm --open --output=home.js' 'elm-test --watch'
Example
I've made an example of this setup, check it out on GitHub.
UPD: I have updated the example to be Windows-compatible. Apparently, it should have escaped double quotes on the package.json instead of single quotes.
Is it expected that benchmarks don't run unless all tests in the package have passed.
I've looked at the testing package doc and the testing flags and I can't find it documented that benchmarks run only after all tests pass.
Is there a way to force benchmark functions to run even when some tests in the package have failed ?
You can skip the failing tests using the -run flag, or choose to run none at all
go test -bench . -run NONE