I am using cypress and mochawesome to generate reports for testing. I want to be alerted only when there is a failure. Is it possible to have the number of failing tests without parsing the json file?
The exit code of the cypress process will give you the number of failed tests:
npm run cypress
# ... cypress runs...
echo $? # print number of failed tests
Or for Windows cmd prompt: print exit code in cmd in windows os
Related
My Cloud build fails with a timeout on npm test, and no useful information is sent to stdout. A complete log can be found in a file, but I couldn't find a way to ssh in the cloud build environment.
Already have image: node
> project-client#v3.0.14 test
> jest
ts-jest[versions] (WARN) Version 4.1.0-beta of typescript installed has not been tested with ts-jest. If you're experiencing issues, consider using a supported version (>=3.8.0 <5.0.0-0). Please do not report issues in ts-jest if you are using unsupported versions.
npm ERR! path /workspace/v3/client
npm ERR! command failed
npm ERR! signal SIGTERM
npm ERR! command sh -c jest
npm ERR! A complete log of this run can be found in:
npm ERR! /builder/home/.npm/_logs/2020-11-09T07_56_23_573Z-debug.log
Since I have no problem running the tests locally, I'd like to see the content of that 2020-11-09T07_56_23_573Z-debug.log file to hopefully get a hint at what might be wrong.
Is there a way to retrieve the file content ?
Ssh in a cloud build environment?
Get npm to print the complete log to stdout?
Some way to save the log file artifact to cloud storage ?
I had a similar issue with error management on Gitlab CI and my workaround is inspired from there.
The trick is to embed your command in something that exit with a return code 0. Here an example
- name: node
entrypoint: "bash"
args:
- "-c"
- |
RETURN_CODE=$$(jtest > log.stdout 2>log.stderr;echo $${?})
cat log.stdout
cat log.stderr
if [ $${RETURN_CODE} -gt 0 ];
then
#Do what you want in case of error, like a cat of the files in the _logs dir
# Break the build: exit 1
else
#Do what you want in case of success. Nothing to continue to the next step
fi
Some explanations:
echo $${?}: the double $ is to indicate to Cloud Build to not use substitution variables but to ignore it and let Linux command being interpreted.
The $? allows you to get the exit code of the previous command
Then you test the exit code, if > 0, you can perform actions. At the end, I recommend to break the build to not continue with erroneous sources.
You can parse the log.stderr file to get useful info in it (the log file for example)
If I am able to get the filename of the failed test, I could run the test a second time to check if it really is a error or just a coincidence. Until now I couldnt find a sulution.
You get the instance of test in _failed hook, so you can do the same thing as RunFailed extension:
use Codeception\Test\Descriptor;
public function _failed(\Codeception\TestInterface $test, $fail)
{
$fullName = Descriptor::getTestFullName($test);
}
Original answer:
Codeception has RunFailed extension, which is enabled by default.
It saves the names of all failed tests to tests/_output/failed file, and failed tests can be executed by running codecept run -g failed command.
In order to make CI pass if failed tests pass on second run you can use this command:
codecept run || codecept run -g failed
I am trying to get Protactor.js unit testing up and running with Jenkins on a windows server but I am having trouble getting it to run.
Currently this is the steps I have so far:
#install protractor via package.json
cd ./src/My\ Editor/WebApp/Scripts/protractor && D:Jenkins/nodesjs/npm install
[other build steps here]
#start webdriver-manager
./src/My\ Editor/WebApp/Scripts/protractor/node_modules/protractor/bin/webdriver-manager start > /dev/null 2>&1 &
#wait until selenium up
while ! curl http://localhost:4444/wd/hub/status &>/dev/null
do
true
done
#run protractor
cd ./src/My\ Editor/WebApp/Scripts/protractor/node_modules/protractor/bin/
./protractor ../../../conf.js
#stop selenium
curl -s -L http://localhost:4444/selenium-server/driver?cmd-shutDownSeleniumServer > /dev/null 2>&1
everything works well until the command 'protractor conf.js'. I get this output:
[21:12:34] I/hosted - Using the selenium server at http://localhost:4444/wd/hub
[21:12:34] I/launcher - Running 1 instances of WebDriver
[21:12:34] E/launcher - Error forwarding the new session empty pool of VM for setup capabilities [{count=1, browserName=chrome}]
Any help for what I am doing wrong is appreciated, thank you.
Is there a specific need to start a Selenium Server/hub?
If you dont have a plan to setup a Selenium GRID and you just need to run protractor test scripts stand-alone you can skip the part where you are starting server.
Can you leave the remove seleniumAddress option in Protractor.conf.js. This will enable Protractor to start its own selenium server & wait till server is completely up and execute and will shutdown the server once execution is done
More information here
/**
* The address of a running Selenium Server. If specified, Protractor will
* connect to an already running instance of Selenium. This usually looks like
* seleniumAddress: 'http://localhost:4444/wd/hub'
*/
seleniumAddress?: string;
just have the step #run protractor and rest is taken care by Protractor
I have a Travis CI project which builds an iOS app then starts Appium and runs tests with Appium/Mocha.
The problem is that even though the Mocha tests fail and throw exception, the shell script which runs them via Gulp still exits with 0 and the build is deemed passing.
How can I make the build break/fail when the Mocha tests fail?
Here is how I managed to make this work:
Instead of running the Mocha tests via Gulp, run them directly from the shell script
Save the output to mocha.log besides displaying on stdout
./node_modules/.bin/mocha --reporter spec "appium/hybrid/*uat.js" 2>&1 | tee mocha.log
Check mocha.log for the string " failing" and exit with 1 if found
.
if grep -q " failing" mocha.log; then
exit 1
fi
The exit 1 will make the Travis build fail.
I'm using Codeception and I want to do Selenium tests.
Obviously, I need Selenium server running and I can spin it up with:
java -jar ./path/to/selenium/binary
However, I've used another testing framework in past which allowed me to specify this path in config file. Then, whenever I did something like codecept run, the framework automatically started selenium server and also shutted it down after all tests were completed.
Can I do this with Codeception? I tried to put exec() in _bootstrap file but it didn't work..
I'm using phantomjs - headless alternative to selenium. This extension starts phantomjs automatically with codecept run. Also phantomjs is faster than selenium.
You can write your own extension for running selenium - have a look at this file.
I start my tests with a bash script and then start and stop selenium via screen:
#!/usr/bin/env bash
# Start selenium in detached screen
screen -d -m -S "selenium" java -Dwebdriver.chrome.driver=./path/to/chromedriver -jar ./path/to/selenium/binary || true
./vendor/bin/codecept run -c codeception.yml --fail-fast --coverage --steps --coverage-html --html || error=true
# quit selenium
screen -S selenium -X quit || true
This might not fit the use-case but I can execute my code in my bash on Windows and Ubuntu as well as via my jenkins build process.