mochawesome cypress failure report - testing

I am using cypress and mochawesome to generate reports for testing. I want to be alerted only when there is a failure. Is it possible to have the number of failing tests without parsing the json file?

The exit code of the cypress process will give you the number of failed tests:
npm run cypress
# ... cypress runs...
echo $? # print number of failed tests
Or for Windows cmd prompt: print exit code in cmd in windows os

Related

How to access log files created by GCP cloud build steps?

My Cloud build fails with a timeout on npm test, and no useful information is sent to stdout. A complete log can be found in a file, but I couldn't find a way to ssh in the cloud build environment.
Already have image: node
> project-client#v3.0.14 test
> jest
ts-jest[versions] (WARN) Version 4.1.0-beta of typescript installed has not been tested with ts-jest. If you're experiencing issues, consider using a supported version (>=3.8.0 <5.0.0-0). Please do not report issues in ts-jest if you are using unsupported versions.
npm ERR! path /workspace/v3/client
npm ERR! command failed
npm ERR! signal SIGTERM
npm ERR! command sh -c jest
npm ERR! A complete log of this run can be found in:
npm ERR! /builder/home/.npm/_logs/2020-11-09T07_56_23_573Z-debug.log
Since I have no problem running the tests locally, I'd like to see the content of that 2020-11-09T07_56_23_573Z-debug.log file to hopefully get a hint at what might be wrong.
Is there a way to retrieve the file content ?
Ssh in a cloud build environment?
Get npm to print the complete log to stdout?
Some way to save the log file artifact to cloud storage ?
I had a similar issue with error management on Gitlab CI and my workaround is inspired from there.
The trick is to embed your command in something that exit with a return code 0. Here an example
- name: node
entrypoint: "bash"
args:
- "-c"
- |
RETURN_CODE=$$(jtest > log.stdout 2>log.stderr;echo $${?})
cat log.stdout
cat log.stderr
if [ $${RETURN_CODE} -gt 0 ];
then
#Do what you want in case of error, like a cat of the files in the _logs dir
# Break the build: exit 1
else
#Do what you want in case of success. Nothing to continue to the next step
fi
Some explanations:
echo $${?}: the double $ is to indicate to Cloud Build to not use substitution variables but to ignore it and let Linux command being interpreted.
The $? allows you to get the exit code of the previous command
Then you test the exit code, if > 0, you can perform actions. At the end, I recommend to break the build to not continue with erroneous sources.
You can parse the log.stderr file to get useful info in it (the log file for example)

Is it possible to get the filename of the failed test in the _failed() hook?

If I am able to get the filename of the failed test, I could run the test a second time to check if it really is a error or just a coincidence. Until now I couldnt find a sulution.
You get the instance of test in _failed hook, so you can do the same thing as RunFailed extension:
use Codeception\Test\Descriptor;
public function _failed(\Codeception\TestInterface $test, $fail)
{
$fullName = Descriptor::getTestFullName($test);
}
Original answer:
Codeception has RunFailed extension, which is enabled by default.
It saves the names of all failed tests to tests/_output/failed file, and failed tests can be executed by running codecept run -g failed command.
In order to make CI pass if failed tests pass on second run you can use this command:
codecept run || codecept run -g failed

Getting Protractor.js to run on Jenkins build failing -- Error forwarding the new session empty pool of VM for setup capabilities

I am trying to get Protactor.js unit testing up and running with Jenkins on a windows server but I am having trouble getting it to run.
Currently this is the steps I have so far:
#install protractor via package.json
cd ./src/My\ Editor/WebApp/Scripts/protractor && D:Jenkins/nodesjs/npm install
[other build steps here]
#start webdriver-manager
./src/My\ Editor/WebApp/Scripts/protractor/node_modules/protractor/bin/webdriver-manager start > /dev/null 2>&1 &
#wait until selenium up
while ! curl http://localhost:4444/wd/hub/status &>/dev/null
do
true
done
#run protractor
cd ./src/My\ Editor/WebApp/Scripts/protractor/node_modules/protractor/bin/
./protractor ../../../conf.js
#stop selenium
curl -s -L http://localhost:4444/selenium-server/driver?cmd-shutDownSeleniumServer > /dev/null 2>&1
everything works well until the command 'protractor conf.js'. I get this output:
[21:12:34] I/hosted - Using the selenium server at http://localhost:4444/wd/hub
[21:12:34] I/launcher - Running 1 instances of WebDriver
[21:12:34] E/launcher - Error forwarding the new session empty pool of VM for setup capabilities [{count=1, browserName=chrome}]
Any help for what I am doing wrong is appreciated, thank you.
Is there a specific need to start a Selenium Server/hub?
If you dont have a plan to setup a Selenium GRID and you just need to run protractor test scripts stand-alone you can skip the part where you are starting server.
Can you leave the remove seleniumAddress option in Protractor.conf.js. This will enable Protractor to start its own selenium server & wait till server is completely up and execute and will shutdown the server once execution is done
More information here
/**
* The address of a running Selenium Server. If specified, Protractor will
* connect to an already running instance of Selenium. This usually looks like
* seleniumAddress: 'http://localhost:4444/wd/hub'
*/
seleniumAddress?: string;
just have the step #run protractor and rest is taken care by Protractor

How to break Travis CI build if Appium/Mocha tests fail?

I have a Travis CI project which builds an iOS app then starts Appium and runs tests with Appium/Mocha.
The problem is that even though the Mocha tests fail and throw exception, the shell script which runs them via Gulp still exits with 0 and the build is deemed passing.
How can I make the build break/fail when the Mocha tests fail?
Here is how I managed to make this work:
Instead of running the Mocha tests via Gulp, run them directly from the shell script
Save the output to mocha.log besides displaying on stdout
./node_modules/.bin/mocha --reporter spec "appium/hybrid/*uat.js" 2>&1 | tee mocha.log
Check mocha.log for the string " failing" and exit with 1 if found
.
if grep -q " failing" mocha.log; then
exit 1
fi
The exit 1 will make the Travis build fail.

Codeception auto start and auto shutdown selenium server?

I'm using Codeception and I want to do Selenium tests.
Obviously, I need Selenium server running and I can spin it up with:
java -jar ./path/to/selenium/binary
However, I've used another testing framework in past which allowed me to specify this path in config file. Then, whenever I did something like codecept run, the framework automatically started selenium server and also shutted it down after all tests were completed.
Can I do this with Codeception? I tried to put exec() in _bootstrap file but it didn't work..
I'm using phantomjs - headless alternative to selenium. This extension starts phantomjs automatically with codecept run. Also phantomjs is faster than selenium.
You can write your own extension for running selenium - have a look at this file.
I start my tests with a bash script and then start and stop selenium via screen:
#!/usr/bin/env bash
# Start selenium in detached screen
screen -d -m -S "selenium" java -Dwebdriver.chrome.driver=./path/to/chromedriver -jar ./path/to/selenium/binary || true
./vendor/bin/codecept run -c codeception.yml --fail-fast --coverage --steps --coverage-html --html || error=true
# quit selenium
screen -S selenium -X quit || true
This might not fit the use-case but I can execute my code in my bash on Windows and Ubuntu as well as via my jenkins build process.