If I am able to get the filename of the failed test, I could run the test a second time to check if it really is a error or just a coincidence. Until now I couldnt find a sulution.
You get the instance of test in _failed hook, so you can do the same thing as RunFailed extension:
use Codeception\Test\Descriptor;
public function _failed(\Codeception\TestInterface $test, $fail)
{
$fullName = Descriptor::getTestFullName($test);
}
Original answer:
Codeception has RunFailed extension, which is enabled by default.
It saves the names of all failed tests to tests/_output/failed file, and failed tests can be executed by running codecept run -g failed command.
In order to make CI pass if failed tests pass on second run you can use this command:
codecept run || codecept run -g failed
Related
Trying to run my postman collection in azure devops inside a self-hosted agent. When I try to run the command inside the agent "newman run postman_collection.json -e postman_environment.json -r cli,htmlextra" it's running fine. But when I run the same through a a command line script task in release pipeline it's throwing the error "newman is not recognized..". I also tried to have a npm task for newman installation i.e. "npm install -g newman" it's also throwing the erro "##[error]Unable to locate executable file: 'newman'. Please verify either the file path exists or the file can be found within a d...."
azure devop selfhosted agent, newman command not recognized
According to the error message "##[error]Unable to locate executable file: 'newman" when you using the npm install -g newman, you could try to add C:\Users\[BUILDSERVER-USERNAME]\AppData\Roaming\npm to the PATH variable for the [BUILDSERVER-USERNAME] user.
You could refer to this document How to fix the Newman task for Team Foundation Server silently failing for some more details.
Besides, when we use command line to install the newman, it will take a few minutes to install it, so we need to wait for a few minutes before we using the command line:
"newman run postman_collection.json -e postman_environment.json -r cli,htmlextra"
You could add powershell task to sleep a few minutes：
echo "Sleeping for 10 mins..."
Start-Sleep -s 600
I am working with Codeception Acceptance Tests in docker containers.
I can run the whole project, but I can't run single tests or classes.
Is there any configuration option or parameter to solve this?
This works (run all tests):
codecept -c /app/web/codeception.yml run
This does not work:
codecept -c /app/web/codeception.yml run SupplierFormCest (class name)
codecept -c /app/web/codeception.yml run Tests/SupplierFormCest (Namespace path)
codecept -c /app/web/codeception.yml run /app/web/tests/acceptance/SupplierFormCest (file path)
Error Message: Suite 'SupplierFormCest' could not be found
Looks like you need to specify the suite, acceptance, before the class name:
codecept -c /app/web/codeception.yml run acceptance SupplierFormCest
I am using cypress and mochawesome to generate reports for testing. I want to be alerted only when there is a failure. Is it possible to have the number of failing tests without parsing the json file?
The exit code of the cypress process will give you the number of failed tests:
npm run cypress
# ... cypress runs...
echo $? # print number of failed tests
Or for Windows cmd prompt: print exit code in cmd in windows os
I am Pretty new to Team City and have been set with a task of creating a CI build.
The thing I trying to build is an angular2 app with protractor e2e tests.
All the other build steps in Team City run ok but I am having trouble trying to run the step that does the e2e test.
if I was to do this locally I would open a cmd window and type...
npm run start
I would then open another command window and type...
npm run e2e
How do I run parallel steps in Team City?
Build steps cannot be run parallel in TeamCity. What you need to do, is create a script that runs 'npm run start' in background, and then runs 'npm run e2e'. You can use command line runner to run the script
I still couldn't get the forever thing working properly for me so I created my own node script that fires up live-server and then executes npm run e2e and that seems to have done the trick thanks for your help though Oleg.
This is how I did it in the end...
const exec = require('child_process').exec;
var psTree = require('ps-tree');
const server = exec('live-server ./dist --port=3000 --no-browser');
const tests = exec('npm run e2e');
tests.stdout.on('data', function(data) {
console.log(data);
});
tests.stderr.on('data', function(data) {
console.log(data);
});
tests.on('close', function(code) {
console.log('closing code: ' + code);
exec('taskkill /PID ' + server.pid + ' /T /F');
});
I have a Travis CI project which builds an iOS app then starts Appium and runs tests with Appium/Mocha.
The problem is that even though the Mocha tests fail and throw exception, the shell script which runs them via Gulp still exits with 0 and the build is deemed passing.
How can I make the build break/fail when the Mocha tests fail?
Here is how I managed to make this work:
Instead of running the Mocha tests via Gulp, run them directly from the shell script
Save the output to mocha.log besides displaying on stdout
./node_modules/.bin/mocha --reporter spec "appium/hybrid/*uat.js" 2>&1 | tee mocha.log
Check mocha.log for the string " failing" and exit with 1 if found
.
if grep -q " failing" mocha.log; then
exit 1
fi
The exit 1 will make the Travis build fail.