so I have some flakiness in my e2e tests and want to use testcafe's quarantine mode to minimize pipeline failures. For some reason the tests wont rerun after failing.
testcafe 'chrome:headless --no-sandbox --autoplay-policy=no-user-gesture-required' --skip-js-errors --skip-uncaught-errors -q attemptLimit=5,successThreshold=1
thanks in advance
EDIT: I found out that it works, if I add the quarantine-mode key to the testcafe config file. CLI wise its just getting ignored somehow.
Related
When running my test suite in Selenium IDE, the tests pass as the IDE allows for enough time (30000) to find the elements.
When running my test suite using command line, the tests fail with the Timeout error below.
I have tried adding a timeout to the command by adding the words "--timeout [60000]" to the command line,
example:
selenium-side-runner --timeout [60000] -c "browserName=chrome" "Desktop/SIDE with CMD/CmdWithoutInit2.side"
I have also added "Pause" and various "waits" to the actual tests in Selenium IDE, all of which work in IDE but are null when running in Command.
TimeoutError: Waiting for element to be located By(css selector,
#select2-ProjectID-v1-container > .select2-selection__placeholder)
Wait timed out after 15051ms
I need a way to run the Selenium IDE (.side) test suite using command line. To do this I need CMD to give enough time for the test to find each element, like the IDE does. I am hoping there is a short few words I can add to the command itself.
The answer is to add --timeout 6000 to the command line.
Example, in command prompt, run test by entering
selenium-side-runner --timeout 60000 -c "browserName=chrome" "Desktop/Name.side"
Timeout syntax does not include square bracers, can remove --debug if needed, doesn't provide useful info on actual running test only on running side-runner itself.
I have a .side file generated by the Selenium IDE, which I need to run on CI using Jenkins.
I am running it as a build step with the following shell command:
selenium-side-runner /path/to/file.ide
The problem arises due to the fact that no matter if the selenium test fails, Jenkins always shows is as success.
In this thread it's suggested to upload the file as generic, but still, the commands to execute it are missing
How to upload a generic file into a Jenkins job?
I've found a possible solution to it on this posts, but I would appreciate having a cleaner way to solve this instead of parsing the results checking for errors.
How to mark a build unstable in Jenkins when running shell scripts
Is there a plugin able to run selenium .side files on Jenkins and this one showing the success/failures of the test?
You can generate a Junit test report file and then use the Jenkins Junit plugin after your tests execution.
selenium-side-runner --output-directory=results --output-format=junit
# Outputs results in `junit` frormat in `./results/projectName.xml'
Check the official documentation for more details.
my original aim was to run a headless selenium webdriver on a Raspberry Pi 3 (rasbian). After hours and hours of failing, I make a step back and now I only try to run chromium-browser which needed for the webdriver.
There... I recognize some errors after execute:
sudo ./chromium-browser --headless --no-sandbox --disable-gpu --disable-extensions
Error-Stack:
--disable-quic --enable-tcp-fast-open --disable-gpu-compositing --ppapi-flash-path=/usr/lib/chromium-browser/libpepflashplayer.so --ppapi-flash-args=enable_stagevideo_auto=0 --ppapi-flash-version=
[1015/183516.617458:ERROR:browser_main_loop.cc(670)] Failed to put Xlib into threaded mode.
[1015/183516.625190:ERROR:gpu_process_transport_factory.cc(1029)] Lost UI shared context.
I search for solution in internet but I found no results.
Following I have to add:
-everything works fine if run the webdriver on my windows system
-I reproduce the error on two complete different raspberry Pi's
-I also try to run on a Raspbian virtual machine
-I try to run the webdriver with iceweasel and geckodriver with the result "Error: connection refused"
So I am out of ideas, thankful for any response.
For others who are still struggling to find a solution:
It seems that the issue is with diplay resource when running the program on a remote server having multiple display.
You may try setting the DISPLAY environment parameter
export DISPLAY=:1.0;
Once I've built my container with my Meteor app in it, I'd really like to be able to go
docker run me/myapp velocity test-app --ci --once --settings settings-test.json
And have it exit with 0 if successful, in which case I'll push it to docker hub, deploy it somewhere etc.
However when I try this it just hangs:
[velocity] is in continuous integration mode
[velocity] mocha is starting a mirror at http://localhost:56381/.
[velocity] *** Meteor Tools is installing ***
This takes a few minutes the first time.
[velocity] You can see the mirror logs at: tail -/app/.meteor/local/log/mocha.log
I'm using jasmine as per https://github.com/meteor-velocity/velocity-examples (I started with Mocha, but switched over to see if it made any difference).
Inspecting my .meteor/local/log files I find jasmine-client- unit.log has this at the bottom:
WARN [watcher]: [39m Pattern "/app/tests/jasmine/client/unit/**/*-+(stub|stubs|mock|mocks).+(js|coffee|litcoffee|coffee.md)" does not match any file.
WARN [karma]: [39m No captured browser, open http://localhost:9876/
INFO [karma]: [39m Karma v0.13.9 server started at http://localhost:9876/
INFO [launcher]: [39m Starting browser Chrome
ERROR [launcher]: [39m No binary for Chrome browser on your platform.
Please, set "CHROME_BIN" env variable
Parent process ( 725 ) is dead! Exiting jasmine-client-unit
Chrome clearly isn't going to be available in docker - should phantomjs be installed at this point and specified as a the running option? I would have expected this to be the case by default if the --ci option has been specified?
Thanks.
Ideally you should be using a real browser such as Chrome or Firefox to do automated testing, not PhantomJS. You can run browsers headlessly using Xvfb.
These might be useful:
http://codeutopia.net/blog/2013/07/13/headless-chromefirefox-testing-in-nodejs-with-selenium-and-xvfb/
http://elementalselenium.com/tips/38-headless
Selenium is the way to go for testing. IMO the best setup would be to use a CI server like Jenkins or TeamCity, with Xvfb installed, and have a test/deploy shell script in Jenkins such as:
#!/bin/sh
set -ex
cd $WORKSPACE
export VELOCITY_CI=1
meteor --test --settings $WORKSPACE/.deploy-staging/settings.json
cd .deploy-staging
mupx setup
mupx deploy
(Note that Xvfb is not implemented here, though). This test is not entirely working, I have yet to jump into Xvfb myself, though I know this is the right direction.
I'm using mupx to deploy my apps, which automatically creates a Docker instance on the remote server for me and handles deployment completely.
I'm creating my tests (though I'm a beginner, learning) using Codeception. This includes acceptance and unit tests for now.
I want to add my repo to Travis CI so I can automate testing process after each commit and put build-status tag.
I would like to ask;
Can Travis-CI run codeception tests?
Can Travis-CI run codeception acceptance tests emulating browser?
If both answers are no, is there any other CI tool which can?
Thank you.
Yes, it is possible to run Codeception tests, including acceptance tests that run using WebDriver, on Travis CI.
It is possible to run your tests with a real browser on Travis, but it is easiest to use a headless browser, since Travis is running on a headless machine. PhantomJS is perfect for this, and it comes pre-installed with Travis CI's build bootstrap.
To run the tests with PhantomJS, you'll need to configure the WebDriver module like this in your .yml Codeception configuration file:
modules:
config:
WPWebDriver:
url: 'http://127.0.0.1:8888'
browser: phantomjs
The URL is important. I have found that attempting to use localhost instead of 127.0.0.1 will not work. Also, if you accidentally leave out the http://, that won't work either. You can use most any 8*** port, since most of them are open, but of course you'll need to have a web server running on that port to serve your static files or run your PHP application. The easiest way to do this, I find, is to use PHP's built-in webserver.
Your .travis.yml file might look something like this:
# Travis CI configuration file.
language: php
php:
- 5.6
- 7.0
before_script:
# Start up a web server.
- php -S 127.0.0.1:8888 -t /path/to/web/root >/dev/null 2>&1 &
# Start up the webdriver.
- phantomjs --webdriver=4444 >/dev/null 2>&1 &
# Install Codeception.
# Doing this last gives the webdriver and server time to start up.
- composer install --prefer-source
script:
- vendor/bin/codecept run
You will of course need to add Codeception to your project's composer.json file:
composer require --dev codeception/codeception
You'll also need to change path/to/web/root above to the path to the directory where you want the server's document root to be.
If you'd like to see a working demo running WebDriver tests against WordPress, you can check out this GitHub repo.
I'd think that it can be done, but gluing everything tohether is not going to be for the faint of heart. Reason why I think it can be done is that codeception, itself, is ci-ed on Travis. See https://travis-ci.org/Codeception/Codeception. I'd contact the people at codeception and ask for their thoughts.
Or you can take a peek at how they do it in the build logs, such as:
https://travis-ci.org/Codeception/Codeception/jobs/14432638
Looks like they're running headless with a downloaded standalone selenium server.
Travis-ci have some information on how to run GUI tests. In particular, they allow you to use a sauce labs account and run distributed selenium tests from there.
I ran into this problem today and I solved it by adding Codeception to my composer.json:
"require-dev": {
"codeception/codeception": "^2.1"
},
and referring to it on my .travis.yml:
install:
- composer self-update
- composer install
before_script:
- #Code that creates and seeds my database and so on
script: php vendor/codeception/codeception/codecept run