How to set up working automated browser tests on Travis CI? - selenium

I am trying to set up automated tests for my web app on Travis CI. I am talking about automated browser tests - running unit tests is simple, but I need a process/system test that will start a headless browser and perform scripted tests of various use cases.
My application is in PHP so I decided to go with PHPUnit, Selenium and headless Firefox.
After a lot of research and trial and error I ended with following .travis.yml file:
language: php
php:
- '7.1'
services:
- mysql
addons:
firefox: latest
env:
- MOZ_HEADLESS=1
DISPLAY=:99.0
SELENIUM_FIREFOX_DRIVER=/home/travis/build/lotcz/zSample/geckodriver
before_install:
- sudo apt-get update > /dev/null
- wget https://selenium/download/url -O selenium-server.jar
- wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux32.tar.gz
- tar -xzf geckodriver-v0.21.0-linux32.tar.gz
install:
- sudo apt-get install apache2
- sudo service apache2 start
- mysql -e 'CREATE DATABASE IF NOT EXISTS zsample;'
before_script:
- nohup java -jar -Dwebdriver.gecko.driver=$SELENIUM_FIREFOX_DRIVER selenium-server.jar &
- composer install
script:
- phpunit --fail-on-risky --fail-on-warning --stop-on-skipped --stop-on-incomplete --verbose --debug --colors
after_failure:
- cat nohup.out
I edited out some pieces specific to my application. Just believe me that I set my application correctly before running the test.
Now a very simple test may look something like this:
class VisitorLoginTest extends PHPUnit_Extensions_Selenium2TestCase {
public function setUp() {
$this->setHost('localhost');
$this->setPort(4444);
$this->setBrowserUrl('http://localhost');
$this->setBrowser('firefox');
}
public function tearDown() {
$this->stop();
}
public function testFrontPage() {
$this->url('/');
$content = $this->byClass('main-title')->text();
$this->assertEquals('Hello', $content);
}
}
When my test is run I get this:
The Selenium Server is not active on host localhost at port 4444.
OK, but incomplete, skipped, or risky tests!
Tests: 1, Assertions: 0, Skipped: 1.
The command "make test" exited with 0.
Now my problems are these:
this file seems to be really long to me and too complicated given that I want to do something rather standard (I assume that automated browser tests are common these days). I would expect Travis CI to provide an easier way to perform automated browser tests. Here I have to download, install and start Apache, Selenium driver, Selenium server, use composer to get PHPUnit Selenium plugin etc...
Selenium server doesn't seem to be running and I can't find out why.
In the end, because PHPUnit ingeniously return 0 even when it couldn't even run the tests, Travis reports this test as successful. All those flags like --fail-on-risky or --stop-on-skipped still don't force PHPUnit to report a test failure which is in my opinion the only logical result as test clearly failed.
I know this is too broad and contains multiple questions. I am afraid that I took wrong direction somewhere and I am probably trying to do something simple in a complicated way.
Can somebody provide working example of .travis.yml file for automated browser tests? My application is in PHP, but I can write tests in Node.js, Python, Java or anything else as long as tests will really work and failure will be reported if anything goes wrong.

To Execute the above tests, Selenium server should be up and running on localhost with port 4444. Since the server is not up and running on the machine where execution happens try removing below mentioned lines and try.
$this->setHost('localhost');
$this->setPort(4444);

Related

Selenium tests works in local but not in Gitlab CI

I am currently working on a Java Spring Boot project which involves a classic backend/frontend architecture. I am trying to write some basic integration test by using the Selenium WebDriver.
The problem is that the tests I write pass without any problem on my local development machine but do not pass when I run them thorugh the continuous integration setup (Gitlab CI).
The code of the example test is the following:
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
#ActiveProfiles("test")
public class ExampleTest {
#LocalServerPort
private int port;
WebDriver wd;
#Test
public void successfulLogin(){
String url = "http://localhost:" + port;
wd = new HtmlUnitDriver();
wd.manage().timeouts().pageLoadTimeout(30, TimeUnit.SECONDS);
wd.get(url);
}
}
The relative gitlab-ci.yml portion is:
stages:
- test
maven-test:
image: maven:3.5.0-jdk-8
stage: test
script: "mvn test -B -Dmaven.repo.local=/root/.m2/"
only:
- master
The CI has a single runner (version 9.5.0) with concurrency 1. It uses the docker executor with the docker:stable image. I don't know if it is needed, but it is running in priviledged mode.
When running the tests on the CI environment, they fail with the following error:
org.openqa.selenium.TimeoutException: java.net.SocketTimeoutException: Read timed out
I have tried both with url = "http://localhost:" + port and url = InetAddress.getLocalHost().getHostName() + ":" + port, both passed locally, none passed in the CI environment.
What am I doing wrong?
EDIT: Alfageme suggested a more accurate testing methodology. On the same server on which the CI is running, I cloned my repository with git clone and then run the following command:
sudo gitlab-runner exec docker maven-test
The test passed without any problem. I am really running out of ideas, does someone have any?
I am not exactly sure why, but clearing the various runner-XXXXX-project-21-concurrent-0-cache-XXXXXXXXXXXXXXXXXXX docker containers on the CI machine seemed to have solved the issue.
EDIT: This fixed the issue only momentarely. The problem happened again, this time clearing the cache voulmes did not help. Anyone has any suggestion?

teamcity with grunt protractor

After tons of troubleshooting I finally got selenium to run but it just starts the server and sits on it. Anybody have any idea how to make it run the tests and exit?
[Step 9/9] Executing /home/ubuntu/buildAgent/work/d653d615082dd19f/node_modules/.bin/grunt via wrapping shell script
[16:47:57][Step 9/9] Starting: /home/ubuntu/buildAgent/temp/agentTmp/wrapper3464043714269626324.sh --no-color --teamcity.properties.all=/home/ubuntu/buildAgent/temp/agentTmp/teamcity7605733587545998535.json --teamcity.properties=/home/ubuntu/buildAgent/temp/agentTmp/teamcity414610972945921232.json protractor
[16:47:57][Step 9/9] in directory: /home/ubuntu/buildAgent/work/d653d615082dd19f
[16:48:01][Step 9/9] Running "protractor:myApp" (protractor) task
[16:48:01][Step 9/9] Starting selenium standalone server...
[16:48:01][Step 9/9] [launcher] Running 1 instances of WebDriver
[16:48:02][Step 9/9] Selenium standalone server started at http://172.30.0.51:40590/wd/hub
Selenium is a service, which "starts and runs" :) This is the purpose of Selenium as well as any other service.
If you want to run Protractor tests which runs their own Selenium service, then you should just set Selenium jar and do not set the Selenium address.
export.config = {
// Excerpt of protractor.conf.js
seleniumServerJar: 'node_modules/protractor/selenium/selenium-server-standalone-2.45.0.jar',
// Comment out seleniumAddress
// seleniumAddress: 'http://localhost:4444'
};
FYI People. I had the same use case and I needed the selenium process to close after all the tests were executed.
I wrote a simple power shell script which is what I ended up using later.
# Check if chrome driver is already running.
$java = Get-Process java*
Write-Host $java
if ($java) {
$java | Stop-Process -Force
}
This is obviously not the best script I could write but then it did the job. You can use process Explorer to find that selenium runs on the Java process thread and hence, it is the one you have to kill.
This tiny script is now a part of the build process.

Issue running Selenium IDE test suite using Selenium Standalone server

I am try to follow following references to run the selenium IDE test suites on standalone server . My ultimate purpose is to run headless data-driven tests on ubuntu.
Selenium RC with DataDriven
Running Tests from command line
Selenium Headless tests on Ubuntu
So after looking into these tutorials I have setup very simple test suite and trying to run from command line e.g
java -jar ~/selenium/selenium-server-standalone-2.39.0.jar -userExtensions user-extensions.js -htmlSuite "*firefox" http://google.com ./suite.html ./out/firefox-results.html
Output
org.openqa.grid.selenium.GridLauncher main
INFO: Launching a standalone server
10:16:54.164 INFO - Java: Apple Inc. 20.65-b04-462
10:16:54.164 INFO - OS: Mac OS X 10.6.8 x86_64
10:16:54.178 INFO - v2.39.0, with Core v2.39.0. Built from revision ff23eac
10:16:54.250 INFO - Default driver org.openqa.selenium.ie.InternetExplorerDriver registration is skipped: registration capabilities Capabilities [{platform=WINDOWS, ensureCleanSession=true, browserName=internet explorer, version=}] does not match with current platform: MAC
10:16:54.336 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4444/wd/hub
10:16:54.338 INFO - Version Jetty/5.1.x
10:16:54.339 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
10:16:54.340 INFO - Started HttpContext[/selenium-server,/selenium-server]
10:16:54.340 INFO - Started HttpContext[/,/]
10:16:54.406 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler#5ff06dc3
10:16:54.407 INFO - Started HttpContext[/wd,/wd]
10:16:54.413 INFO - Started SocketListener on 0.0.0.0:4444
10:16:54.414 INFO - Started org.openqa.jetty.jetty.Server#4ca31e1b
jar:file:/Users/myname/selenium/selenium-server-standalone-2.39.0.jar!/customProfileDirCUSTFFCHROME
10:16:54.580 INFO - Preparing Firefox profile...
10:16:55.946 INFO - Launching Firefox...
10:16:57.246 INFO - Checking Resource aliases
It does open two firefox windows and hangs there after INFO - Checking Resource aliases
It does not matter If I skip -userExtensions user-extensions.js,it gives same output , show two popup windows and hangs there. The firefox popup window shows test suite loaded successfully , but not command is executed further.
I wonder if I need to run/configure sth else
Example Test Suite and Test Case:
suite.html
case_a.html
Command
java -jar ~/selenium/selenium-server-standalone-2.39.0.jar -htmlSuite "*firefox" http://www.google.com suite.html firefox-results.html
Tested Under:
A) OS: Mac OS X 10.6.8 x86_64 /java version "1.6.0_65"
B) OS: Windows 7 / java version : 1.7.0_02
Hacks Tried without any luck:
A) Removing xml tag
B) .html file extension
Since this question got much attention and needed a proper solution, which I found this way:
First of all we need proper addons to setup headless tests
Getting the right addons
I am already using following selenium IDE addons
datadriven v0.2
flowcontrol v08
includecommand 1.3
What I need is respective selenium addons which work with selenium core server for which I need
datadriven_v0.2-core.js
flowcontrol ( goto_sel08.js)
includecommand ( includeCommand_2.3.js) origional source
since openqa.org does not maintain these addons anymore (probably broken or so) You have to use wayback machine to get js files ( no luck with zip attachments). e g flowcontrol in wayback machine
Since I have done this already and created a gist with some fixes
datadriven_v0.2-core.js
flowcontrol ( goto_sel08.js)
includeCommand_2.3.js
user-extension.js ( all three of above combined)
Setting up Headless Tests
a) install xvfb ( X Vritual Frame Buffer and firefox )
sudo apt-get update && sudo apt-get install -y xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic xvfb x11-apps firefox
b) create xvfb init script /var/init.d/xvfb
if [ -z "$1" ]; then
echo "`basename $0` {start|stop}"
exit
fi
case "$1" in
start)
/usr/bin/Xvfb :99 -ac -screen 0 1024x768x8 &
;;
stop)
killall Xvfb
;;
esac
c) add to startup script:
sudo update-rc.d xvfb defaults
sudo chmod +x /etc/init.d/xvfb
d) start the xvfb
/etc/init.d/xvfb start
e) Setup selenium Tests (you can put test anywhere)
sudo mkdir /usr/local/SeleniumTests && cd /usr/local/SeleniumTests
checkout/copy your html tests to /usr/local/SeleniumTests
f) create alias to test directory (config file 'selenium_alias')
(only allow from localhost)
<IfModule alias_module>
Alias /SeleniumTests/ /usr/local/SeleniumTests/
</IfModule>
<Directory "/usr/local/SeleniumTests/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Directory>
sudo mv SeleniumTests/selenium_alias /etc/apache2/sites-avaiable
sudo a2ensite selenium
sudo service apache2 restart
g) update selenium tests data
(/usr/local/SeleniumTests/tests/data)
e.g change test username/password and baseUrl to test app (which I have in xml file)
h) Download Seleninum Server jar file
sudo mkdir /var/lib/selenium/
sudo wget http://selenium.googlecode.com/files/selenium-server-standalone-2.39.0.jar -o /var/lib/selenium/selenium-server.jar
i) Run the headless /Selenese Tests
export DISPLAY=":99" && java -jar /var/lib/selenium/selenium-server.jar -htmlSuite *firefox https://localhost "/usr/local/SeleniumTests/tests/my_app_smoke_testing_suite.html" "firefox-results.html" -trustAllSSLCertificates -ensureCleanSession -port 5561 -userExtensions /usr/local/SeleniumTests/addons/user-extensions.js
Note I have my selenium addons setup under /usr/local/SeleniumTests/addons
Some of the the steps might not be desirable in all situations.
Try this:
<Selenium server location path>java -jar <selenium-server name>.jar –htmlSuite "*<browser_name>" "<Url of base website>" "<Path of html_suite>" "<Path to store reports>"
selenium-server name :- selenium-server-standalone-2.xx.xx was used.
browser_name :- "iexplore" for Internet Explorer/ "firefox" for
Mozilla Firefox
Url of base website :- Base URL.
Path of html_suite :- The path of test suite you used to save file
(E.g.:"C:\TestSuite.htm", assuming "C" drive as location you used
to save test suite).
Path to store reports :-The path you want to save your test result
into (E.g.:"C:\TestResult.htm", assuming "C" drive as location you
used to save test result).
Some commands are not working in Firefox:
The user extension goto_sel_ide that defines the While and Goto
commands does not work with Selenium RC. Include the user extension
goto_sel08.js instead.
From: https://wiki.mozilla.org/Running_IDE_scripts_with_Selenium_RC
I think you are following these steps from this page:
Download goto_sel08.js
Download the 1.3 version of the includeCommand4IDE extension.
Download the most recent version (0.2) of the datadriven.js extension.
Merged these files to new user-extensions.js file. This order is critical!
create an xml file which must contain all the data–input and output–for a single test case.
Utilize your .xml file
Use Selenium-IDE or an editor to create an HTML test suite that includes your data-driven test as one of the tests to be executed.
Execute a command line:
-jar selenium-server.jar
-userExtensions user-extensions.js
-htmlSuite
"*chrome"
"`<base URL>`"
"`<Selenium test suite file>`"
"`<results log file>`"
-timeout nnnn
I don't post all steps. I think these steps are important for create data-driven testing from a Selenium-RC command line. Please check all steps!
Perhaps its working on Ubuntu OS, if its works on Windows. Two popup window is weird for me, but may be not is problem. I can only offer check these points:
create suite.html with firefox selenium IDE
try to use *chrome in command line.
Please check order of scripts in user-extensions.js file.
I hope it helps!

Can Travis-CI run Codeception tests?

I'm creating my tests (though I'm a beginner, learning) using Codeception. This includes acceptance and unit tests for now.
I want to add my repo to Travis CI so I can automate testing process after each commit and put build-status tag.
I would like to ask;
Can Travis-CI run codeception tests?
Can Travis-CI run codeception acceptance tests emulating browser?
If both answers are no, is there any other CI tool which can?
Thank you.
Yes, it is possible to run Codeception tests, including acceptance tests that run using WebDriver, on Travis CI.
It is possible to run your tests with a real browser on Travis, but it is easiest to use a headless browser, since Travis is running on a headless machine. PhantomJS is perfect for this, and it comes pre-installed with Travis CI's build bootstrap.
To run the tests with PhantomJS, you'll need to configure the WebDriver module like this in your .yml Codeception configuration file:
modules:
config:
WPWebDriver:
url: 'http://127.0.0.1:8888'
browser: phantomjs
The URL is important. I have found that attempting to use localhost instead of 127.0.0.1 will not work. Also, if you accidentally leave out the http://, that won't work either. You can use most any 8*** port, since most of them are open, but of course you'll need to have a web server running on that port to serve your static files or run your PHP application. The easiest way to do this, I find, is to use PHP's built-in webserver.
Your .travis.yml file might look something like this:
# Travis CI configuration file.
language: php
php:
- 5.6
- 7.0
before_script:
# Start up a web server.
- php -S 127.0.0.1:8888 -t /path/to/web/root >/dev/null 2>&1 &
# Start up the webdriver.
- phantomjs --webdriver=4444 >/dev/null 2>&1 &
# Install Codeception.
# Doing this last gives the webdriver and server time to start up.
- composer install --prefer-source
script:
- vendor/bin/codecept run
You will of course need to add Codeception to your project's composer.json file:
composer require --dev codeception/codeception
You'll also need to change path/to/web/root above to the path to the directory where you want the server's document root to be.
If you'd like to see a working demo running WebDriver tests against WordPress, you can check out this GitHub repo.
I'd think that it can be done, but gluing everything tohether is not going to be for the faint of heart. Reason why I think it can be done is that codeception, itself, is ci-ed on Travis. See https://travis-ci.org/Codeception/Codeception. I'd contact the people at codeception and ask for their thoughts.
Or you can take a peek at how they do it in the build logs, such as:
https://travis-ci.org/Codeception/Codeception/jobs/14432638
Looks like they're running headless with a downloaded standalone selenium server.
Travis-ci have some information on how to run GUI tests. In particular, they allow you to use a sauce labs account and run distributed selenium tests from there.
I ran into this problem today and I solved it by adding Codeception to my composer.json:
"require-dev": {
"codeception/codeception": "^2.1"
},
and referring to it on my .travis.yml:
install:
- composer self-update
- composer install
before_script:
- #Code that creates and seeds my database and so on
script: php vendor/codeception/codeception/codecept run

Why is selenium hanging on INFO - Checking Resource aliases, and how do I even debug this?

I'm trying to follow the tutorial here to setup a headless selenium test-run with jenkins. I'm running CentOS 5.6, and I've followed the instructions. Now, when I run this:
export DISPLAY=":99" && java -jar /var/lib/selenium/selenium-server.jar -browserSessionReuse -htmlSuite *firefox http://www.google.com ./test/selenium/html/TestSuite.html ./target/selenium/html/TestSuiteResults.html
Selenium hangs on INFO - Checking Resource Aliases. I can run the TestSuite.html file manually, and the path is correct.
How can I even begin to try and figure out what's going on? Is there a way I could connect to the display to see what's happening? I am behind a corporate proxy, but with or without -Dhttp.proxyHost arguments, I get the same hung result.
Well, after pointing at an internal server, I get right on past the INFO - Checking Resource Aliases step, so clearly the proxy was the issue.
By trying to hit a site that required the proxy, I was doing too much at once. Confounding variables confounded me.
Selenium is not hanging on INFO - Checking Resource Aliases. Its waiting for a command to execute. You need to trigger your tests using ANT or some other build tool in Jenkins. That should get you going