Selenium tests works in local but not in Gitlab CI - selenium

I am currently working on a Java Spring Boot project which involves a classic backend/frontend architecture. I am trying to write some basic integration test by using the Selenium WebDriver.
The problem is that the tests I write pass without any problem on my local development machine but do not pass when I run them thorugh the continuous integration setup (Gitlab CI).
The code of the example test is the following:
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
#ActiveProfiles("test")
public class ExampleTest {
#LocalServerPort
private int port;
WebDriver wd;
#Test
public void successfulLogin(){
String url = "http://localhost:" + port;
wd = new HtmlUnitDriver();
wd.manage().timeouts().pageLoadTimeout(30, TimeUnit.SECONDS);
wd.get(url);
}
}
The relative gitlab-ci.yml portion is:
stages:
- test
maven-test:
image: maven:3.5.0-jdk-8
stage: test
script: "mvn test -B -Dmaven.repo.local=/root/.m2/"
only:
- master
The CI has a single runner (version 9.5.0) with concurrency 1. It uses the docker executor with the docker:stable image. I don't know if it is needed, but it is running in priviledged mode.
When running the tests on the CI environment, they fail with the following error:
org.openqa.selenium.TimeoutException: java.net.SocketTimeoutException: Read timed out
I have tried both with url = "http://localhost:" + port and url = InetAddress.getLocalHost().getHostName() + ":" + port, both passed locally, none passed in the CI environment.
What am I doing wrong?
EDIT: Alfageme suggested a more accurate testing methodology. On the same server on which the CI is running, I cloned my repository with git clone and then run the following command:
sudo gitlab-runner exec docker maven-test
The test passed without any problem. I am really running out of ideas, does someone have any?

I am not exactly sure why, but clearing the various runner-XXXXX-project-21-concurrent-0-cache-XXXXXXXXXXXXXXXXXXX docker containers on the CI machine seemed to have solved the issue.
EDIT: This fixed the issue only momentarely. The problem happened again, this time clearing the cache voulmes did not help. Anyone has any suggestion?

Related

Selenium navigate().to() stuck without error

I am running tests using TestNG, chromium and Selenium, in Java, on two machines:
my own laptop
a Mac Mini I do not have physical access to, that I connect via SSH, that has the same identical project structure as the project on my laptop.
Both machines operate behind the same corporate proxy.
The issue I am having is the following:
while the tests run smoothly on my machine, when I execute the command to start them on the Mac Mini, the execution stops when telling the Selenium WebDriver to navigate to any specified url using driver.navigate().to(url) or driver.get(url); causing the program to freeze without throwing any exception, essentially remaining on hold.
The code resembles the following:
in the #BeforeClass beforeClass() driver is initialized with an instance of ChromeDriver using ChromeOptions. Among other settings, I add the following arguments to the ChromeOptions like such chromeOptions.addArguments(options): where options is things like --disable-web-security, --allow-running-insecure-content, --allow-insecure-localhost and most importantly: --proxy-server=address:port (e.g. --proxy-server=172.26.44.146:3128).
in the #BeforeMethod beforeMethod() after checking that the driver is not null and that the URL url is a valid one, I execute something similar to the stylized following code:
try {
log.debug("Navigating to: " + url);
driver.navigate().to(url);
log.debug("Navigated to url: " + driver.getCurrentUrl());
} catch (Exception e) {
log.error("Unable to navigate at page: " + url.toString());
throw e;
}
As I said, when running the test on the Mac Mini via SSH using the command line in a similar fashion to: java -cp WORKSPACE org.testng.TestNG testng.xml the code stops executing and freezes at log.debug("Navigating to: " + url); essentially not producing other output and forcing me to manually stop the execution of the test.
As additional information, tests are run using chromium version 72.0.3609.0.
The ChromeOptions also receive the following proxy setting (like such: chromeOptions.setProxy(p);) where p is a Proxy with
p.setProxyType(ProxyType.MANUAL);
p.setAutodetect(false);
I suspect an issue with the proxy (and am doubtful I have correctly preconfigured the program), but, quite frankly, I am puzzled by the fact that the same script runs locally on my machine and is unable to run locally on the Mac Mini when the tests are launched via SSH.

How to set up working automated browser tests on Travis CI?

I am trying to set up automated tests for my web app on Travis CI. I am talking about automated browser tests - running unit tests is simple, but I need a process/system test that will start a headless browser and perform scripted tests of various use cases.
My application is in PHP so I decided to go with PHPUnit, Selenium and headless Firefox.
After a lot of research and trial and error I ended with following .travis.yml file:
language: php
php:
- '7.1'
services:
- mysql
addons:
firefox: latest
env:
- MOZ_HEADLESS=1
DISPLAY=:99.0
SELENIUM_FIREFOX_DRIVER=/home/travis/build/lotcz/zSample/geckodriver
before_install:
- sudo apt-get update > /dev/null
- wget https://selenium/download/url -O selenium-server.jar
- wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux32.tar.gz
- tar -xzf geckodriver-v0.21.0-linux32.tar.gz
install:
- sudo apt-get install apache2
- sudo service apache2 start
- mysql -e 'CREATE DATABASE IF NOT EXISTS zsample;'
before_script:
- nohup java -jar -Dwebdriver.gecko.driver=$SELENIUM_FIREFOX_DRIVER selenium-server.jar &
- composer install
script:
- phpunit --fail-on-risky --fail-on-warning --stop-on-skipped --stop-on-incomplete --verbose --debug --colors
after_failure:
- cat nohup.out
I edited out some pieces specific to my application. Just believe me that I set my application correctly before running the test.
Now a very simple test may look something like this:
class VisitorLoginTest extends PHPUnit_Extensions_Selenium2TestCase {
public function setUp() {
$this->setHost('localhost');
$this->setPort(4444);
$this->setBrowserUrl('http://localhost');
$this->setBrowser('firefox');
}
public function tearDown() {
$this->stop();
}
public function testFrontPage() {
$this->url('/');
$content = $this->byClass('main-title')->text();
$this->assertEquals('Hello', $content);
}
}
When my test is run I get this:
The Selenium Server is not active on host localhost at port 4444.
OK, but incomplete, skipped, or risky tests!
Tests: 1, Assertions: 0, Skipped: 1.
The command "make test" exited with 0.
Now my problems are these:
this file seems to be really long to me and too complicated given that I want to do something rather standard (I assume that automated browser tests are common these days). I would expect Travis CI to provide an easier way to perform automated browser tests. Here I have to download, install and start Apache, Selenium driver, Selenium server, use composer to get PHPUnit Selenium plugin etc...
Selenium server doesn't seem to be running and I can't find out why.
In the end, because PHPUnit ingeniously return 0 even when it couldn't even run the tests, Travis reports this test as successful. All those flags like --fail-on-risky or --stop-on-skipped still don't force PHPUnit to report a test failure which is in my opinion the only logical result as test clearly failed.
I know this is too broad and contains multiple questions. I am afraid that I took wrong direction somewhere and I am probably trying to do something simple in a complicated way.
Can somebody provide working example of .travis.yml file for automated browser tests? My application is in PHP, but I can write tests in Node.js, Python, Java or anything else as long as tests will really work and failure will be reported if anything goes wrong.
To Execute the above tests, Selenium server should be up and running on localhost with port 4444. Since the server is not up and running on the machine where execution happens try removing below mentioned lines and try.
$this->setHost('localhost');
$this->setPort(4444);

Running Jenkins tests in Docker containers build from dockerfile in codebase

I want to deploy a continuous integration platform based on Jenkins. As I have various kinds of projects (PHP / Symfony, node, angular, …) and as I want these tests to run both locally and on Jenkins, I was thinking about using Dockers containers.
The process I’m aiming for is :
A merge request is opened on Github / Gitlab
A webhook notifies Jenkins of the merge request
Jenkins pulls the repo, builds the containers and runs a shell script to execute the tests
Once the tests are finished, Jenkins retrieves the results from one of the containers (through a shared volume) and process the results.
I do not want Jenkins to be in a container.
With this kind of process, I’m hoping to be able to run very easily the tests on each developer machine with something like a docker-composer up and then in one of the container ./tests all.
I’m not very familiar with Jenkins. I’ve read a lot of documentation, but most of them suggested to define Jenkins slaves for each kind of projects beforehand. I would like everything to be as dynamic as possible and require as less configuration on Jenkins as possible.
I would appreciate a description of your test process if you have ever implemented something similar. If you think what I’m aiming for is impossible, I would also appreciate if you could explain to me why.
A setup I suggest is Docker in Docker.
The base is a derived Docker image, which extends the jenkins:2.x image by adding a Docker commandline client.
The Jenkins is started as a container with its home folder (a folder e.g. /var/jenkins_home mounted from the Docker host) and the Docker socket file to be able to start Docker containers from Jenkins build jobs.
docker run -d --name jenkins -v /var/jenkins_home:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock ... <yourDerivedJenkinsImage>
To check, if this setup is working just execute following command after starting the Jenkins container:
docker exec jenkins docker version
If the "docker version" output does NOT show:
Is the docker daemon running on this host?
Everythin is fine.
In your build jobs, you could configure the process you mentioned above. Let Jenkins simply check out the repository. The repository should contain your build and test scripts.
Use a freestyle build job with a shell execution. A shell execution could look like this:
docker run --rm --volumes-from jenkins <yourImageToBuildAndTestTheProject> bash $WORKSPACE/<pathToYourProjectWithinTheGitRepository>/build.sh
This command simply starts a new container (to build and/or test your project) with the volumes from jenkins. Which means that the cloned repository will be available under $WORKSPACE. So if you run "bash $WORKSPACE/<pathToYourProjectWithinTheGitRepository>/build.sh" your project will be built within a container of "yourImageToBuildAndTestTheProject". After running this, you could start other containers for integration tests or combine this with "docker-compose" by installing it on the derived Jenkins image.
Advantages are the minimal configuration affort you have within Jenkins - only the SCM configuration for cloning the GIT repository is required. Since each Jenkins job uses the Docker client directly you could use for each project one or Docker image to build and/or test, WITHOUT further Jenkins configuration.
If you need additional configuration e.g. SSH keys or Maven settings, just put them on the Docker host and start the Jenkins container with the additional volumes, which contain those configuration files.
Using this Docker option within the shell execution of your build jobs:
--volumes-from jenkins
Automatically adds workspace and configuration files to each of your build jobs.

teamcity with grunt protractor

After tons of troubleshooting I finally got selenium to run but it just starts the server and sits on it. Anybody have any idea how to make it run the tests and exit?
[Step 9/9] Executing /home/ubuntu/buildAgent/work/d653d615082dd19f/node_modules/.bin/grunt via wrapping shell script
[16:47:57][Step 9/9] Starting: /home/ubuntu/buildAgent/temp/agentTmp/wrapper3464043714269626324.sh --no-color --teamcity.properties.all=/home/ubuntu/buildAgent/temp/agentTmp/teamcity7605733587545998535.json --teamcity.properties=/home/ubuntu/buildAgent/temp/agentTmp/teamcity414610972945921232.json protractor
[16:47:57][Step 9/9] in directory: /home/ubuntu/buildAgent/work/d653d615082dd19f
[16:48:01][Step 9/9] Running "protractor:myApp" (protractor) task
[16:48:01][Step 9/9] Starting selenium standalone server...
[16:48:01][Step 9/9] [launcher] Running 1 instances of WebDriver
[16:48:02][Step 9/9] Selenium standalone server started at http://172.30.0.51:40590/wd/hub
Selenium is a service, which "starts and runs" :) This is the purpose of Selenium as well as any other service.
If you want to run Protractor tests which runs their own Selenium service, then you should just set Selenium jar and do not set the Selenium address.
export.config = {
// Excerpt of protractor.conf.js
seleniumServerJar: 'node_modules/protractor/selenium/selenium-server-standalone-2.45.0.jar',
// Comment out seleniumAddress
// seleniumAddress: 'http://localhost:4444'
};
FYI People. I had the same use case and I needed the selenium process to close after all the tests were executed.
I wrote a simple power shell script which is what I ended up using later.
# Check if chrome driver is already running.
$java = Get-Process java*
Write-Host $java
if ($java) {
$java | Stop-Process -Force
}
This is obviously not the best script I could write but then it did the job. You can use process Explorer to find that selenium runs on the Java process thread and hence, it is the one you have to kill.
This tiny script is now a part of the build process.

Can Travis-CI run Codeception tests?

I'm creating my tests (though I'm a beginner, learning) using Codeception. This includes acceptance and unit tests for now.
I want to add my repo to Travis CI so I can automate testing process after each commit and put build-status tag.
I would like to ask;
Can Travis-CI run codeception tests?
Can Travis-CI run codeception acceptance tests emulating browser?
If both answers are no, is there any other CI tool which can?
Thank you.
Yes, it is possible to run Codeception tests, including acceptance tests that run using WebDriver, on Travis CI.
It is possible to run your tests with a real browser on Travis, but it is easiest to use a headless browser, since Travis is running on a headless machine. PhantomJS is perfect for this, and it comes pre-installed with Travis CI's build bootstrap.
To run the tests with PhantomJS, you'll need to configure the WebDriver module like this in your .yml Codeception configuration file:
modules:
config:
WPWebDriver:
url: 'http://127.0.0.1:8888'
browser: phantomjs
The URL is important. I have found that attempting to use localhost instead of 127.0.0.1 will not work. Also, if you accidentally leave out the http://, that won't work either. You can use most any 8*** port, since most of them are open, but of course you'll need to have a web server running on that port to serve your static files or run your PHP application. The easiest way to do this, I find, is to use PHP's built-in webserver.
Your .travis.yml file might look something like this:
# Travis CI configuration file.
language: php
php:
- 5.6
- 7.0
before_script:
# Start up a web server.
- php -S 127.0.0.1:8888 -t /path/to/web/root >/dev/null 2>&1 &
# Start up the webdriver.
- phantomjs --webdriver=4444 >/dev/null 2>&1 &
# Install Codeception.
# Doing this last gives the webdriver and server time to start up.
- composer install --prefer-source
script:
- vendor/bin/codecept run
You will of course need to add Codeception to your project's composer.json file:
composer require --dev codeception/codeception
You'll also need to change path/to/web/root above to the path to the directory where you want the server's document root to be.
If you'd like to see a working demo running WebDriver tests against WordPress, you can check out this GitHub repo.
I'd think that it can be done, but gluing everything tohether is not going to be for the faint of heart. Reason why I think it can be done is that codeception, itself, is ci-ed on Travis. See https://travis-ci.org/Codeception/Codeception. I'd contact the people at codeception and ask for their thoughts.
Or you can take a peek at how they do it in the build logs, such as:
https://travis-ci.org/Codeception/Codeception/jobs/14432638
Looks like they're running headless with a downloaded standalone selenium server.
Travis-ci have some information on how to run GUI tests. In particular, they allow you to use a sauce labs account and run distributed selenium tests from there.
I ran into this problem today and I solved it by adding Codeception to my composer.json:
"require-dev": {
"codeception/codeception": "^2.1"
},
and referring to it on my .travis.yml:
install:
- composer self-update
- composer install
before_script:
- #Code that creates and seeds my database and so on
script: php vendor/codeception/codeception/codecept run