I have a configuration and tests written that don't work when I pass them through Selenium grid (but work without).
Selenium Tests / JUnit / Eclipse
One Jenkins job that clones the repository and runs the tests
Selenium Grid plugin installed on Jenkins, and all the nodes (Jenkins slaves on Linux with FF) connected to the hub.
Question 1: The Jenkins job for my test suite starts on a slave. Is this correct? Should I start it on master (I tried and got other issues, but at least I would focus on the real problems if I knew what is the correct way to go).
Question 2. I initialize the driver in #Before test. Is this correct?
driver = new RemoteWebDriver(new URL("http://......"), DesiredCapabilities.firefox());
This is where I initialize the firefox driver when not using grid, and it is working.
Question 3. How do I tell Selenium grid to offer me some logs. In all fairness, I have no idea if my tests even attempt to visit the hub.
The last question is regarding the error that I get only when using grid (again, all my tests work fine without).
org.openqa.selenium.WebDriverException: Specified firefox binary location does not exist or is not a real file: /usr/bin/firefox
There are quite a few solutions for the last error that I have googled and tried, without any luck. Being a beginner, I now doubt my configuration.
Thanks for your time.
Ann
Answers:
Question 1
It's not incorrect to run it on a slave. That's perfectly acceptable.
Question 2
You can initialize the driver wherever you want. You can put it in a #Before, a constructor, or even the method. It's all dependent on what works best for you.
Question 3
i'm not sure if you can do this with firefox, but with the chromedriver you can pass arguments like this: --verbose --log-path=/tmp/chromedriver.log \$*
Last Question
I will answer this question by first asking you a question. your grid and nodes are different servers than your jenkins server? If so, then this most likely means that you need to make sure that your firefox executables are under /usr/bin/firefox. If they are, then make sure that they are executable! e.g.: sudo chmod u+x /usr/bin/firefox
Also, you have your RemoteWebDriver initialization masked so i can't see, but make sure that you have the url of your grid, with /wd/hub concat'ed at the end. e.g.: http://selenium-grid:4444/wd/hub
Related
When using the APIs defined by Protractor & Jasmine (the default/supported runner for Protractor), the tests will always work okay on individual developer laptops. For some reason when the test runs on the Jenkins CI server, they will fail (despite being in the same docker containers on both hosts, and that was wildly frustrating.)
This error occurs: A Jasmine spec timed out. Resetting the WebDriver Control Flow.
This error also appears: Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
Setting getPageTimeout & allScriptsTimeout to 30 seconds had no effect on this.
I tried changing jasmine.DEFAULT_TIMEOUT_INTERVAL to 60 seconds for all tests in this suite, once the first error appears then every test will wait the full 60 seconds and time out.
I've read and reread Protractor's page on timeouts but none of that seems relevant to this situation.
Even stranger still, it seems like some kind of buffer issue - at first the tests would always fail on a particular spec, and nothing about that spec looked wrong. While debugging I upgraded the selenium docker container from 2.53.1-beryllium to 3.4.0-einsteinium and the tests still failed but they failed a couple specs down - suggesting that maybe there was some optimization in the update and so it was able to get more done before it gave out.
I confirmed that by rearranging the order of the specs - the specs that had failed consistently before were now passing and a test that previously passed began to fail (but around the same time in the test duration as the other failures before the reorder.)
Environment:
protractor - 5.1.2
selenium/standalone-chrome-debug - 3.4.0-einsteinium
docker - 1.12.5
The solution ended up being simple - I first found it on a chrome bug report, and it turned out it was also listed right on the front page of the docker-selenium repo but the text wasn't clear as to what it was for when I'd read it the first time. (It says that selenium will crash without it, but the errors I was getting from Jasmine were just talking about timeouts, and that was quite misleading.)
Chrome apparently utilizes /dev/shm, and apparently that's fairly small in docker. There are workarounds for chrome and firefox linked from their README that explain how to resolve the issue.
I had a couple test suites fail after applying the fix but all the test suites have been running and passing for the last day, so I think that was actually the problem and that this solution works. Hope this helps!
I am running my spec flow feature files with the help of resharper and certain feature files have been ignored for later use with #ignore tag. What happens is that chromedriver opens these tests, realizes that they are ignored and stop the test. As I run the tests from a folder, it takes upto 5 mins for every folder that contain these ignored tests. Is there a way to prevent the test runner,runnning these ignored tests so that I can save some time ?
That really depends on where you are starting chromedriver from. If you are starting the driver in a [BeforeScenario] step then yes. Its as simple as this:
if(!ScenarioContext.Current.ScenarioInfo.Tags.Contains("ignore")){
//start chromedriver
}
in this way you can check to make sure if the current scenario should be ignored or not and avoid starting chromedriver if it is.
I have a problem with Selenium and can't get it to work.
What I want to do:
A customer of ours sends us a Selenium test case which shall be executed automatically in several locations and the time taken shall be recorded.
We want to use Selenium and Firefox Portable, because we want to make the tests completely independent of any user input and the installed software at the different locations.
So much for the starting conditions ;)
What we did so far:
The first version was completely written in Java, we exported the test case from the customer to Java with the Selenium IDE Plugin -> Export to Java WebDriver.
This cannot be done anymore, because the customer now uses some functions the WebDriver export does not support. And as we don't want to alter the test from the customer, Java export is no longer an option.
So for the first run we are using this command (any variables are set correctly):
java -jar selenium-2.33.0/selenium-server-standalone-2.33.0.jar -port 5555
-firefoxProfileTemplate "Firefox\Data\profile" -log logs\selenium_server.log
-htmlSuite "*firefox" http://localhost:5555 Testsuite.html
logs\results-firefox-%curTimestamp%.html
This starts my preinstalled firefox, not the portable one. On the customers machine, no firefox is started whatsoever, because it is not installed. So I had to provide the path to the firefox instead, using the "custom" htmlSuite:
java -jar selenium-2.33.0/selenium-server-standalone-2.33.0.jar -port 5555
-firefoxProfileTemplate "Firefox\Data\profile" -log logs\selenium_server.log
-htmlSuite "*custom %FF_DIR%\FirefoxPortable.exe" http://localhost:5555 Testsuite.html
logs\results-firefox-%curTimestamp%.html
This does not work, as the Selenium Server cannot execute this command if run under Windows, which we do (see: http://code.google.com/p/selenium/issues/detail?id=3274)
As comment #6 has some diffs, we patched the selenium Server standalone Jar and ran the test again. Now the browser could be started, but the test could not be run. After the first page loaded we get the error "Permission denied to access property 'document'".
A solution here suggests, a user-rights problem could be the cause and you should try the "chrome" htmlSuite (see: https://sqa.stackexchange.com/questions/1453/how-to-fix-permission-denied-to-access-property-document)
So we did:
java -jar selenium-2.33.0/selenium-server-standalone-2.33.0-patched.jar
-port 5555 -firefoxProfileTemplate "FirefoxPortable\Data\profile"
-log logs\selenium_server.log -htmlSuite "*chrome %FF_DIR%\FirefoxPortable.exe"
http://localhost:5555 Testsuite.html logs\results-firefox-%curTimestamp%.html
Notice our "patched" selenium and the "chrome" htmlSuite.
That didn't work, as well.
So, here in short the results:
htmlSuite = firefox: the preinstalled Firefox is used, if installed, not the Portable one. In case, no FF is installed, the test fails altogether
htmlSuite = chrome: the server cannot start the browser, as it tries to set EnvironmentVariables, which is not supported running Windows (see: http://code.google.com/p/selenium/source/browse/java/client/src/org/openqa/selenium/os/WindowsProcessGroup.java#67 lines 67 following)
htmlSuite = googleChrome: Google Chrome Portable can be started, but the Chrome browser cannot find some elements specified by the test, so we cannot use Chrome (altering the test is no option, as stated above)
htmlSuite = iexplore: Internet Explorer starts, but then a JavaScript error appears, referencing a custom Profile created by Selenium, so the test does not work in IE, either
htmlSuite = custom: the Portable Firefox is started (yeehaw), but does not have sufficient rights to execute the test.
You may use a Continuous Integration System like Jenkins, or TeamCity to execute your tests automatically.
we have now decided to support the customer in installing Firefox on the machines to test, so we can use our batchfile without problems.
As for the bug in selenium look here (code.google.com/p/selenium/issues/detail?id=5554#c14), there is a link to a nightly build that does work (at least for us) with Firefox v23, which Selenium 2.33 does not.
Thanks for everyone who contributed, but I think my first approach can not be achieved the way I thought it could :(
I solved this problem.
Visit this link: http://www.townx.org/blog/elliot/dealing-self-signed-ssl-certificates-when-running-selenium-server-firefox
Point 9:
Delete everything in the directory except for the cert_override.txt and cert8.db files.
Hope it helps
I'm trying to follow the tutorial here to setup a headless selenium test-run with jenkins. I'm running CentOS 5.6, and I've followed the instructions. Now, when I run this:
export DISPLAY=":99" && java -jar /var/lib/selenium/selenium-server.jar -browserSessionReuse -htmlSuite *firefox http://www.google.com ./test/selenium/html/TestSuite.html ./target/selenium/html/TestSuiteResults.html
Selenium hangs on INFO - Checking Resource Aliases. I can run the TestSuite.html file manually, and the path is correct.
How can I even begin to try and figure out what's going on? Is there a way I could connect to the display to see what's happening? I am behind a corporate proxy, but with or without -Dhttp.proxyHost arguments, I get the same hung result.
Well, after pointing at an internal server, I get right on past the INFO - Checking Resource Aliases step, so clearly the proxy was the issue.
By trying to hit a site that required the proxy, I was doing too much at once. Confounding variables confounded me.
Selenium is not hanging on INFO - Checking Resource Aliases. Its waiting for a command to execute. You need to trigger your tests using ANT or some other build tool in Jenkins. That should get you going
I'm having a hard time trying to grasp some concepts on selenium Grid/RC. What I need is to provide specific environments (ie6-on-xp, ie7-on-xp, etc) to the tests. For what I've been reading, the browser line in grid_configuration.yml do not make any reference of what version of MSIE or Firefox I'm running. So I can't get my head around in which form I can tell Grid/RC that I want some specific browsers and the path to run them (how RC knows which exe to run?)
Second, I'd like to run portable versions of those browsers. I've only seen that specified in the tests, and not in the RC's command line to run them. That is the way to do it, per test?
I will answer your question by breaking up the info that you need
What I need is to provide specific
environments (ie6-on-xp, ie7-on-xp,
etc) to the tests.
Well since you can't have multiple IE instances on the same machine, I know there are apps that allow you to do that but in my experience they cause more issues than solving them. Ideally you want different machines to run the tests. By doing this you are also setting up a selenium farm for your devs to use because they can target a test at a specific instance. So setting up Grid as an Infrastructure is a good step.
For what I've been reading, the
browser line in grid_configuration.yml
do not make any reference of what
version of MSIE or Firefox I'm
running. So I can't get my head around
in which form I can tell Grid/RC that
I want some specific browsers and the
path to run them (how RC knows which
exe to run?)
The YAML just lets you know what the grid can handle. When starting up the grid you need to make sure that you pass in similar configurations. Think of Se:GRID like you would Se:RC except you don't care where the RC server is because you speak to a central place that works the rest out for you.
If you need it to run tests against a specific items then you will need to handle this in your test setup. There is a common misconception that all tests will run the same in every single browser. This will happen if you never rely on XPath or CSS selectors in your tests because browsers always handle this slightly differently and the slight differences can lead to flaky tests which should always be avoided.
One way to specify which browser to use for a test is to have a central configuration file. In C# this would be using the app.config that has a collection for each browser and doing
Config
<Firefox>
<addKey browserVersion='3.5.6' OS='WindowsXP'>
</Firefox>
Central Config Class looking inside 1 element
public class BoothElement : ConfigurationElement
{
[ConfigurationProperty("browserVersion", DefaultValue = "", IsKey = true, IsRequired = true)]
public string browserVersion
{
get
{
return ((string)(base["browserVersion"]));
}
set
{
base["browserVersion"] = value;
}
}
Tests
selenium = new DefaultSelenium(HubPort, HubPort, browserVersion, SUTServer);
selenium.Open("/test.htm");
//Rest of the test
In python you could create an Array in a module that you include on all your tests
include.py
hubServer = 'hub'
hubPort = 5555
sut = 'http://serverUnderTest'
firefox = [hubServer,hubPort,"\*chrome",sut]
iexplore = [hubServer,hubPort,"\*iehta",sut]
test.py
sel = selenium(firefox)
sel.open("/test.html")
#rest of the test
When using Selenium Grid try thinking of it more as a test infrastructure help framework and hopefully that will help you a little more.
Second, I'd like to run portable
versions of those browsers. I've only
seen that specified in the tests, and
not in the RC's command line to run
them. That is the way to do it, per
test?
I have never tried to get Selenium to work on mobile browsers and don't think it would work to well, however with Selenium 2 which is currently in alpha there is android support for testing apps.
EDIT FROM COMMENT
- name: "Firefox on OS X"
browser: "*firefox"
- name: "Firefox on Linux"
browser: "*firefox"
- name: "IE on Windows"
browser: "*iehta"
- name: "Safari on OS X"
browser: "*safari"
So say we have the above setup, according to the YAML file we have a number of different *firefox instances. So to call those different ones in our tests our browser setup command would look like
selenium.Start(hubHost, hubPort, "Firefox on Linux", "http://serverUnderTest");
or selenium.Start(hubHost, hubPort, "Firefox on OS X", "http://serverUnderTest");
The hub will translate that into *firefox for you. I prefer having very granular names for my environment instead of the usual *firefox so that if there is a failure its easier to spot where it was and on which specific browser.
Virtual machines can be very handy for setting up "inexpensive" mules in the Selenium Grid farm.