Selenium grid - node configuration file for Firefox not reading moz:firefoxOptions - selenium

I have setup my selenium grid (currently 1 hub and 1 node).
The hub is working fine but for my node I setup the below configuration file (nodeconfig.json)
I wanted to start the firefox in headless mode but it looks like the "moz:firefoxOptions" configuration did not catch up.
I tried setting the "moz:firefoxOptions" in a variety of location in side the json file without any luck :( .
I are using :
windows server 2016 64bit ,
selenium-server-standalone 3.9.1
geckodriver v0.19.1
firefox v59.0
we are running the node using the following command :
java -Dwebdriver.gecko.driver="geckodriver.exe" -Dwebdriver.chrome.driver="chromedriver.exe" -jar selenium-server-standalone-3.9.1.jar -role node -nodeConfig nodeconfig.json
I can tell that the configuration file was read as I can see it in the selenium grid console.
nodeconfig.json file:
{
"capabilities": [
{
"browserName": "firefox",
"maxInstances": 5,
"seleniumProtocol": "WebDriver",
"alwaysMatch": {
"moz:firefoxOptions": {
"args": [
"-headless"
]
}
}
},
{
"browserName": "chrome",
"maxInstances": 5,
"seleniumProtocol": "WebDriver"
}
],
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"maxSession": 5,
"port": 5555,
"register": true,
"registerCycle": 5000,
"hub": "http://localhost:4444",
"nodeStatusCheckTimeout": 5000,
"nodePolling": 5000,
"role": "node",
"unregisterIfStillDownAfter": 60000,
"downPollingLimit": 2,
"debug": false,
"servlets": [],
"withoutServlets": [],
"custom": {}
}
In addition from the logs I can tell that the argument is not added.
The relevant line is :
1520854660258 mozrunner::runner INFO Running command: "C:\Program Files\Mozilla Firefox\firefox.exe" "-marionette" "-profile" "C:\Users\avi\AppData\Local\Temp\rust_mozprofile.03tQm3hduVDR"

As per the documentation the argument is --headless not -headless. So possibly you have to change to :
"moz:firefoxOptions": {
"args": [
"--headless"
]
}
You can find a detailed discussion in How to make firefox headless programatically in Selenium with python?

Related

How do you setup an HtmlUnit node with Selenium Grid?

I'm having trouble setting up an HtmlUnit node with Selenium Grid.
I'm launching my Hub with:
java -jar selenium-server-standalone.jar -role hub -port 4444
And HtmlUnitDriver nodes with:
java -jar selenium-server-standalone.jar -role node -browser browserName=htmlunit,maxInstances=5 -hub http://localhost:4444/grid/register -port 5564
My code to connect to the node, which is getting the exception, looks something like:
DesiredCapabilities cap = new DesiredCapabilities();
cap.setBrowserName("htmlunit");
cap.setJavascriptEnabled(false);
driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), cap);
The exception I'm getting is:
Unable to create session from {
"desiredCapabilities": {
"browserName": "htmlunit",
"server:CONFIG_UUID": "93a0486d-c9f6-46da-8065-603ab07c0294",
"javascriptEnabled": false
},
"capabilities": {
"firstMatch": [
{
"browserName": "htmlunit",
"server:CONFIG_UUID": "93a0486d-c9f6-46da-8065-603ab07c0294"
}
]
}
}
To set up a Selenium Grid instance, I use the Selenium Foundation library. The "standalone" JAR brings in scads of transitive dependencies you don't need that end up creating conflicts. I created a sample project (local-grid-utility) that launches a Selenium 2 Grid that dispenses PhantomJS sessions, which could be easily altered to launch a Selenium 3 Grid dispensing HtmlUnit sessions.
The key to getting HtmlUnit sessions stood up in Grid is providing a complete class path that includes all of the dependencies. Here's what the library assembles for HtmlUnitDriver 2.40.0:
${JAVA_HOME}/jre/bin/java -cp ${M2_ROOT}/repository/org/seleniumhq/selenium/selenium-support/3.141.59/selenium-support-3.141.59.jar:${M2_ROOT}/repository/org/seleniumhq/selenium/htmlunit-driver/2.40.0/htmlunit-driver-2.40.0.jar:${M2_ROOT}/repository/org/apache/commons/commons-lang3/3.8.1/commons-lang3-3.8.1.jar:${M2_ROOT}/repository/xerces/xercesImpl/2.12.0/xercesImpl-2.12.0.jar:${M2_ROOT}/repository/org/seleniumhq/selenium/selenium-api/3.141.59/selenium-api-3.141.59.jar:${M2_ROOT}/repository/net/sourceforge/htmlunit/htmlunit-core-js/2.40.0/htmlunit-core-js-2.40.0.jar:${M2_ROOT}/repository/org/eclipse/jetty/websocket/websocket-api/9.4.28.v20200408/websocket-api-9.4.28.v20200408.jar:${M2_ROOT}/repository/xalan/xalan/2.7.2/xalan-2.7.2.jar:${M2_ROOT}/repository/org/eclipse/jetty/jetty-xml/9.4.28.v20200408/jetty-xml-9.4.28.v20200408.jar:${M2_ROOT}/repository/org/apache/httpcomponents/httpclient/4.5.12/httpclient-4.5.12.jar:${M2_ROOT}/repository/org/eclipse/jetty/jetty-http/9.4.28.v20200408/jetty-http-9.4.28.v20200408.jar:/Applications/Eclipse.app/Contents/Eclipse/plugins/com.beust.jcommander_1.72.0.jar:${M2_ROOT}/repository/org/apache/httpcomponents/httpcore/4.4.13/httpcore-4.4.13.jar:${M2_ROOT}/repository/xml-apis/xml-apis/1.4.01/xml-apis-1.4.01.jar:${M2_ROOT}/repository/commons-codec/commons-codec/1.11/commons-codec-1.11.jar:${M2_ROOT}/repository/net/sourceforge/htmlunit/neko-htmlunit/2.40.0/neko-htmlunit-2.40.0.jar:${M2_ROOT}/repository/com/squareup/okhttp3/okhttp/3.11.0/okhttp-3.11.0.jar:${M2_ROOT}/repository/commons-net/commons-net/3.6/commons-net-3.6.jar:${M2_ROOT}/repository/org/seleniumhq/selenium/selenium-server/3.141.59/selenium-server-3.141.59.jar:${M2_ROOT}/repository/xalan/serializer/2.7.2/serializer-2.7.2.jar:${M2_ROOT}/repository/org/eclipse/jetty/jetty-client/9.4.28.v20200408/jetty-client-9.4.28.v20200408.jar:${M2_ROOT}/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar:${M2_ROOT}/repository/org/apache/commons/commons-text/1.8/commons-text-1.8.jar:${M2_ROOT}/repository/org/eclipse/jetty/jetty-io/9.4.28.v20200408/jetty-io-9.4.28.v20200408.jar:${M2_ROOT}/repository/net/sourceforge/htmlunit/htmlunit-cssparser/1.5.0/htmlunit-cssparser-1.5.0.jar:${M2_ROOT}/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:${M2_ROOT}/repository/org/apache/httpcomponents/httpmime/4.5.12/httpmime-4.5.12.jar:${M2_ROOT}/repository/org/seleniumhq/selenium/jetty-repacked/9.4.12.v20180830/jetty-repacked-9.4.12.v20180830.jar:${M2_ROOT}/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:${M2_ROOT}/repository/org/eclipse/jetty/websocket/websocket-client/9.4.28.v20200408/websocket-client-9.4.28.v20200408.jar:${M2_ROOT}/repository/org/brotli/dec/0.1.2/dec-0.1.2.jar:${M2_ROOT}/repository/com/google/guava/guava/28.1-jre/guava-28.1-jre.jar:${M2_ROOT}/repository/com/squareup/okio/okio/1.14.0/okio-1.14.0.jar:${M2_ROOT}/repository/net/sourceforge/htmlunit/htmlunit/2.40.0/htmlunit-2.40.0.jar:${M2_ROOT}/repository/org/seleniumhq/selenium/selenium-remote-driver/3.141.59/selenium-remote-driver-3.141.59.jar:${M2_ROOT}/repository/commons-io/commons-io/2.6/commons-io-2.6.jar:${M2_ROOT}/repository/org/eclipse/jetty/jetty-util/9.4.28.v20200408/jetty-util-9.4.28.v20200408.jar:${M2_ROOT}/repository/org/eclipse/jetty/websocket/websocket-common/9.4.28.v20200408/websocket-common-9.4.28.v20200408.jar org.openqa.grid.selenium.GridLauncherV3 -role node -servlets org.openqa.grid.web.servlet.LifecycleServlet -host 192.168.1.6 -port 34464 -nodeConfig /Users/bb8d/github/Selenium-Foundation/target-s3/classes/nodeConfig-s3-9CF40153.json
You could create your own standalone package to bundle up the dependencies if that suits your fancy. The node server is provided by the GridLauncherV3 class. I also add the LifecycleServlet plugin to enable remote shutdown. The host specification corresponds to the local machine, and the port is auto-selected from whatever is currently available.
The JSON configuration file looks like this:
{
"capabilities": [
{
"browserName": "htmlunit",
"javascriptEnabled": true,
"maxInstances": 5,
"seleniumProtocol": "WebDriver"
}
],
"custom": {
},
"debug": false,
"downPollingLimit": 2,
"enablePlatformVerification": true,
"host": "0.0.0.0",
"hub": "http:\u002f\u002f192.168.1.6:4445\u002fwd\u002fhub",
"maxSession": 5,
"nodePolling": 5000,
"nodeStatusCheckTimeout": 5000,
"port": -1,
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"register": true,
"registerCycle": 5000,
"role": "node",
"servlets": [
],
"unregisterIfStillDownAfter": 60000,
"withoutServlets": [
]
}
The critical settings can be specified in the command line, but I think the config file is more manageable.

Selenium: Session externalKey not available

The Selenium Grid Hub I am running displays this error message whenever a Robot Framework automation test (kicked off by a Jenkins job) requests a Chrome browser from the Hub:
Session [(null externalkey)] not available and is not among the last
1000 terminated sessions.
I've spent hours searching and trying proposed solutions with no luck.
Running a Linux server as a Selenium Grid Hub and a windows server as a Selenium Grid Node. Have tried each node with versions 3.0.1, 3.5.3, 3.13.0, and 3.141.59 of the selenium-server-standalone-<version>.jar files. A third server runs Jenkins.
The Hub appears to be config'd properly because this works fine:
http://10.1.10.231:4444/grid/console
I've also run the jar files at the command line with -debug at the end but no further information is displayed. Have also tried -Dselenium.LOGGER.level=WARNING on the "java side" of the command line with no further information displayed than using -debug.
We have another set of servers with the same config that runs just fine.
Anyone have an idea of what this message is indicating?
Nitty-gritty follows.
Grid Hub:
java -jar -Xms1024m -Xmx2048m selenium-server-standalone-3.141.59.jar -role hub -hubConfig ./hub-config.json
hub-config.json:
{
"port": 4444,
"newSessionWaitTimeout": -1,
"servlets" : [],
"withoutServlets": [],
"custom": {},
"capabilityMatcher": "org.openqa.grid.internal.utils.DefaultCapabilityMatcher",
"throwOnCapabilityNotPresent": true,
"cleanUpCycle": 5000,
"role": "hub",
"debug": false,
"browserTimeout": 360,
"timeout": 1800
}
Grid Node:
SET CHROMEDRIVER=.\bin\chromedriver_win32-v2.37\chromedriver.exe
SET IEDRIVER=.\bin\IEDriverServer.exe
SET FFDRIVER=.\bin\geckodriver-v0.11.1-win32\geckodriver.exe
START "SeleniumNode" java -Xms1024m -Xmx2048m ^
-Dwebdriver.gecko.driver=%FFDRIVER% ^
-Dwebdriver.ie.driver=%IEDRIVER% ^
-Dwebdriver.chrome.driver=%CHROMEDRIVER% -Dwebdriver.chrome.args="--disable-extensions" ^
-jar .\bin\selenium-server-standalone-3.141.59.jar -role node -nodeConfig .\node-config.json
node-config.json:
{
"capabilities":
[
{
"browserName": "firefox",
"maxInstances": 5,
"platform": "WINDOWS",
"seleniumProtocol": "WebDriver"
},
{
"browserName": "chrome",
"maxInstances": 5,
"platform": "WINDOWS",
"seleniumProtocol": "WebDriver"
},
{
"browserName": "internet explorer",
"maxInstances": 5,
"platform": "WINDOWS",
"seleniumProtocol": "WebDriver"
}
],
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"maxSession": 5,
"port": 5555,
"register": true,
"registerCycle": 5000,
"hub": "http://10.1.10.231:4444",
"nodeStatusCheckTimeout": 5000,
"nodePolling": 5000,
"role": "node",
"unregisterIfStillDownAfter": 60000,
"downPollingLimit": 2,
"debug": false,
"servlets" : [],
"withoutServlets": [],
"custom": {},
"browserTimeout": 0,
"timeout": 1800,
"cleanUpCycle": 2000
}
Figured this one out! Hopefully it will help others who have reported this error message and not received a solution.
As is most often the case with an issue like this, especially when there's a known-working setup right beside the broken instance, it was an (embarrassingly) silly config issue. The Robot tests were pointing to this URL to get a browser:
http://10.1.10.75:4444/wd/hub/static/resource/hub.html
and it should have been pointing to this path:
http://10.1.10.75:4444/wd/hub
At least it was an easy fix.

Can't reach connection to Selenium Server hub from virtual node

So, I have PC with Windows 10 Pro, and virtual Windows 10 Pro installed on the same PC via VMWare workstation, network adapter mode in virtual OS set to "NAT".
I run this command on real OS (hub):
java -jar selenium-server-standalone-3.141.59.jar -role hub
And this command on virtual OS (node):
java -jar selenium-server-standalone-3.141.59.jar -role node -nodeConfig nodeconfig.json
nodeconfig.json contains this:
{
"capabilities":
[
{
"browserName": "chrome",
"maxInstances": 5,
"seleniumProtocol": "WebDriver"
}
],
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"maxSession": 5,
"port": -1,
"register": true,
"registerCycle": 5000,
"hub": "http://192.168.116.1:4444/grid/register/",
"nodeStatusCheckTimeout": 5000,
"nodePolling": 5000,
"role": "node",
"unregisterIfStillDownAfter": 60000,
"downPollingLimit": 2,
"debug": false,
"servlets" : [],
"withoutServlets": [],
"custom": {}
}
When I'm trying to connect node to hub, I got this message in console:
Registered a node http://192.168.116.1:4444
Marking the node http://192.168.116.1:4444 as down: cannot reach the node for 2 tries
How can I solve this ? I already tried to set virtual OS network adapter mode to "Bridged", but in this case I can't get connection at all ...
That was a problem with firewall, after turning it off everything seems to work.

Opera browser in Selenium grid: Driver class not found

I'm trying to setup an environment with Selenium grid to execute my Selenium test on a remote machine. The only browser I can`t get to work is Opera. I configured it the same as the other browsers. However when I start the node it shows a Driver class not found.
I`m Running windows 8.1 Enterprise 64 bit on my host computer
The node and the host are on windows 7 Enterprise 32 bit service pack 1
On the host opera is installed, the opera webdriver is in C:/GUI-Tests/Drivers/operadriver.exe, I tried the 32 bit webdriver and the 64 bit webdriver and I`m still getting the error:
13:30:37.169 INFO - Driver class not found: com.opera.core.systems.OperaDriver
13:30:37.169 INFO - Driver provider com.opera.core.systems.OperaDriver registration is skipped: Unable to create new instances on this machine.
13:30:37.178 INFO - Driver class not found: com.opera.core.systems.OperaDriver
13:30:37.178 INFO - Driver provider com.opera.core.systems.OperaDriver is not registered
If anyone knows something about this problem please help me.
Extra information about my environment
Starting the hub:
java -jar selenium-server-standalone-3.3.1.jar -role hub -hubConfig hubConfig.json
My Hub config:
{
"host": ip,
"maxSessions": 5,
"port": 4444,
"cleanupCycle": 5000,
"timeout": 300000,
"newSessionWaitTimeout": -1,
"servlets": [],
"prioritizer": null,
"capabilityMatcher": "org.openqa.grid.internal.utils.DefaultCapabilityMatcher",
"throwOnCapabilityNotPresent": true,
"nodePolling": 180000,
"platform": "WINDOWS"
}
Starting the node:
java -jar selenium-server-standalone-3.3.1.jar -role node -nodeConfig nodeConfig.json
My node config file:
{
"capabilities":
[
{
"browserName": "opera",
"platform": "WINDOWS",
"maxInstances": 5,
"seleniumProtocol": "WebDriver",
"webdriver.opera.driver": "C:/GUI-Tests/Drivers/operadriver.exe"
}
],
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"maxSession": 1,
"port": 5555,
"register": true,
"registerCycle": 5000,
"hub": "http://localhost:4444",
"nodeStatusCheckTimeout": 5000,
"nodePolling": 5000,
"role": "node",
"unregisterIfStillDownAfter": 60000,
"downPollingLimit": 2,
"debug": false,
"servlets" : [],
"withoutServlets": [],
"custom": {}
}
A screenshot of the drivers location:
Answering my own question because I discovered the answer, and I`m hoping to help everyone that googles on this question :)
Well it turns out opera driver is a legacy driver, you won't need it.
You do need the Opera browser and operachromiumdriver. In c# define your capabilities as:
capabilities = new DesiredCapabilities();
capabilities.SetCapability(CapabilityType.BrowserName, "operablink");
capabilities.Platform = new Platform(PlatformType.Windows);
_webDriver = new RemoteWebDriver(_gridServerUri, capabilities);
On your node:
{
"capabilities":
[
{
"browserName": "operablink",
"platform": "WINDOWS",
"maxInstances": 5,
"seleniumProtocol": "WebDriver"
}
],
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"maxSession": 1,
"port": 5555,
"register": true,
"registerCycle": 5000,
"hub": "http://localhost:4444",
"nodeStatusCheckTimeout": 5000,
"nodePolling": 5000,
"role": "node",
"unregisterIfStillDownAfter": 60000,
"downPollingLimit": 2,
"debug": false,
"servlets" : [],
"withoutServlets": [],
"custom": {}
}
I hope this will be useful to someone at least it was useful to me.
Note:
operachromiumdriver is in my path variable
my node and hub are running Windows 7

Setting up a node on a VM

I have my selenium all configured but seem to be having issues getting it to launch the browser on my windows VM. I think it's because I haven't haven't included the correct path to the driver. Currently it stands as...
java -jar .\selenium-server-standalone-2.39.0.jar -role node -port 1001 -hub http://xxx.xxx.xx.xxx:4444/grid/register
After reading online I assumed it's cause I'm missing the following...
-Dwebdriver.ie.driver="C:\selenium\IEDriverServer.exe"
I've tried various different ways to include this path into the above statement but keep getting 'poorly formatted Java property setting (I expect to see '=') -Dwebdriver: Usage: java -jar selenium-server.jar [-interactive] [options]'
if anyone could show me the correct format I'd really appreciate it.
Thanks.
According to documentation, you can use a system property:
System.setProperty("webdriver.ie.driver", "C:\selenium\IEDriverServer.exe")
Of course this path would have be the correct on the remote machine!
If you use Maven for your builds, then the Selenium binary downloader might interest you.
I would suggest running the node and having the node load a .json config file that looks something like this.
{
"capabilities":
[
{
"browserName": "firefox",
"acceptSslCerts": true,
"javascriptEnabled": true,
"takesScreenshot": false,
"firefox_profile": "",
"browser-version": "31",
"platform": "WINDOWS",
"maxInstances": 5
},
{
"browserName": "chrome",
"maxInstances": 5,
"platform": "WINDOWS"
},
{
"browserName": "internetExplorer",
"maxInstances": 1,
"platform": "WINDOWS"
}
],
"configuration":
{
"_comment" : "This is configuration for the grid node 1.",
"cleanUpCycle": 2000,
"timeout": 30000,
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"port": 5555,
"host": ip,
"Dwebdriver.chrome.driver=C:\\Selenium\\chromedriver.exe": "",
"Dwebdriver.ie.driver=C:\\Selenium\\IEDriverServer.exe": "",
"register": true,
"hubPort": 4444,
"maxSessions": 5
}
}