very slow tests through selenium grid - selenium

machine1:
java -jar selenium-server-standalone-2.43.1.jar -role hub
machine2:
java -jar selenium-server-standalone-2.43.1.jar -role node -browser browserName=chrome,maxInstances=1
All tests go to web app on the third machine. From fourth machine I run same amount of tests:
locally: 0m27s
remotely through grid: 2m50s
All of them running on the same local network. Why it's so slow then?
What should I check to find the bottleneck?
UPDATE1: ping between all machines ~0.03ms-0.89ms

I'd check the ping times between the servers. (I'm assuming in every case the test are running through the grid.)
If not, the grid is going to be your bottleneck.
You can speed things up if you keep in mind that every time you try to retrieve something from the web page, it has to go through the grid. Many times you can cache information.
I wrote about that here: http://blog.dmbcllc.com/selenium-performance-improvements/

So the problem was in abnormal load at the machine1 with grid server proccess running.
# uptime
14:35:31 up 9 days, 21:55, 2 users, load average: 2.86, 2.18, 2.16
I moved selenium grid server process to other fresh machine and everything speeds up.

Related

Selenium Grid -- Hub and Node performance issue

I am trying to run Selenium Grid.
Currently, I'm using v3.8.1 with one hub on 1 network and 20+ nodes of different networks registering to that Selenium hub.
It's executing fast when hub and node are created on the same machine where the application has deployed. Nodes created in other remote machines are comparatively slow.
Slow when we try to access particular Node by passing applicationName in capability instead of HUB selecting the node randomly.
More Info:
Windows server 2008, Ruby gem - selenium-webdriver-2.53.4, selenium-server-standalone-3.8.1, Java 8.
Tried Selenium Grid versions - 2.48, 2.49, 2.52, 2.53 and 3.8.1 as per https://github.com/SeleniumHQ/selenium/issues/1565.
Any help on this is appreciated. Thank You.
At last I got it. Its not a selenium node performance issue.
Its a rdp performance issue as it has shared resources.
Its working fine in individual VMs and in server machine.
Thanks You.

Selenium Standalone server

I want to run the selenium test cases in the remote machine.
These are the steps I have followed:
Steps I did in server machine:
I have downloaded selenium-server-standalone-3.9.1.jar + Chrome Driver & placed it in a folder and added it to $PATH environmental variable.
I have started selenium server using java -jar selenium-server-standalone-3.9.1.jar -role hub in the Terminal.
Steps I did in local machine:
I have installed selenium-webdriver & selenium-server-standalone-3.9.1.jar
I wrote a sample testcase using selenium webdriver.
I have started a node here using command java -jar selenium-server-standalone-3.9.1.ja-role node -hub http://172.24.111.70:4444/grid/register in the terminal. The node got connected to the hub.
I run the command node test.js in the terminal. I expect the chrome instance to open in the remote machine. But it is running in the local machine.
Screenshots:
Hub:(Remote)
Node:(Local)
Can you please explain me the mistake I have did here? Have I understood the concept wrong?
The web browser instance which will run your tests will get created on a node not the hub. The hub is how you can register a number of nodes so you can spread the load between different machines/nodes. From what I can tell, you have a hub on the remote machine which you connect a local node to.
What I think you want is to run the node on the remote machine also, registering to the hub on the same machine (you might have to change the IP address of the hub to localhost when registering the node).
So, the problem is you are wanting the tests to run on the Remote Machine, but they are running locally?
From the sounds of it you have done all the setup correctly, and the Hub/Nodes should be working fine. All you need to do, is instead of running the Node from your local machine, run it from the Remote Machine. You just need to think of the Hub as the Supervisor, who hands out work to the Nodes (their employees) to do. The Hub doesn't run any tests itself, it just sends the data to the Nodes and load balances between multiple Nodes.
We have a set up of 6 Remote Machines. Machine 1 has the Hub and a Node, and the other 5 have Nodes. In our project we are pointing towards the Hub, so when the tests run, they get sent there. The hub does the rest (depending on the nodecfg file and the settings you've got in there)

Selenium Grid 100 Instances

I want to run an automatic test develop with Protractor in parallel on 50 instances of google chrome using selenium grid on an 8CPU machine and 16GB RAM. but the machine becomes very slow and the load averge exceeds 40.
There is someone who can help me to increase the instances of chrome on this machine
Check Aerokube guys solution, it works way faster than original Selenium Grid:
Selenoid - Go implementation of original Selenium hub code. It
is using Docker to launch browsers.
GGR - A lightweight load
balancer used to create big Selenium clusters

Selenium grid - limit of the nodes on the same machine

In Selenium Grid, is there any limit of nodes on the same machine.
I have 500 test suites to run in parallel, and considering to setup a hub with 25 nodes each with MaxSessions of 20 Firefox browsers in the same machine or should I run virtual machines with Linux platform and register my nodes there to the hub?
Thank you for your advises.
Yes, you can limit the number of nodes with selenium grid
you can do something like this:
Start Hub with max number of session (use -maxSession 500)
Start node with limited number browsers. Since you want 25 nodes, with 20 firefox browser. It would look like something like this (assuming you are using java):
Java -jar %SELENIUM_GRID_VERSION% -role node -hub %HUB_URL% -browser browserName=firefox,maxInstances=20,platform=LINUX
This will insure that only 20 firefox browser will open on node1.
I would advice you to use Virtual Machines to run your test suite. Especially since you want run your test suites parallel. Running on same machine will use up a lot of your computer resources and the nodes might not register with hub or give you an error like "Unable to bind to locking port XXXX within 45000 ms", causing your test to fail.

Running Selenium Server & ChromeDriver as a Windows Service

So that we may perform front-to-back web UI testing, we are using Selenium and ChromeDriver to automate page loads/interaction as part of our testing pack.
This is behaving as expected during developer testing (on a developer's local machine), but we are struggling to perform these checks as part of our continuous integration build.
Our server plant is *NIX based, and all of our CI infrastructure runs on these machines. So that we may test Chrome under Windows (our delivery mechanism), we have configured a Selenium Grid. When the CI tests run, they access the grid, in order to locate a Windows node to run the tests on.
We have had a Windows desktop provisioned solely for the purpose of running these tests. This contains our standard enterprise build of Windows 7. This machine will be periodically rebooted in-line with the IT department's update policy.
In an effort to ensure the Selenium server is always running, we have added the Selenium Server (running in "node" mode) as a Windows service. The selenium Server is configured to start-up ChromeDriver to invoke the simulated user-interaction.
However, when running the tests from CI they fail due to timeout. Our working theory is, the system user that is running the service cannot create interactive windows. A web search has raised reference to the "Session 0" problem, but with little to no constructive advice on how to move forward.
Starting the Selenium Server process manually from an interactive session is not a viable solution, as this is leading to brittle tests - which are failing due to an infrastructure problem, rather than a genuine test regression.
How can we have an instance of Selenium Server started via a Windows Service whenever the system reboots, that is capable of launching Chrome instances?
It could be easily done with NSSM.
Installation of services looks like these:
nssm install seleniumhub java -jar C:\selenium\selenium-server-standalone-2.45.0.jar -role hub -hubConfig C:\selenium\hub.json
nssm install seleniumnode java -jar C:\selenium\selenium-server-standalone-2.45.0.jar -role node -nodeConfig C:\selenium\node.json
It provides easily way to remove service if needed:
nssm remove seleniumnode confirm
Add destination to nssm to your PATH variable and run from console as admin
UPDATE April 2021
NSSM is not supported for more than 3 years. So please consider other options like winsw or any other. WinSW does the same job as NSSM and allows to keep run configuration in xml.
You cannot run Selenium Grid as a windows service ever since Windows Vista. Microsoft calls it "Session 0 Isolation". You could do it in Windows 2000 or XP but since the time that Vista came out, Microsoft no longer will let Grid interact with the desktop (or any other UI programs for that matter). Regardless of the fact that you still see that "interact with desktop" checkbox, it is a red herring. So, you MUST run Selenium Grid in the foreground on that server in order for it to get access to the session. If it is running Windows Server, you could in theory have multiple sessions and leave Grid running in the foreground on one of the non-zero user sessions.
Right now you can't help it - it used to work fine in session 0 but for the past few days after chrome update only works for interactive sessions.
Related bugs:
https://code.google.com/p/selenium/issues/detail?id=8029
https://code.google.com/p/chromium/issues/detail?id=422218
My preferred solution to this problem (and my default choice for running Selenium Grid as a service) is to use a simple tool called AlwaysUp. It has a free 30 day trial to try it out.
What to do:
Download AlwaysUp
Configure AlwaysUp to start the Selenium Grid node on startup
Configure AlwaysUp to run the Selenium node as a specific user (not the default System user)
This way the the node will run as a service, survive machine restarts and work with the latest version Chrome.
If the user account you use to login to the machine is different from the user account you specify to run the node as a service then you will not see the browsers pop up on the desktop as they are running in a different user session. The end result is that it is almost identical to running as a normal service but gets round the Session 0 issue.
Yeah, you should use NSSM. Important is, that you add your windows account in the "Log on" tab, or any other valid account. If you run your node with the "Local System account" option, you will get the session 0 problem. With a normal user session, the nodes run smoothly invisible in the background :)
we don't use selenium GRID, we were disappointed with its stability. We use a "Jenkins Grid", that is jenkins slaves nodes on various servers.
The slaves are services with the interact with desktop flag. They run as services with NSSM, and the SERVICE_INTERACTIVE_PROCESS flag. Making sure that NoInteractiveProcess is set to 1 (cf https://learn.microsoft.com/en-us/windows/desktop/services/interactive-services).
We don't have the fancy features of the grid (that is balancing according the browser types slots). Instead, we have Jenkins balancing the test jobs using a slave node or another.
Initially we did not use the interact with desktop flag, having browsers to run without "real" display, but the behavior was not very stable (especially with resize commands).
Hope this helps.
As I explained on this thread, I found that using a small paid tool FireDaemon Pro saved me a lot of time from trying to configure NSSM and other free tools.
It works well in the background, and restarts Selenium along with the server, which was my main requirement for running Selenium Standalone Server as a Windows Service.
This free tool would probably do it:
http://yajsw.sourceforge.net/
For that to work, you need a wrapper.conf file and a script to run the YAJSW wrapper. I takes time to read the documentation, but it is a free solution.
I wrote an example shared here, that installs JBoss7 as a Windows service.
Of course, you can simplify my example by a lot.