I've got a situation here where I have a central selenium grid hub, and several RC's running on my gogrid account. When I access it to run tests, it basically queues all the incoming test requests and executes them serially on only one of the RC's, instead of spreading them out to use available RC's. The tests come from multiple projects, so I'm not looking to parallelize the tests themselves, just to split the requests that come from multiple projects across the multiple RC's. From everything I've read, it seems like selenium grid should be doing this already, yet I only see one RC used to run every single test. Is there something I'm missing?
Do you have multiple RCs registered with the same environment? Each worker can currently only broadcast one environment and the hub assigns work to workers in a circular queue fashion (first registered gets work first).
http://selenium-grid.seleniumhq.org/configuring-and-tuning.html
Related
I need some help on the below query:
Currently we are using local VMs for Selenium grid execution. (Migration is happening from Selenium 3 to 4 but currently Docker is not in scope of things. )
The question is how can we create a sort of monitor to get the data of which test case is being executed on which node, the current status of the test cases executed to understand how many have passed/failed and how many are pending. Is there any readily available solution or any kind of plugins or open source library to achieve this?
how to run my selenium script on multiple servers at the same time so that i would save one by one execution time ?
Scenario: I have 1 hour of total testing time and 20 servers to test and 1 test script takes about 30mins to execute so i want to run my test script simultaneously on all 20 servers in order to save time.
The answer to your question is parallel execution
There are multiple ways to achieve this like 1> Creating a Jenkins Job, Registering all the server machines as slave and executing jobon servers
Or
2> A comparitively simpler and more widely used concept, using Selenium Grid
To Implement Selenium Grid ( for instance with java as a programming language and TestNG framework) we need to take care that
A. we have implemented Threading for our webdriver, so that each execution works on its own driver instance.
B. We have our Testng.xml with attribute set as parallel=tests
You can easily get step by step guide to establish selenium grid to summarize it, We have one machine as hub(master), we have multiple machines or nodes(slave) registered to the hub, your automation code is supplied to the hub, Hub routes test on multiple nodes keeps track of individual execution and gets you the result on the Hub machine itself
I have set up a Selenium Grid with 3 different servers running nodes as windows services. I need to restart those machines regularly to avoid memory leaks and under performance. To do this I need to schedule a job to shutdown the nodes on a server and restart while tests would be pushed to the remaining servers. Then repeat the same process with scheduled jobs on the other servers.
Is there a way to shut down a selenium node once the current test finishes ? Or to get the status of a particular node so I can check each one myself in a scheduled job to make sure the node is not running a test before I shut it down ?
You could check the number of active sessions by requesting each node with the /sessions command:
http://127.0.0.1:4444/wd/hub/sessions
Response :
{"state":"success","sessionId":null,"hCode":3217742,"value":[],"class":"org.openqa.selenium.remote.Response","status":0}
#Sh3mm
Sometime back I wrote up a blog post which basically talks about how to go about building a "Self Healing Grid" which is what you essentially are after.
You can read through my blog post on that from here.
We essentially used the same approach when we working on building the SeLion Grid. The SeLion Grid packs in a few more sophistications. Read more about it here
There's another flavor of essentially the same sort of functionalities that was built by GroupOn as part of their Grid Extras. You can take a look at it here
My Situation:
I have 60 dedicated selenium nodes that I use, in AWS. My Selenium Grid and all my Selenium Nodes are in the AWS. (cuts down network requests)
My Problem:
I am experiencing an issue when executing my regression suite. I run only 4 tests at a time due to our nightly servers only being able to handle so many tests going at it at once. The problem that i've diagnosed is that when using t1.micro instances, the CPU tops off when launching only 1 test per 1 node. I keep getting several timeouts. Fine. I can easily upgrade these to m1.small but it makes me think more about the way that the selenium grid selects it's nodes. Per my understanding, this is a quick getup on how the hub handles it's nodes.
So if we only have 4 tests running at a time, it will select the first four nodes. Once these nodes are finished running the tests, it will launch another test against the same instances. Is there any way that I can bypass this logic to just select a random available node rather than "the next one in queue"? I'm noticing that the X amount of nodes after the four nodes are NEVER being utilized.
Sidenote:
I have 60 nodes because this grid will have several different types of regression suites against it. Not only mine.
Edit (July 25, 2015)
A pull request was created for this functionality: https://github.com/SeleniumHQ/selenium/pull/832
This pull request should fix the issue
https://github.com/SeleniumHQ/selenium/pull/832
In our environment we have quite a few long-running functional tests which currently tie up build agents and force other builds to queue. Since these agents are only waiting on test results they could theoretically just be handing off the tests to other machines (test agents) and then run queued builds until the test results are available.
For CI builds (including unit tests) this should remain inline as we want instant feedback on failures, but it would be great to get a better balance between the time taken to run functional tests, the lead time of their results, and the throughput of our collective builds.
As far as I can tell, TeamCity does not natively support this scenario so I'm thinking there are a few options:
Spin up more agents and assign them to a 'Test' pool. Trigger functional build configs to run on these agents (triggered by successful Ci builds). While this seems the cleanest it doesn't scale very well as we then have a lead time of purchasing licenses and will often have need to run tests in alternate environments which would temporarily double (or more) the required number of test agents.
Add builds or build steps to launch tests on external machines, then immediately mark the build as successful so queued builds can be processed then, when the tests are complete, we mark the build as succeeded/failed. This is reliant on being able to update the results of a previous build (REST API perhaps?). It also feels ugly to mark something as successful then update it as failed later but we could always be selective in what we monitor so we only see the final result.
Just keep spinning up agents until we no longer have builds queueing. The problem with this is that it's a moving target. If we knew where the plateau was (or whether it existed) this would be the way to go, but our usage pattern means this isn't viable.
Has anyone had success with a similar scenario, or knows pros/cons of any of the above I haven't thought of?
Your description of the available options seems to be pretty accurate.
If you want live update of the builds progress you will need to have one TeamCity agent "busy" for each running build.
The only downside here seems to be the agent licenses cost.
If the testing builds just launch processes on other machines, the TeamCity agent processes themselves can be run on a low-end machine and even many agents on a single computer.
An extension to your second scenario can be two build configurations instead of single one: one would start external process and another one can be triggered on external process completeness and then publish all the external process results as it's own. It can also have a snapshot dependency on the starting build to maintain the relation.
For anyone curious we ended up buying more agents and assigning them to a test pool. Investigations proved that it isn't possible to update build results (I can definitely understand why this ugliness wouldn't be supported out of the box).