I execute automation Selenium Scripts on remote machine which is set to perform 5 jobs at a time (5 scripts can be executed at a time by instantiating 5 browser instances).But everytime atleast 1 or 2 random scripts fail with random errors such as null pointer exception, Element not visible or not able to click the element. But it won't happen if only 3 jobs are run at a time. What's the best way i can prevent the scripts from failing.
very vague question. Cant really comment on why they are failing without seeing the implementation of classes.
It can happen for a lot of things including latency in network when you are running 5 instances. If that is the case you might considering using selenium grid and distribute your tests across 2 or more nodes in each of which you can run 3 instances.
Related
I need some help on the below query:
Currently we are using local VMs for Selenium grid execution. (Migration is happening from Selenium 3 to 4 but currently Docker is not in scope of things. )
The question is how can we create a sort of monitor to get the data of which test case is being executed on which node, the current status of the test cases executed to understand how many have passed/failed and how many are pending. Is there any readily available solution or any kind of plugins or open source library to achieve this?
how to run my selenium script on multiple servers at the same time so that i would save one by one execution time ?
Scenario: I have 1 hour of total testing time and 20 servers to test and 1 test script takes about 30mins to execute so i want to run my test script simultaneously on all 20 servers in order to save time.
The answer to your question is parallel execution
There are multiple ways to achieve this like 1> Creating a Jenkins Job, Registering all the server machines as slave and executing jobon servers
Or
2> A comparitively simpler and more widely used concept, using Selenium Grid
To Implement Selenium Grid ( for instance with java as a programming language and TestNG framework) we need to take care that
A. we have implemented Threading for our webdriver, so that each execution works on its own driver instance.
B. We have our Testng.xml with attribute set as parallel=tests
You can easily get step by step guide to establish selenium grid to summarize it, We have one machine as hub(master), we have multiple machines or nodes(slave) registered to the hub, your automation code is supplied to the hub, Hub routes test on multiple nodes keeps track of individual execution and gets you the result on the Hub machine itself
I am implementing a Cucumber - JVM based Selenium automation framework.
One of the workflow in the webapps i test, requires a long wait so that a batch job that is scheduled as frequently as in every 3 minutes, runs, and creates a login id, which the user can utilize, to continue with the workflow.
I am currently handling it in such a way that i execute the initial part test case first and continue with other test cases, so that the framework gets ample time to wait for the user id to be created.
After all other test cases are run the second part of the test case is run. But, before running the second part of the test case, i query the database and verify whether the id is created. If the id is created then the execution continues else, fails saying that the user id was not created.
Although this works for now, i am sure there are better ways to handle such scenarios. Have any one of you come across such a scenario? How did you handle it ?
I think I understand your problem. You actually would like to have an execution sequence like this probably:
Test 1
Test 2
Test 3
But if you implement Test 1 "correctly" it will take very long because it has to wait for the system under test to do several long running things, right?
Therefore you split Test 1 into to parts and run the tests like this:
Test 1 (Part A)
Test 2
Test 3
Test 1 (Part B)
So that your system under test has time to finish the tasks triggered by Test 1 (Part A).
As you acknowledged before, this is considered bad test design, as your test cases are no longer independent from each other. (Generally speaking no test case should rely on side effects created by another test case beforehand.)
My recommendation for that scenario would be to leave Test 1 atomic, i.e. avoid splitting it in two parts, but still run the rest of the tests in parallel. Of course whether or ot this is feasible depends on your environment and on how you trigger the tests, but that would allow you to have a better structure of your tests plus the benefit of fast execution. So you would end up with this:
Test 1 starts Test 2 starts
Test 1 waits Test 2 finishes
Test 1 waits Test 3 starts
Test 1 runs Test 3 finishes
Test 1 finishes
I am not sure about the start—>wait—>wait—>run approach. It might work for few tests and may not work well for hundreds of tests, as the wait time would be more. Even if we run it in parallel mode it would consume some time. What if we wait for more such time taking components in the same flow? More the components more the wait time, I guess. You may also need to consider timeout of the system, if you wait for longer time...
I feel even the first approach should be fine. No need to create multiple files for a test case. You can structure it in the same file in a way that you run first part and end it. And, after ensuring the batch processing you can start with second part of your test case(file). The first part can be run in parallel mode and after waiting time part 2 can also be executed in parallel.
I have read, that using grid, we can distribute tests across multiple nodes.
Say I have 4 test cases and 2 nodes(both have same configuration, platform:window, browser:firefox and maxSession=1, maxInstances:1) I was expecting 2 test cases to run on node 1 and 2 on node 2.
But when I run I see all 4 test cases run on either node 1 or node 2, based on who registers last. I have both registered to bug on different ports.
Is my understanding correct of running 2 test per node, or is this not possible.?
How do you achieve distributed testing for WebDriver?
I think what you mean is parallel execution (which is distributed testing)? Selenium grid will provide the architecture to run these tests in a distributed fashion (i.e. distribute the test execution across multiple nodes), but you still need to trigger these tests in parallel.
Currently your tests are triggered sequentially, so the Selenium Grid Hub utilizes the first available node it gets, which happens to be the first one each time (for obvious reasons).
To quickly visualize how selenium grid can benefit with parallel execution, open 4 command prompts/terminals and in each trigger 1 of the 4 tests. If you have only 2 nodes and each node can only run one browser, you will see
Node A - Test 1
Node B - Test 2
Waiting for a free/available node {Test 3 and Test 4}
Then depending on which test (1 or 2) finishes executing first that node gets free and Test 3 will start on that node.
Of course you need to programmatically trigger these tests in parallel.
In python you could use nose and trigger your test run like so :-
nosetests --processes=2 allmytests #Starts 2 processes
My Situation:
I have 60 dedicated selenium nodes that I use, in AWS. My Selenium Grid and all my Selenium Nodes are in the AWS. (cuts down network requests)
My Problem:
I am experiencing an issue when executing my regression suite. I run only 4 tests at a time due to our nightly servers only being able to handle so many tests going at it at once. The problem that i've diagnosed is that when using t1.micro instances, the CPU tops off when launching only 1 test per 1 node. I keep getting several timeouts. Fine. I can easily upgrade these to m1.small but it makes me think more about the way that the selenium grid selects it's nodes. Per my understanding, this is a quick getup on how the hub handles it's nodes.
So if we only have 4 tests running at a time, it will select the first four nodes. Once these nodes are finished running the tests, it will launch another test against the same instances. Is there any way that I can bypass this logic to just select a random available node rather than "the next one in queue"? I'm noticing that the X amount of nodes after the four nodes are NEVER being utilized.
Sidenote:
I have 60 nodes because this grid will have several different types of regression suites against it. Not only mine.
Edit (July 25, 2015)
A pull request was created for this functionality: https://github.com/SeleniumHQ/selenium/pull/832
This pull request should fix the issue
https://github.com/SeleniumHQ/selenium/pull/832