In my current project, we have already developed test scripts with junit framework. We have used all sort of locators methods which includes xpath as well.Now we have to do parallel execution on IE8,IE9 and FF21.We implemented grid2 with junit but we see that majority of the scripts are failing due to time out.
What I would like to know is if we have to execute 100 test cases on 5 nodes:
1.what is minimum hardware configuration required in terms of RAM,and JVM configuration
2.What is ideal hub/node configuration
3.What desired capabilities have to be set?
4.what is the best approach to set wait methods for elements and messages?
Has anyone successfully implemented selenium grid2 and ran more than 100 test cases in parallel?
Here goes my question, Have you ever executed those 100 test cases in sequence with out any such timeout issues? If yes, then you can do it in grid as well.For this, you may need minimum of 4 GB RAM and better clocked CUP. JVM version can be the same in which your scripts are done.
In general, you may need to handle some run time behavior of your application.
Related
I have a project with several hundred test files some of the test files use DataJpaTest annotation, some are MockMvc based controller tests and some uses mocked objects without database dependency, Based on test execution order I see context needs to be re-initialized for different flavors of test files, Is there a way to control execution of test files order so that context reload can be avoided? Say all mock tests first followed by controller tests and then DataJpaTest?
Right now test case execution taking about 30 minutes looking for way to improve the speed up test execution.
JUnit Jupiter provides options to control Test Execution Order.
However, you should look into your test setup and verify if your tests create too many Application Contexts.
Spring Test framework can cache Application Contexts and reuse them among different test suites. See Spring Test documentation for more info.
I'm running a big campaign with about 500 tests targeting the API of an orchestration tool using both a parallel and a sequential runner for some of the tests which need to be executed in sequence.
This works fine using the Karate version 0.9.6.
As soon as I upgrade the Karate version to anything higher than v0.9.6, the results of the tests in the parallel runner always consists of a significant portion of failing tests. The sequential runner does not have any issues. Also the parallel runner seems to work fine only if I set it's threadCount = 1.
If the threadCount is higher than 1, the runner seems always to start well but after some time there are quite some failed transactions without any further details like this: http call failed after 55 milliseconds for url: http://...
This is all the error log I have as there is not more than this entry for each failed test (around 40% of the full campaign).
As soon as one thread starts with an issue like that, basically all other threads follow to do the same and make their tests fail for the same reason (only with a slightly different number of milliseconds). While investigating the problem I was not able to identify a common pattern (like always starting with the same test failing, etc.).
Did anybody else face similar issues with Multithreading using a higher version than v0.9.6?
Is there a way to get more detailed logs? I use the value DEBUG in the logback-test.xml
Is there any recommendation of what to try to make it work?
Please don't hesitate in case you need more information.
I would be happy for any kind of help regarding this as I would like to benefit of the new karate-gatling facilities for Performance Testing which is only available in the latest versions of Karate.
Many Thanks for your reply!
we run 8 threads on karate 1.1.0 with no issues at all. Do you have some extra logs? So you don't get same timeout error http call failed after 55 milliseconds for url: http://... in lower version the running multithreads ?
Shot in the dark here, but do some of your scenarios have the same title? If so, try making them all unique and see if that solves the issue.
More logs will be useful. Its very little information here.
I am not sure of the exact error but I faced similar issues sometime ago but that was caused due to race conditions especially if you are trying to use variables set by java methods. Are you using Java interop in your tests? If so worth checking how these are being used. A test may be trying to execute with a variable/parameter that is still in use by other tests when running in parallel. This wouldn't be an issue in sequential.
I have to execute large number of test cases in parallel using TestNG and Selenium. Each test case will be executed in different data set using Data driven testing. How to run these test cases in parallel in different machines? We can use Parallel attribute in TestNG but that is restricted to a single machine.
Can Selenium Grid tweaked and use in this purpose? If yes how to use this or any other suggestion?
I want examples of (https://www.seleniumhq.org/docs/07_selenium_grid.jsp#when-to-use-it)
To reduce the time it takes for the test suite to complete a test
pass.
Basically it's quite complicated it needs lot of understanding i haven't done it but i know that you need to create one root machine and rest machines will be childs of the parent machine then you can run the test scripts parallel but you need to make sure that those script shouldn't be dependent otherwise their will be lot of issue
I have shared the link with you so you can check how you set up?
https://medium.com/#appening/how-to-run-your-test-on-multiple-machines-using-selenium-grid-3aa37d5d2b63
Currently, I have five instances of phantomjs set up on my Selenium Grid all on different nodes. I am trying to run 5 parallel tests at once, but some of these tests will fail. Each time, it is a different test that will fail on different lines.
I have also put in waits and assertions to wait until the element shows up, but the test cases are still failing in different spots each time. When I run the tests one by one however, all of the tests pass 100% of the time.
Is there any way I can optimize my tests for parallel execution?
My tests are written in Java in a Windows environment. I am using phantomjs 1.9.8 and Selenium 3.54
Anecdotal: When I find my tests run fine on my machine but not so well on a Grid Node, it is typically a timing issue. My workstation is way more powerful than my VMs (Grid Nodes) so simple animations take longer to render. It got to the point where I wrote a .ClickViaJavaScript() to use instead of .Click(). I also threw in a 500ms sleep after the js click.
// Click element via JavaScript
((IJavaScriptExecutor)driver).ExecuteScript("arguments[0].click()", element);
// Wait a moment
System.Threading.Thread.Sleep(500);
Parallel doesn't really necessitate any code being rewritten so long as you do not change global variables at run-time and each test is atomic.
We currently use Behat 3 to automate BDD tests for our website.
The current setup uses Jenkins to run Selenium which attaches to Firefox and uses XVFB to render (this allows us to save screenshots when anything goes wrong).
This is great for testing that the site (including JavaScript) works and that a user can perform each documented task successfully.
I am looking to expand our testing facilities, and one thing I would like to add is the ability to check multiple browsers. This is very important as we get occasional quirks that can break functionality.
Since the tests currently take slightly over an hour to run (and we have 4 suites for that site on Jenkins), I'd preferably like to run all the browsers at the same time. If I can't find a way to do it concurrently, then I likely will just set up multiple Behat profiles and run each one in series.
One thing I've been looking at as a possible solution is Ghostlab. This would allow us to test across, multiple browsers and multiple devices, including mobile, at the same time. The problem is that I can't find a way of joining this to Behat in a meaningful way.
I could run one browser connected to Ghostlab, which would cause the same actions to be taken across all connected browsers, however, were a browser other than the one controlled by Selenium to break, I do not know how we would capture that information.
TL;DR: Is there any way for me to run BDD (preferable Behat) tests across multiple browsers in parallel, and capture information from any browser that fails?
This is what multi-configuration jobs (or matrix jobs) are designed for in Jenkins.
You specify your job configuration once, but add one or more variables that should change each time, building a matrix of combinations (in your case, the matrix has one dimension: browser).
Jenkins then runs one main build with multiple sub-builds in parallel — one for each combination in the matrix. You can then clearly see the results for each combination.
This requires that your test job can be parameterised, i.e. you can choose at runtime which browser should be run, rather than running all tests together in a single job.
The Jenkins wiki has minimal documentation on this feature, but there are a few good blog posts (and Stack Overflow questions) out there on how to set it up.
A matrix job will use all available "executors" in Jenkins, to run builds in parallel as much as possible.
In a default Jenkins installation, there are two executors availble, but you can change this, or extend Jenkins by adding further build machines.