I got a requirement from my client to implement Selenium grid. Requirement details as follows:
Number of nodes running concurrent test cases: 200-250
3 Selenium nodes per machine (~ 66 to 85 machines running the 200-250 nodes)
Every machine is a Windows machine and has 8GB RAM
The test cases are long-running in nature. about 10000 steps per test case. (click, type etc are considered to be the test steps)
The option of using grid services from providers like browserstack is ruled out. I will have to do the grid setup locally.
I am looking for answers for questions like, Has anyone tried grid setup with such level of complexity? Are there any limitations of Selenium to support such large setup? Would there be performance impact due to excessive context switches between threads? What problems should I be prepared to face?
Thank you for your help.
It seems possible to me. Some things to consider:
Long running tests with many steps should be broken down into
smaller test units.
Each grid node, if they have 8GB of ram and dual-proc, should be able to handle up to 10+ browsers, not just 3.
If the tests are simple, you could use PhantomJS 2.x headless browser and get more speed and throughput out of your tests. This
is especially useful if the version of the browser your testing with
doesn't much matter, such as with verification tests. Regression
tests, on the other hand, need specific browser versions to verify
regressions caused by code.
I would say that a grid hub with many nodes on it , should be able to handle 50+ threads at a level of 8gb and dual proc. So, for
each 50 threads you would want 5 nodes and 1 hub.
Thats just some thoughts...
Related
We are trying TestCafe with 50 concurrency and TestCafe randomly with "Cannot read property 'stackFrames' of undefined"
Same code base when we run with 20 threads works fine without any issues. Is there any upper limit on number of threads in Testcafe?
Note: - We have enough cpu to support 50 threads in aws (c5a.8xlarge)
TestCafe has no formal limit on the number of threads. However, we have not tested the concurrency mode with so many threads. Therefore, I can only give general recommendations: make sure that you have enough resources to run so many browsers (not only CPU): RAM, disk I/O, GPU. Also, run browsers in headless mode, for example: testcafe -c 50 chrome:headless test.js.
I successfully implemented a solver that fits my needs. However, I need to run the solver on 1500+ different "problems" at 0:00 precisely, everyday. Because my web-app is in ruby, I built a quarkus "micro-service" that takes the data, calculate a solution and return it to my main app.
In my application.properties, I set:
quarkus.optaplanner.solver.termination.spent-limit=5s
which means each request take ~5s to solve. But sending 1500 requests at once will saturate the CPU on my machine.
Is there a way to tell OptaPlanner to stop when the solution is good enough ? ( for example if the score is stable ... ). That way I can maybe reduce the time from 5s to 1-2s depending on the problem?
What are your recommandations for my specific scenario?
The SolverManager will automatically queue solver jobs if too many come in, based on its parallelSolverCount configuration:
quarkus.optaplanner.solver-manager.parallel-solver-count=3
In this case, it will run 3 solvers in parallel. So if 7 datasets come in, it will solve 3 of them and the other 4 later, as the earlier solvers terminate. However if you use moveThreadCount=2, then each solver uses at least 2 cpu cores, so you're using at least 6 CPU cores.
By default parallelSolverCount is currently set to half your CPU cores (it currently ignores moveThreadCount). In containers, it's important to use JDK 11+: the CPU count of the container is often different than from the bare metal machine.
You can indeed tell the OptaPlanner Solvers to stop when the solution is good enough, for example when a certain score is attained or the score hasn't improved in an amount of time, or combinations thereof. See these OptaPlanner docs. Quarkus exposes some of these already (the rest currently still need a solverConfig.xml file), some Quarkus examples:
quarkus.optaplanner.solver.termination.spent-limit=5s
quarkus.optaplanner.solver.termination.unimproved-spent-limit=2s
quarkus.optaplanner.solver.termination.best-score-limit=0hard/-1000soft
We are using chrome lighthouse extension for running tests on our pages and improves performances score. We have however a problem. On a MacBook Pro-2015, we get a 94 performance score and also 94 performance score in a Windows virtual machine running on the laptop. However on a 64-bit Windows laptop with an Intel Pentium n3540 CPU, we get below 50. Even demo pages found online that claim 90+ score will not go above 50 on this system.
Are there any specific requirements that the lighthouse has in order to run correctly? Or some special settings need to be done on the Windows system?
Lighthouse can be configured to run on most systems, nothing precludes it from running on windows (it should run on windows out of the box), but there will definitely be variation based on the system that it runs on un-throttled. If you want a consistent bar to run against consider running against WebPageTest. Or play with their custom settings to get an audit that is exactly what you want that can be run consistently.
I think that running lighthouse on a computer with a chip that was released 4 years ago will probably yield low results, as you are experiencing. Which is probably an accurate depiction of how performant a website will be on that hardware. This might be a case of WAI.
Are you running Lighthouse with simulated throttling, or with throttling off? That can also increase variation.
I've recently switched from FirefoxDriver to GhostDriver.
My test suite (roughlty around 150 end to end tests) used to take 25 minutes to run. Now with Ghostdriver it's taking 23 minutes. So in total I've gained 2 minutes of run time.
It's an improvement but not as huge as I would've expected. Is such a low gain normal ? Or should I expect a much higher cut down time from switching to an headless test runner ?
I'm using the .NET version of webdriver/ghostdriver.
I'd be really interested to compare this "benchmark" with anyone having recently switched to GhostDriver
GhostDriver won't offer any performance gain over ChromeDriver as the only thing it is not doing is rendering the graphics on the screen which Chrome does using a graphics driver and not the CPU.
The only benefit I see is not popping up a browser window and maybe when running on a CI server.
Some stats on my test run are on my blog: http://watirmelon.com/2013/02/05/watir-webdriver-with-ghostdriver-on-osx-headless-browser-testing/
I feel like this is bit vague question. For example, if your test contains lots of Thread.sleep() calls, then there will be no improvement to the tests.
My tests use ChromeDriver and I didn’t think about switching. But I feel like that simple switch will not gain you much, maybe also consider refactoring your code
Does anybody have experience using 20/50/100 RCs connected to a single Selenium Hub? I'm thinking of a centralized hub that multiple team could use together.
I heard that after 20 RCs, performance goes down significantly.
My tests use 104 RCs which are connected to a single Hub. Hub is able to handle all the RCs and the system is stable. One thing that you should note is, how you pass the commands to the Hub.This determines how much RCs Hub is going to utilize. What we do in our grid is, we run multiple builds at the same time - all pointing to the same Hub. This would mean Hub is always kept busy.
I'm not sure where the "20" value is coming from. I run with significantly more than that to power Mogotest and we run many more test sessions across various sites than most users of grid. Granted older versions had some thread problems, but 1.0.8 should be pretty solid and the 2.0 releases have not exhibited any problems.
FWIW, I maintain the grid project and backport any known fix for any known load issues. I also know of several others running large-scale grid installations without incident.
It boils down to how robust your h/w is. Though I can't say exact statistics yet suggest to keep it with in 20
Well there is not but yes very much close to 20 for good performance.