Does anybody have experience using 20/50/100 RCs connected to a single Selenium Hub? I'm thinking of a centralized hub that multiple team could use together.
I heard that after 20 RCs, performance goes down significantly.
My tests use 104 RCs which are connected to a single Hub. Hub is able to handle all the RCs and the system is stable. One thing that you should note is, how you pass the commands to the Hub.This determines how much RCs Hub is going to utilize. What we do in our grid is, we run multiple builds at the same time - all pointing to the same Hub. This would mean Hub is always kept busy.
I'm not sure where the "20" value is coming from. I run with significantly more than that to power Mogotest and we run many more test sessions across various sites than most users of grid. Granted older versions had some thread problems, but 1.0.8 should be pretty solid and the 2.0 releases have not exhibited any problems.
FWIW, I maintain the grid project and backport any known fix for any known load issues. I also know of several others running large-scale grid installations without incident.
It boils down to how robust your h/w is. Though I can't say exact statistics yet suggest to keep it with in 20
Well there is not but yes very much close to 20 for good performance.
Related
I successfully implemented a solver that fits my needs. However, I need to run the solver on 1500+ different "problems" at 0:00 precisely, everyday. Because my web-app is in ruby, I built a quarkus "micro-service" that takes the data, calculate a solution and return it to my main app.
In my application.properties, I set:
quarkus.optaplanner.solver.termination.spent-limit=5s
which means each request take ~5s to solve. But sending 1500 requests at once will saturate the CPU on my machine.
Is there a way to tell OptaPlanner to stop when the solution is good enough ? ( for example if the score is stable ... ). That way I can maybe reduce the time from 5s to 1-2s depending on the problem?
What are your recommandations for my specific scenario?
The SolverManager will automatically queue solver jobs if too many come in, based on its parallelSolverCount configuration:
quarkus.optaplanner.solver-manager.parallel-solver-count=3
In this case, it will run 3 solvers in parallel. So if 7 datasets come in, it will solve 3 of them and the other 4 later, as the earlier solvers terminate. However if you use moveThreadCount=2, then each solver uses at least 2 cpu cores, so you're using at least 6 CPU cores.
By default parallelSolverCount is currently set to half your CPU cores (it currently ignores moveThreadCount). In containers, it's important to use JDK 11+: the CPU count of the container is often different than from the bare metal machine.
You can indeed tell the OptaPlanner Solvers to stop when the solution is good enough, for example when a certain score is attained or the score hasn't improved in an amount of time, or combinations thereof. See these OptaPlanner docs. Quarkus exposes some of these already (the rest currently still need a solverConfig.xml file), some Quarkus examples:
quarkus.optaplanner.solver.termination.spent-limit=5s
quarkus.optaplanner.solver.termination.unimproved-spent-limit=2s
quarkus.optaplanner.solver.termination.best-score-limit=0hard/-1000soft
I got a requirement from my client to implement Selenium grid. Requirement details as follows:
Number of nodes running concurrent test cases: 200-250
3 Selenium nodes per machine (~ 66 to 85 machines running the 200-250 nodes)
Every machine is a Windows machine and has 8GB RAM
The test cases are long-running in nature. about 10000 steps per test case. (click, type etc are considered to be the test steps)
The option of using grid services from providers like browserstack is ruled out. I will have to do the grid setup locally.
I am looking for answers for questions like, Has anyone tried grid setup with such level of complexity? Are there any limitations of Selenium to support such large setup? Would there be performance impact due to excessive context switches between threads? What problems should I be prepared to face?
Thank you for your help.
It seems possible to me. Some things to consider:
Long running tests with many steps should be broken down into
smaller test units.
Each grid node, if they have 8GB of ram and dual-proc, should be able to handle up to 10+ browsers, not just 3.
If the tests are simple, you could use PhantomJS 2.x headless browser and get more speed and throughput out of your tests. This
is especially useful if the version of the browser your testing with
doesn't much matter, such as with verification tests. Regression
tests, on the other hand, need specific browser versions to verify
regressions caused by code.
I would say that a grid hub with many nodes on it , should be able to handle 50+ threads at a level of 8gb and dual proc. So, for
each 50 threads you would want 5 nodes and 1 hub.
Thats just some thoughts...
We're undertaking a large project that is focused on delivering automated testing of the software that we produce.
We have a lot of "events" that trigger certain behavior at specific times. Ideally, we would be able to exercise these tests in an automated fashion without the need to move the system clock in intervals to specific points in time.
To that end, I'm wondering if there is a way (with VMWare, or any other virtualization software) to increase the speed of the system clock of the guest operating system. I'm not interested in measuring performance in these tests, only functionality.
Is there anything out there that would allow for this behavior?
It works for VirtualBox:
VBoxManage setextradata "VM name" "VBoxInternal/TM/WarpDrivePercentage" x
where x is the percentage you want (for instance, 200 is doubling, 50 is halving)
You can also more information here, on the section "Accelerate or slow down the guest clock". Regards.
I was able to work around this using the Win32 API SetSystemTimeAdjustment()
This allows you to increase the amount of time added to the system clock for each OS tick interval. It's meant generally for addressing clock skew, but can be used outside of that particular context.
I don't see what the benefits are of testing this in a fast-forwarding VM instead of unit testing the event trigger using a mock implementation of the date/time dependency.
The only thing you "gain" by testing this in a fast-forwarding VM is that you test both the system's and the programming language's date/time implementation, which I think you are save to trust because it is used, developed and tested by so many for such a long time.
Are there any advantages what so ever to form a cluster if all the nodes are Virtual machines running inside the same physical host? Our small company just purchased a server with 16GB of Ram. I propose to just setup IIS on the box to handle outside requests, but our 'Network Engineer' argue that it will be better to create 3 VMs on the box and form a cluster with the VMs for load balancing. But since they are all in the same box, are there actual benefits for taking the VM approach rather than no VMs?
THanks.
No, as the overheads of running four operating systems would take a toll on performance, plus, I believe all modern web servers (plus IIS) are multithreaded so are optimised for performance anyway.
Maybe the Network Engineer knows something that you don't. Just ask. Use common sense to analyze the answer.
That said, running VMs always needs resources - but you might not notice. Doesn't make sense? Well, even if you attach the computer with a Gigabit link to the Internet, you still won't be able to process more data than the ISP gives you. If your uplink is 1MB/s, that's the best you can get. Any VM today is able to process that little trickle of data while being bored 99.999% of the time.
Running the servers in VMs does have other advantages, though. First of all, you can take them down individually for maintenance. If the load surges because your company is extremely successful, you can easily add more VMs on other physical boxes and move virtual servers around with a mouse click. If the main server dies, you can set up a replacement machine and migrate the VMs without having to reinstall everything.
I'd certainly question this decision myself as from a hardware perspective you obviously still have a single point of failure so there is no benefit.
From an application perspective it could be somewhat tenuously suggested that this would allow zero downtime deployments by taking VMs out of the "farm" one at a time but you won't get any additional application redundancy or performance from virtualisation in this instance. What you will get is considerably more management overhead in terms of infrastructure and deployment for little gain.
If there's a plan to deploy to a "proper" load balanced environment in the near future this might be a good starting point to ensure your application works correctly in a farm (sticky sessions etc). Although this makes your apparently live environment also a QA server, which is far from ideal.
from a performance perspective, 3 VMs on the same hardware is slower
from an availability perspective, 2 VMs will give higher availability (better protects from app software failures, OS failures, you can perform maintenance on one node while the other is up).
Our network team is thinking of setting up a virtual desktop environment (via Windows 2008 virtual host) for each developer.
So we are going to have dumb terminals/laptops and should be using the virtual desktops for all of our work.
Ours is a Microsoft shop and we work with all versions of .net framework. Not having the development environments on the laptops is making the team uncomfortable.
Are there any potential problems with that kind of setup? Is there any reason to be worried about this setup?
Unless there's a very good development-oriented reason for doing this, I'd say don't.
Your developers are going to work best in an environment they want to work in. Unless your developers are the ones suggesting it and pushing for it, you shouldn't be instituting radical changes in their work environments without very good reasons.
I personally am not at all a fan of remote virtualized instances for development work, either. They're often slower, you have to deal with network issues and latency, you often don't have as much control as you would on your own machine. The list goes on and on, and little things add up to create major annoyances.
What happens when the network goes down? Are your dev's just supposed to sit on their hands? Or maybe they could bring cards and play real solitare...
Seriously, though, Unless you have virtual 100% network uptime, and your dev's never work off-site (say, from home) I'm on the "this is a Bad Idea" side.
One option is to get rid of your network team.
Seriously though, I have worked with this same type of setup through VMWare and it wasn't much fun. The only reason why I did it was because my boss thought it might be worth a try. Since I was newly hired, I didn't object. However, after several months of programming this way, I told him that I preferred to have my development studio on my machine and he agreed.
First, the graphical interface isn't really clear with a virtual workstation since it's sending images over the network rather than having your video card's graphical driver render the image. Constant viewing of this gave me a headache.
Secondly, any install of components or tools required the network administrator's help which meant I had to hurry up and wait.
Third, your computer is going to process one application faster than your server is going to process many apps and besides that, it has to send the rendered image over the network. It doesn't sound like it slows you down but it does. Again, hurry up and wait.
Fourth, this may be specific to VMWare but the virtual disk size was fixed to 4GB which to my network guy seemed to think it was enough. This filled up rather quickly. In order for me to expand the drive, I had to wait for the network admin to run partition magic on my drive which screwed it up and I had to have him rebuild my installation.
There are several more reasons but I would strongly encourage you to protest if you can. Your company is probably trying to impliment this because it's a new fad and it can be a way for them to save money. However, your productivity time will be wasted and that needs to be considered as a cost.
Bad Idea. You're taking the most critical tool in your developers' arsenal and making it run much, much, much slower than it needs to, and introducing several critical dependencies along the way.
It's good if you ever have to develop on-site, you can move your dev environment to a laptop and hit the road.
I could see it being required for some highly confidential multiple client work - there is a proof that you didn't leak any test data or debug files from one customer to another.
Down sides:
Few VMs support multiple monitors - without multiple monitors you can't be a productive developer.
Only virtualbox 3 gets close to being able to develop for opengl/activeX on a VM.
In my experience Virtual environments are ideal for test environments (for testing deployments) and not development environments. They are great as a blank slate / clean sheet for testing. I think the risk of alienating your developers is high if you pursue this route. Developers should have all the best tools at their disposal, i.e. high spec laptop / desktop, this keeps morale and productivity high.
Going down this route precludes any home-working which may or may not be an issue. Virtual environments are by their nature slower than dedicated environments, you may also have issues with multiple monitor setups on a VM.
If you go that route, make sure you bench the system aggressively before any serious commitment.
My experience of remote desktops is that it's ok for occasional use, but seldom sufficient for intensive computations and compilation typical of development work, especially at crunch time when everyone needs resources at the same time.
Not sure if that will affect you, but both VMWare and Virtual PC work very slow when viewed via Remote Desktop. For some reason Radmin (http://www.radmin.com/ ) does a much better job.
I regularly work with remote development environments and it is OK (although it takes some time to get used to keep track in which system you're working at the moment ;) ) - but most of the time I'm alone on the system.