I've recently switched from FirefoxDriver to GhostDriver.
My test suite (roughlty around 150 end to end tests) used to take 25 minutes to run. Now with Ghostdriver it's taking 23 minutes. So in total I've gained 2 minutes of run time.
It's an improvement but not as huge as I would've expected. Is such a low gain normal ? Or should I expect a much higher cut down time from switching to an headless test runner ?
I'm using the .NET version of webdriver/ghostdriver.
I'd be really interested to compare this "benchmark" with anyone having recently switched to GhostDriver
GhostDriver won't offer any performance gain over ChromeDriver as the only thing it is not doing is rendering the graphics on the screen which Chrome does using a graphics driver and not the CPU.
The only benefit I see is not popping up a browser window and maybe when running on a CI server.
Some stats on my test run are on my blog: http://watirmelon.com/2013/02/05/watir-webdriver-with-ghostdriver-on-osx-headless-browser-testing/
I feel like this is bit vague question. For example, if your test contains lots of Thread.sleep() calls, then there will be no improvement to the tests.
My tests use ChromeDriver and I didn’t think about switching. But I feel like that simple switch will not gain you much, maybe also consider refactoring your code
Related
I read that the --runInBand flag speeds up Jest test duration by 50% on CI servers. I can't really find an explanation online on what that flag does except that it lets tests run in the same thread and sequentially.
Why does running the test in the same thread and sequentially make it faster? Intuitively, shouldn't that make it slower?
Reading your linked page and some other related sources (like this github issue) some users have found that:
...using the --runInBand helps in an environment with limited resources.
and
... --runInBand took our tests from >1.5 hours (actually I don't know how long because Jenkins timed out at 1.5 hours) to around 4 minutes. (Note: we have really poor resources for our build server)
As we can see, those users had improvements in their performances on their machines even though they had limited resources on them. If we read what does the --runInBand flag does from the docs it says:
Alias: -i. Run all tests serially in the current process, rather than creating a worker pool of child processes that run tests. This can be useful for debugging.
Therefore, taking into consideration these comments and the docs, I believe the improvement in performance is due to the fact that now the process runs in a single thread. This greatly helps a limited-resource-computer because it does not have to spend memory and time dealing and handling multiple threads in a thread pool, a task that could prove to be too expensive for its limited resources.
However, I believe this is the case only if the machine you are using also has limited resources. If you used a more "powerful" machine (i.e.: several cores, decent RAM, SSD, etc.) using multiple threads probably will be better than running a single one.
When you run tests in multi-threads, jest creates a cache for every thread. When you run with --runInBand jest uses one cache storage for all tests.
I found it after runs 20 identical tests files, first with key --runInBand, a first test takes 25 seconds and next identical tests take 2-3s each.
When I run tests without --runInBand key, each identical test file executes in 25 seconds.
At the moment I'm programming with Jython on my Laptop, but want to use it on my Raspberry Pi3 (Raspbian) later.
Well the start-up time of my program on my Laptop is under 2s, but on my Pi3 it's up to 30s.
I know the issues are that Jython needs time to start up the JVM and even a Pi3 is not as fast as my 3 year old Laptop, but Is it maybe still possible to reduce this start-up time anyway (without over-clocking my Pi)?
EDIT:
At the end I want to use my .py scripts with the jython-standalone.jar v2.7.0
Well I found a solution which is pretty ok for my Project: For small projects OpenJDK seems to be ok, but some things WON'T work with Jython. I had a sleepless weekend until I get this idea to look which version of Java I was using. With the oracle Java(newest Version) it seems to be as fast as on my Laptop Windows etc. (yes it is a little bit slower, but a second less or more is not necessary in my case).
If there is still a faster way to startup, surprise me. :)
I got a requirement from my client to implement Selenium grid. Requirement details as follows:
Number of nodes running concurrent test cases: 200-250
3 Selenium nodes per machine (~ 66 to 85 machines running the 200-250 nodes)
Every machine is a Windows machine and has 8GB RAM
The test cases are long-running in nature. about 10000 steps per test case. (click, type etc are considered to be the test steps)
The option of using grid services from providers like browserstack is ruled out. I will have to do the grid setup locally.
I am looking for answers for questions like, Has anyone tried grid setup with such level of complexity? Are there any limitations of Selenium to support such large setup? Would there be performance impact due to excessive context switches between threads? What problems should I be prepared to face?
Thank you for your help.
It seems possible to me. Some things to consider:
Long running tests with many steps should be broken down into
smaller test units.
Each grid node, if they have 8GB of ram and dual-proc, should be able to handle up to 10+ browsers, not just 3.
If the tests are simple, you could use PhantomJS 2.x headless browser and get more speed and throughput out of your tests. This
is especially useful if the version of the browser your testing with
doesn't much matter, such as with verification tests. Regression
tests, on the other hand, need specific browser versions to verify
regressions caused by code.
I would say that a grid hub with many nodes on it , should be able to handle 50+ threads at a level of 8gb and dual proc. So, for
each 50 threads you would want 5 nodes and 1 hub.
Thats just some thoughts...
i profile a big jboss server with a lot of classes in it. When i profile the CPU the result is always something like java.util.TimerThread.run() = 62% and java.util.concurrent.ThreadPoolExecutor$Worker.run() = 34,8%.
Under these two methods thousand other methods have 0%.
I think thats a bad bug, because most of these methods run in these Threads. But how can i see which one...
The ThreadDump - Function isnt usefull for this too.
If you don't know which part of the code is slow, it is better to start with CPU sampling. Once you know better (based on the sampling results) what is wrong, you can profile just part of your jboss server. See Profiling With VisualVM, Part 1 and Profiling With VisualVM, Part 2 to get more information about profiling and how to set profiling roots and instrumentation filter.
Does anybody have experience using 20/50/100 RCs connected to a single Selenium Hub? I'm thinking of a centralized hub that multiple team could use together.
I heard that after 20 RCs, performance goes down significantly.
My tests use 104 RCs which are connected to a single Hub. Hub is able to handle all the RCs and the system is stable. One thing that you should note is, how you pass the commands to the Hub.This determines how much RCs Hub is going to utilize. What we do in our grid is, we run multiple builds at the same time - all pointing to the same Hub. This would mean Hub is always kept busy.
I'm not sure where the "20" value is coming from. I run with significantly more than that to power Mogotest and we run many more test sessions across various sites than most users of grid. Granted older versions had some thread problems, but 1.0.8 should be pretty solid and the 2.0 releases have not exhibited any problems.
FWIW, I maintain the grid project and backport any known fix for any known load issues. I also know of several others running large-scale grid installations without incident.
It boils down to how robust your h/w is. Though I can't say exact statistics yet suggest to keep it with in 20
Well there is not but yes very much close to 20 for good performance.