Selenium Run Functional Tests Distribute By Test Time - selenium

I'm running Run Functional Tests distributed by machines number.
Now I've around 850 tests and it takes around 3.5 hours. (8 Servers)
I added another 8 servers and now it takes around 2.2 hours.
Because it distributed by machine number, every machine gets 53 tests but there are tests which takes 30 seconds and some tests that takes 5 mintues.
That situation is annoying, because there are machine which finish after their job after (53 * 30seconds) = 1500 seconds, around 25 minutes while there are servers which finish their work after 2 hours.
It makes my build very slower.
I want somehow to distribute it by running time, so that every machine will work the same time and start and finish at the same time.
Thanks for helping

Task Run functional tests is deprecated in VSTS and TFS 2018. Instead of using Run functional tests, you should use Visual Studio Test task.
Task Visual Studio Test in VSTS/TFS 2018, supports Based on past running time of tests. Based on past running time of tests: This batching considers past running time to create batches of tests such that each batch has approximately equal running time. This option should meet you requirement.

Related

Repast - Is batch run already parallelized?

If I my computer has 12 cores and my model has 15 scenarios runs, is the batch run automatically distribute the initial 12 runs to each core and run the respective runs concurently to save time? If yes, I'd like to also know if I can control the use of cores, e.g. limit to using 8 cores at a time of runinng the batch runs to prevent OOM if a single run is large scale.
Take a look at section 2.3 Host Panel here. You'll see that the Instances property determines how many independent workers will be used to process the scenarios you define. E.g., if indicate Instances: 8, then 8 workers using 8 cores will be processing your 15 scenarios.

Speed up application's launch time when running UI Tests

When running UI tests, the application's launch oscillates between 4 and 10 seconds, going sometimes up to 37s seconds.
Is there anyway to make it faster?

Why does Jest --runInBand speed up tests?

I read that the --runInBand flag speeds up Jest test duration by 50% on CI servers. I can't really find an explanation online on what that flag does except that it lets tests run in the same thread and sequentially.
Why does running the test in the same thread and sequentially make it faster? Intuitively, shouldn't that make it slower?
Reading your linked page and some other related sources (like this github issue) some users have found that:
...using the --runInBand helps in an environment with limited resources.
and
... --runInBand took our tests from >1.5 hours (actually I don't know how long because Jenkins timed out at 1.5 hours) to around 4 minutes. (Note: we have really poor resources for our build server)
As we can see, those users had improvements in their performances on their machines even though they had limited resources on them. If we read what does the --runInBand flag does from the docs it says:
Alias: -i. Run all tests serially in the current process, rather than creating a worker pool of child processes that run tests. This can be useful for debugging.
Therefore, taking into consideration these comments and the docs, I believe the improvement in performance is due to the fact that now the process runs in a single thread. This greatly helps a limited-resource-computer because it does not have to spend memory and time dealing and handling multiple threads in a thread pool, a task that could prove to be too expensive for its limited resources.
However, I believe this is the case only if the machine you are using also has limited resources. If you used a more "powerful" machine (i.e.: several cores, decent RAM, SSD, etc.) using multiple threads probably will be better than running a single one.
When you run tests in multi-threads, jest creates a cache for every thread. When you run with --runInBand jest uses one cache storage for all tests.
I found it after runs 20 identical tests files, first with key --runInBand, a first test takes 25 seconds and next identical tests take 2-3s each.
When I run tests without --runInBand key, each identical test file executes in 25 seconds.

What's the best way to 'indicate/numerate' performance of an application?

In the old (single-threaded) days we instructed our testing team to always report the CPU time and not the real-time of an application. That way, if they said that in version 1 an action took 5 CPU seconds, and in version 2 it took 10 CPU seconds, that we had a problem.
Now, with more and more multi-threading, this doesn't seem to make sense anymore. It could be that the version 1 of an application takes 5 CPU seconds, and version 2 10 CPU seconds, but that version 2 is still faster if version 1 is single-threaded, and version 2 uses 4 threads (each consuming 2.5 CPU seconds).
On the other hand, using real-time to compare performance isn't reliable either since it can be influenced by lots of other elements (other applications running, network congestion, very busy database server, fragmented disk, ...).
What is in your opinion the best way to 'numerate' performance?
Hopefully it's not intuition since that is not an objective 'value' and probably leads to conflicts between the development team and the testing team.
Performance needs to be defined before it is measured.
Is it:
memory consumption?
task completion times?
disk space allocation?
Once defined, you can decide on metrics.

What types of testing do you include in your build process?

I use TFS 2008. We run unit tests as part of our continuous integration build and integration tests nightly.
What other types of testing do you automate and include in your build process? what technologies do you use to do so?
I'm thinking about smoke tests, performance tests, load tests but don't know how realistic it is to integrate these with Team Build.
First, we have check-in (smoke) tests that must run before code can be checked in. It's done automatically by running a job that runs the tests and then makes the check-in to source control upon successful test completion. Second, cruise control kicks off build and regression tests. The product is built then several sets of integration tests are run. The number of tests vary by where we are in the release cycle. More testing is added late in the cycle during ramp down. Cruise control takes all submissions within a certain time window (12 minutes) so your changes may be built and tested with a small number of others. Third, there's an automated nightly build and tests that are quite extensive. We have load or milestone points every 2 or 3 weeks. At a load point, all automated tests are run plus manual testing is done. Performance testing is also done for each milestone. Performance tests can be kicked off on request but the hardware available is limited so people have to queue up for performance tests. Usually people rely on the load performance tests unless they are making changes specifically to improve performance. Finally, stress tests are also done for each load. These tests are focussed on making sure the product has no memory leaks or anything else that prevents 24/7 running of the product as opposed to performance. All of this is done with ant, cruise control, and Python scripts.
Integrating load testing during you build process is a bad idea, just do your normal unit testing to make sure that all your codes work as expected. Load and performance testing should be done separately.