My existing code base consists of 500+ unitests, and due to resource constraints in test environments running them all in a single process fails with memory problems.
Is there a tool to run the tests in batches.
Eg.
MyTests --gtest_list_tests
1A.A
...
1000A.A
Tool MyTests --batch-count 10
will first run the first 10 in a single process.
After process stops it will start the next 10 in another process.
Manually adding a filter is not feasible.
Google Test has support for sharding tests. You can specify the total number of shares you want to split your tests to and the index of the current shard to run in environment variables.
Related
how to run my selenium script on multiple servers at the same time so that i would save one by one execution time ?
Scenario: I have 1 hour of total testing time and 20 servers to test and 1 test script takes about 30mins to execute so i want to run my test script simultaneously on all 20 servers in order to save time.
The answer to your question is parallel execution
There are multiple ways to achieve this like 1> Creating a Jenkins Job, Registering all the server machines as slave and executing jobon servers
Or
2> A comparitively simpler and more widely used concept, using Selenium Grid
To Implement Selenium Grid ( for instance with java as a programming language and TestNG framework) we need to take care that
A. we have implemented Threading for our webdriver, so that each execution works on its own driver instance.
B. We have our Testng.xml with attribute set as parallel=tests
You can easily get step by step guide to establish selenium grid to summarize it, We have one machine as hub(master), we have multiple machines or nodes(slave) registered to the hub, your automation code is supplied to the hub, Hub routes test on multiple nodes keeps track of individual execution and gets you the result on the Hub machine itself
This is not clear to me from the docs. Here's our scenario and why we need this as succinctly as I can:
We have 60 coordinators running, launching workflows usually hourly, some of which have sub-workflows (some multiple in parallel). This works out to around 40 workflows running at any given time. However when cluster is under load or some underlying service is slow (e.g. impala or hbase), workflows will run longer than usual and back up so we can end up with 80+ workflows (including sub-workflows) running.
This sometimes results in ALL workflows hanging indefinitely, because we have only enough memory and cores allocated to this pool that oozie can start the launcher jobs (i.e. oozie:launcher:T=sqoop:W=JobABC:A=sqoop-d596:ID=XYZ), but not their corresponding actions (i.e. oozie:action:T=sqoop:W=JobABC:A=sqoop-d596:ID=XYZ).
We could simply allocate enough resources to the pool to accommodate for these spikes, but that would be a massive waste (hundreds of cores and GBs that other pools/tenants could never use).
So I'm trying to enforce some limit on number of workflows running, even if that means some will be running behind sometimes. BTW all our coordinators are configured with execution=LAST_ONLY, and any delayed workflow will simply catch up fully on the next run. We are on CDH 5.13 with Oozie 4.1; pools are setup with DRF scheduler.
Thanks in advance for your ideas.
AFAIK there is not a configuration parameter that let you control the number of workflows running at a given time.
If your coordinators are scheduled to run approximately in the same time-window, you could think to collapse them in just one coordinator/workflow and use the fork/join control nodes to control the degree of parallelism. Thus you can distribute your actions in a K number of queues in your workflow and this will ensure that you will not have more than K actions running at the same time, limiting the load on the cluster.
We use a script to generate automatically the fork queues inside the workflow and distribute the actions (of course this is only for actions that can run in parallel, i.e. there no data dependencies etc).
Hope this helps
I am implementing a Cucumber - JVM based Selenium automation framework.
One of the workflow in the webapps i test, requires a long wait so that a batch job that is scheduled as frequently as in every 3 minutes, runs, and creates a login id, which the user can utilize, to continue with the workflow.
I am currently handling it in such a way that i execute the initial part test case first and continue with other test cases, so that the framework gets ample time to wait for the user id to be created.
After all other test cases are run the second part of the test case is run. But, before running the second part of the test case, i query the database and verify whether the id is created. If the id is created then the execution continues else, fails saying that the user id was not created.
Although this works for now, i am sure there are better ways to handle such scenarios. Have any one of you come across such a scenario? How did you handle it ?
I think I understand your problem. You actually would like to have an execution sequence like this probably:
Test 1
Test 2
Test 3
But if you implement Test 1 "correctly" it will take very long because it has to wait for the system under test to do several long running things, right?
Therefore you split Test 1 into to parts and run the tests like this:
Test 1 (Part A)
Test 2
Test 3
Test 1 (Part B)
So that your system under test has time to finish the tasks triggered by Test 1 (Part A).
As you acknowledged before, this is considered bad test design, as your test cases are no longer independent from each other. (Generally speaking no test case should rely on side effects created by another test case beforehand.)
My recommendation for that scenario would be to leave Test 1 atomic, i.e. avoid splitting it in two parts, but still run the rest of the tests in parallel. Of course whether or ot this is feasible depends on your environment and on how you trigger the tests, but that would allow you to have a better structure of your tests plus the benefit of fast execution. So you would end up with this:
Test 1 starts Test 2 starts
Test 1 waits Test 2 finishes
Test 1 waits Test 3 starts
Test 1 runs Test 3 finishes
Test 1 finishes
I am not sure about the start—>wait—>wait—>run approach. It might work for few tests and may not work well for hundreds of tests, as the wait time would be more. Even if we run it in parallel mode it would consume some time. What if we wait for more such time taking components in the same flow? More the components more the wait time, I guess. You may also need to consider timeout of the system, if you wait for longer time...
I feel even the first approach should be fine. No need to create multiple files for a test case. You can structure it in the same file in a way that you run first part and end it. And, after ensuring the batch processing you can start with second part of your test case(file). The first part can be run in parallel mode and after waiting time part 2 can also be executed in parallel.
My Situation:
I have 60 dedicated selenium nodes that I use, in AWS. My Selenium Grid and all my Selenium Nodes are in the AWS. (cuts down network requests)
My Problem:
I am experiencing an issue when executing my regression suite. I run only 4 tests at a time due to our nightly servers only being able to handle so many tests going at it at once. The problem that i've diagnosed is that when using t1.micro instances, the CPU tops off when launching only 1 test per 1 node. I keep getting several timeouts. Fine. I can easily upgrade these to m1.small but it makes me think more about the way that the selenium grid selects it's nodes. Per my understanding, this is a quick getup on how the hub handles it's nodes.
So if we only have 4 tests running at a time, it will select the first four nodes. Once these nodes are finished running the tests, it will launch another test against the same instances. Is there any way that I can bypass this logic to just select a random available node rather than "the next one in queue"? I'm noticing that the X amount of nodes after the four nodes are NEVER being utilized.
Sidenote:
I have 60 nodes because this grid will have several different types of regression suites against it. Not only mine.
Edit (July 25, 2015)
A pull request was created for this functionality: https://github.com/SeleniumHQ/selenium/pull/832
This pull request should fix the issue
https://github.com/SeleniumHQ/selenium/pull/832
In our environment we have quite a few long-running functional tests which currently tie up build agents and force other builds to queue. Since these agents are only waiting on test results they could theoretically just be handing off the tests to other machines (test agents) and then run queued builds until the test results are available.
For CI builds (including unit tests) this should remain inline as we want instant feedback on failures, but it would be great to get a better balance between the time taken to run functional tests, the lead time of their results, and the throughput of our collective builds.
As far as I can tell, TeamCity does not natively support this scenario so I'm thinking there are a few options:
Spin up more agents and assign them to a 'Test' pool. Trigger functional build configs to run on these agents (triggered by successful Ci builds). While this seems the cleanest it doesn't scale very well as we then have a lead time of purchasing licenses and will often have need to run tests in alternate environments which would temporarily double (or more) the required number of test agents.
Add builds or build steps to launch tests on external machines, then immediately mark the build as successful so queued builds can be processed then, when the tests are complete, we mark the build as succeeded/failed. This is reliant on being able to update the results of a previous build (REST API perhaps?). It also feels ugly to mark something as successful then update it as failed later but we could always be selective in what we monitor so we only see the final result.
Just keep spinning up agents until we no longer have builds queueing. The problem with this is that it's a moving target. If we knew where the plateau was (or whether it existed) this would be the way to go, but our usage pattern means this isn't viable.
Has anyone had success with a similar scenario, or knows pros/cons of any of the above I haven't thought of?
Your description of the available options seems to be pretty accurate.
If you want live update of the builds progress you will need to have one TeamCity agent "busy" for each running build.
The only downside here seems to be the agent licenses cost.
If the testing builds just launch processes on other machines, the TeamCity agent processes themselves can be run on a low-end machine and even many agents on a single computer.
An extension to your second scenario can be two build configurations instead of single one: one would start external process and another one can be triggered on external process completeness and then publish all the external process results as it's own. It can also have a snapshot dependency on the starting build to maintain the relation.
For anyone curious we ended up buying more agents and assigning them to a test pool. Investigations proved that it isn't possible to update build results (I can definitely understand why this ugliness wouldn't be supported out of the box).