TestNG reports retried methods as separate test runs - testing

I started working with TESTNG for the first time few days back.
I implemented a retry analyzer using the IRetryAnalyzer interface which retries a failed test method for a maximum of 3 times.
I have a test method called retryInvoker() which fails twice and succeeds the third time.
The way TestNG reports this is 3 separate test runs, 2 of which failed and 1 succeeded, which is correct actually.
Although I just wish to capture it as a single test run, which ultimately succeeded (if it did before the maximum allowed retries ended, which in this case were 3). Or even if it didn't succeed after 3 retries, I wish to report it as a single test run that failed instead of 4 separate test runs, all of which failed.
Any leads on this?

You could try the approach, suggested in this SO answer, removing the duplicates from the test context in case the same method is found either in failed or passed tests:
#Overrride
public void onFinish(ITestContext context) {
Iterator<ITestResult> failedTestCases = context.getFailedTests().getAllResults().iterator();
while (failedTestCases.hasNext()) {
System.out.println("failedTestCases");
ITestResult failedTestCase = failedTestCases.next();
ITestNGMethod method = failedTestCase.getMethod();
if (context.getFailedTests().getResults(method).size() > 1) {
System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
} else {
if (context.getPassedTests().getResults(method).size() > 0) {
System.out.println(
"failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
}
}
}
}

Related

How to re execute failed automation scenarios from Jenkins

I am running cucumber tests in the TestNG framework using the maven command. Daily I am executing the test cases from Jenkins and generating the cucumber report in Jenkins. (using cucumber report plugin)
I am looking for a solution to re-run the failed test cases in Jenkins and it should give the final report.
Please provide me the approach to achieve this.
On of the simple way is, use IRetryAnalyzer in TestNG. It will re-run the failed test case.
In final report if re-run passed then i will display as passed (initially failed one display as skipped)
if re-run also failed, then marked as failure.
example:
public class Retry implements IRetryAnalyzer {
private int count = 0;
private static int maxTry = 3;
#Override
public boolean retry(ITestResult iTestResult) {
if (!iTestResult.isSuccess()) { //Check if test not succeed
if (count < maxTry) { //Check if maxtry count is reached
count++; //Increase the maxTry count by 1
iTestResult.setStatus(ITestResult.FAILURE); //Mark test as failed
return true; //Tells TestNG to re-run the test
} else {
iTestResult.setStatus(ITestResult.FAILURE); //If maxCount reached,test marked as failed
}
} else {
iTestResult.setStatus(ITestResult.SUCCESS); //If test passes, TestNG marks it as passed
}
return false;
}
}
Add in Testng.xml file
You can add test to also
#Test(retryAnalyzer = Retry.class)

What is the correct way to get the result of Pipeline?

My code for handling my pipeline result looks like this (some snipped for brevity's sake):
PipelineResult result = pipeline.run();
switch (result.getState()) {
case DONE: {
handleDone();
break;
}
case FAILED: {
handleFailed();
break;
}
case CANCELLED: {
handleCancelled();
break;
}
case UNKNOWN:
case RUNNING:
case STOPPED: {
handleUnknownRunningStopped();
break;
}
default: {
assert false;
throw new IllegalStateException();
}
}
However, I've noticed that instead of returning a value of the enum PipelineResult.State for e.g. FAILED or CANCELLED, an exception is thrown:
For a failed job, a DataflowJobExecutionException is thrown
For a cancelled job, a DataflowJobCancelledException is thrown
What is the correct way (programmatically) to handle the result of a pipeline?
Both DataflowPipelineRunners return the PipelineResult that allows you to query the current status of the pipeline. The DataflowPipelineRunner returns a PipelineResult immediately upon job submission, but the BlockingDataflowPipelineRunner doesn't return it until the job has completed.
In addition, the BlockingDataflowPipeline runner throws an exception if the job does not complete successfully -- since you've specified a blocking run(), we assume you want to know if something goes wrong. So if you've hard coded the Blocking Runner, then relying on an exception is an easy way to handle failure.
Note that the code snippet you wrote uses the more general PipelineResult option, but won't work with the non-blocking runner, since that will return a result while the job is still running.

Retry only specific Failed tests with TestNG

How to execute only the specific failed tests. By using 'IRetryAnalyzer' we can re-run the failed tests for x number of time. As mentioned here Restart failed test case automatically. I have also implemented iTestListener to make the tests count more meaning full by following Retry Only failed Tests and update test run count by implementing 'ITestListener'
Is there any way to Re-run ONLY specific failed tests.
Example: We need to execute only tests which are failed because of NoSuchElementException and TimeoutException.
Please find the below screen shot where total 8 tests are failed and there are 6 tests which are failed because of NoSuchElementException-1 and TimeoutException-5.
Please help.
You can try out by checking the result of your tests like:
#Override
public boolean retry(ITestResult result) {
try {
if (result.getThrowable().toString()
.contains("NoSuchElementException")) // Checking for specific reasons of failure
if (retryCount < maxRetryCount) {
retryCount++;
return true;
}
return false;
} catch (Exception e) {
return false;
}
}
Since every result has an attribute m_throwable in case of exception has occurred, you can use it to get your task done in Retry class.

Can't perform a Laravel 4 action/route test more than once

I'm running into a Laravel 4 testing issue: An action/route test can only be run once, and it has to be the first test run. Any subsequent action/route test will fail with an exception before the assert is called.
route/action tests run as long as they are the first test run.
Non-route/action tests run normally, although they cause subsequent route/action tests to throw an exception
It's important to note that the tests in question don't fail, they throw an exception when the action is fired, for example:
Symfony\Component\Routing\Exception\RouteNotFoundException: Unable to generate a URL for the named route "home" as such route does not exist.
Sample test class:
class ExampleTest extends TestCase {
// passes
public function testOne()
{
$class = MyApp::ApiResponse();
$this->assertInstanceOf('\MyApp\Services\ApiResponse', $class);
}
// this fails unless moved the top of the file
public function testRoute()
{
$this->route('GET','home');
$this->assertTrue($this->client->getResponse()->isOk());
}
// passes
public function testTwo()
{
$class = MyApp::ProjectService();
$this->assertInstanceOf('\MyApp\Services\ProjectService', $class);
}
}
This is implementation-specific, a fresh Laravel 4 project does not exhibit the issue. What could be causing this behaviour? How would you go about tracking down the problem?
In this case, the routes file was being called using an include_once call. When subsequent tests were run the routes were empty.
Changing to include() fixed the issue exhibited in the question

markTestSkipped() not working with sausage-based Selenium tests via Sauce Labs

I am using the sausage framework to run parallelized phpunit-based Selenium web driver tests through Sauce Labs. Everything is working well until I want to mark a test as skipped via markTestSkipped(). I have tried this via two methods:
setting markTestSkipped() in the test method itself:
class MyTest
{
public function setUp()
{
//Some set up
parent::setUp();
}
public function testMyTest()
{
$this->markTestSkipped('Skipping test');
}
}
In this case, the test gets skipped, but only after performing setUp, which performs a lot of unnecessary work for a skipped test. To top it off, phpunit does not track the test as skipped -- in fact it doesn't track the test at all. I get the following output:
Running phpunit in 4 processes with <PATH_TO>/vendor/bin/phpunit
Time: <num> seconds, Memory: <mem used>
OK (0 tests, 0 assertions)
The other method is by setting markTestSkipped() in the setUp method:
class MyTest
{
public function setUp()
{
if (!shouldRunTest()) {
$this->markTestSkipped('Skipping test');
} else {
parent::setUp();
}
}
protected function shouldRunTest()
{
$shouldrun = //some checks to see if test should be run
return $shouldrun;
}
public function testMyTest()
{
//run the test
}
}
In this case, setUp is skipped, but the test still fails to track the tests as skipped. phpunit still returns the above output. Any ideas why phpunit is not tracking my skipped test when they are executed in this fashion?
It looks like, at the moment, there is no support for logging markTestSkipped() and markTestIncomplete() results in PHPunit when using paratest. More accurately, PHPunit won't log tests which call markTestSkipped() or markTestIncomplete() if called with arguments['junitLogfile'] set -- and paratest calls PHPunit with a junitLogfile.
For more info, see: https://github.com/brianium/paratest/issues/60
I suppose I can hack away at either phpunit or paratest...